How to test an AI annual-review workflow

Define the finished task

“Prepare an annual review” is too broad for a useful comparison. Specify a sourced household summary, an exceptions list, a preparation checklist and a draft invitation. Define what the workflow must leave unresolved and which actions are outside scope.

For example, a missing mortgage payment should remain missing. A closed account’s old valuation should not silently enter a current total. A draft invitation should arrange a review without introducing a new investment recommendation. Name the person who can accept the output.

Agree a representative sample and a reusable failure set

Our standard one-workflow pilot agrees 12–20 representative cases, with a maximum of 20 included cases, before work starts. That sample is supplemented by reusable synthetic failure cases. It is an evaluation sample, not proof of universal accuracy.

Use only information and systems authorised for the trial. The public example uses fixed fictional records and prepared outputs; it does not demonstrate that a live integration has been validated.

Measure preparation, checking and correction

Record active minutes for the existing process and the proposed process separately. Use the same task boundary and comparable case complexity. Include preparing inputs, operating the tool, checking sources and correcting the output. Record waiting time separately so a faster provider response is not mistaken for staff capacity.

For each case, retain the expected source set, omissions, unsupported statements, severity, useful exceptions, output decision, evaluator, date and workflow/product/model version. Record per-task software or API cost only where measured. A proposed 25% reduction in median total task time is a target to test, not an achieved result or guarantee.

Case ID / complexity / source set / expected output
Baseline: preparation __ + checking __ + correction __ = __ minutes
Assisted: preparation __ + checking __ + correction __ = __ minutes
Omissions / unsupported statements / severity / useful exceptions
Version / evaluator / date / measured task cost / decision / remediation

Set an explicit human review gate

Every consequential statement should have supporting evidence or be visibly unresolved. A resolved exception needs an explanation, and missing authority or unavailable evidence cannot be repaired by clicking a status button. Check dates and record identity as well as the wording.

Keep recommendations, client communications and changes to source systems subject to the firm’s authorised approval. Source access should begin read-only. CPS implementation support and separately agreed case-by-case paraplanning work have different scopes.

Choose rollout, adjustment or stop

Our proposed pilot gate is no observed critical errors or access failures in the agreed test set, supported or explicitly unresolved consequential facts, no unapproved actions, and quality accepted by the named reviewer. Disclose the measured time and cost result even if disappointing.

After the agreed adjustment cycle, repeat the affected tests and retain the original result. Record a decision to roll out, narrow the scope, adjust further under a new agreement or stop. Check continued usage after rollout; a successful test that nobody uses has not established adoption.