{ "title": "A fictional IFA meeting-note review: source, AI draft and checks", "experiment_date": "2026-09-08", "data_status": "Entirely fictional. No client records or personal data were used.", "purpose": "Demonstrate a source-to-draft review method for an internal meeting handover, not test suitability advice or predict production performance.", "source_id": "DEMO-IFA-01", "source": "FICTIONAL TRAINING RECORD — DEMO-IFA-01\nMeeting: 8 September 2026. Attendees: Client A and Adviser A. Internal handover notes, not a recommendation.\n\nClient A is 52 and employed. They mentioned stopping work at 63, then said they would like to explore 60 if affordable. No retirement age was agreed.\n\nWorkplace pension: £198,000 according to a statement dated 31 August 2026, described in the meeting notes. A value of £168,000 in the old preparation sheet was copied from 2024. Current charges and investment holdings were not available at this meeting. Client asked whether consolidating an old pension might simplify administration. No transfer or investment recommendation was made.\n\nCash savings: client estimates £18,500; supporting statement not supplied. Mortgage balance: client estimates £94,000. Mortgage statement still required. These are client estimates, not verified balances.\n\nMonthly essential spending appears as £2,200 in the opening notes and £2,600 in the budget discussion. Both are labelled current. Nobody clarified the difference. Annual travel spending of £3,000 was mentioned separately, with no decision about whether it is included in either monthly figure.\n\nClient wants to protect time with family and avoid a long commute. Risk questionnaire was not updated. Capacity for loss was not discussed in these notes. Recording permission and permission to process the notes with an AI tool are not recorded here; do not assume either.\n\nActions dictated at the end: Adviser A to request pension charges and current holdings by 11 September 2026. Client A to send the mortgage statement by 15 September 2026. Client A to clarify essential spending and whether the travel budget is additional; no deadline agreed. Administrator to arrange a follow-up; date not agreed. Client prefers a telephone follow-up rather than video.\n\nNo provider name, policy number, email address, tax calculation or recommendation is recorded.", "prompt": "Using only the fictional record supplied, write an internal adviser handover of no more than 250 words. Use the headings Recorded facts, Unresolved questions, and Actions. Preserve the distinction between reported and verified information. Identify conflicting figures without choosing one. Do not infer suitability, permission, provider details, deadlines or recommendations. Include action owners and recorded deadlines. If something necessary is not recorded, say so. This is a draft for source checking, not client-facing advice.", "method": { "generation": "The assistant performing this task generated one draft directly from the source and prompt, without external tools, retrieval or a second drafting pass. The draft below was saved before the comparison checklist was written.", "review": "A subsequent source comparison was performed by the same assistant. It was not independent or blinded human review.", "model": "AI assistant in this task; no independently recorded model identifier or configuration.", "sample_size": 1, "measured": [ "Presence or explicit treatment of specified source facts in the saved draft" ], "not_measured": [ "Human baseline", "Elapsed drafting or review time", "Cost", "Suitability", "Privacy compliance", "Accuracy on real client records", "Any client outcome" ] }, "draft": "Recorded facts\nClient A is 52 and employed. They want more family time and less commuting. Retirement at 60 and 63 was discussed; neither was agreed.\nThe notes report a workplace pension of £198,000 from a statement dated 31 August 2026. The £168,000 preparation figure is from 2024. This draft does not verify the statement. Cash of £18,500 and a mortgage of £94,000 are client estimates; supporting statements have not been supplied.\nConsolidation was raised as a question. No transfer or investment recommendation was made.\n\nUnresolved questions\nEssential spending is recorded as both £2,200 and £2,600 monthly. Neither figure is confirmed. Whether the £3,000 annual travel budget is additional is unclear.\nCurrent pension charges and holdings are unavailable. The risk questionnaire was not updated; capacity for loss is not recorded. Recording and AI-processing permissions are not recorded and must not be assumed.\n\nActions\nAdviser A: request current pension charges and holdings by 11 September 2026.\nClient A: send the mortgage statement by 15 September 2026; clarify essential spending and treatment of travel, with no deadline agreed for that clarification.\nAdministrator: arrange a follow-up; no date agreed. Client A prefers telephone to video.", "draft_word_count": 192, "review": { "check_selection": "These checks were written after the draft was saved. This is an illustrative, same-assistant source comparison, not a preregistered or independent benchmark.", "checks": [ { "id": "C01", "source_requirement": "Client is 52 and employed.", "draft_evidence": "Client A is 52 and employed.", "result": "preserved" }, { "id": "C02", "source_requirement": "More family time and avoiding a long commute are personal priorities.", "draft_evidence": "They want more family time and less commuting.", "result": "preserved" }, { "id": "C03", "source_requirement": "Retirement at 60 and 63 was discussed without an agreed age.", "draft_evidence": "Retirement at 60 and 63 was discussed; neither was agreed.", "result": "uncertainty preserved" }, { "id": "C04", "source_requirement": "£198,000 is the value reported from a 31 August 2026 statement; £168,000 is old 2024 information.", "draft_evidence": "The notes report a workplace pension of £198,000 from a statement dated 31 August 2026. The £168,000 preparation figure is from 2024. This draft does not verify the statement.", "result": "value, dates and evidence limitation preserved" }, { "id": "C05", "source_requirement": "£18,500 cash and £94,000 mortgage are estimates with supporting statements outstanding.", "draft_evidence": "Cash of £18,500 and a mortgage of £94,000 are client estimates; supporting statements have not been supplied.", "result": "estimates preserved" }, { "id": "C06", "source_requirement": "Two conflicting current essential-spending figures, £2,200 and £2,600, remain unresolved.", "draft_evidence": "Essential spending is recorded as both £2,200 and £2,600 monthly. Neither figure is confirmed.", "result": "conflict explicitly flagged" }, { "id": "C07", "source_requirement": "It is unclear whether £3,000 annual travel spending is additional to monthly spending.", "draft_evidence": "Whether the £3,000 annual travel budget is additional is unclear.", "result": "uncertainty preserved" }, { "id": "C08", "source_requirement": "Consolidation was a client question; there was no transfer or investment recommendation.", "draft_evidence": "Consolidation was raised as a question. No transfer or investment recommendation was made.", "result": "question and recommendation distinguished" }, { "id": "C09", "source_requirement": "Current pension charges and holdings were unavailable.", "draft_evidence": "Current pension charges and holdings are unavailable.", "result": "missing information flagged" }, { "id": "C10", "source_requirement": "Risk questionnaire not updated; capacity for loss not discussed in these notes.", "draft_evidence": "The risk questionnaire was not updated; capacity for loss is not recorded.", "result": "record limitations preserved" }, { "id": "C11", "source_requirement": "Recording and AI-processing permissions were not recorded; neither should be assumed.", "draft_evidence": "Recording and AI-processing permissions are not recorded and must not be assumed.", "result": "both missing entries flagged" }, { "id": "C12", "source_requirement": "Adviser A requests pension charges and holdings by 11 September 2026; Client A sends mortgage statement by 15 September 2026.", "draft_evidence": "Adviser A: request current pension charges and holdings by 11 September 2026. Client A: send the mortgage statement by 15 September 2026", "result": "owners and recorded deadlines preserved" }, { "id": "C13", "source_requirement": "Client A clarifies spending and travel; no clarification deadline was agreed.", "draft_evidence": "clarify essential spending and treatment of travel, with no deadline agreed for that clarification.", "result": "missing deadline preserved" }, { "id": "C14", "source_requirement": "Administrator arranges follow-up; date not agreed; telephone preferred to video.", "draft_evidence": "Administrator: arrange a follow-up; no date agreed. Client A prefers telephone to video.", "result": "owner, missing date and preference preserved" } ], "finding": "The saved draft explicitly flags the conflicting spending figures and the unrecorded permission entries. The listed source checks were preserved in this one draft. The reviewer did not identify an unsupported financial recommendation in it. This is coverage against the listed checks, not an accuracy percentage or approval for client use.", "limitations": [ "One short synthetic record and a prompt designed to preserve uncertainty.", "The source, draft and review were produced by the same assistant; the check list was written after generation.", "No independent human review, repeated runs, competing tools, audio transcription, real client data or connected systems.", "No measured time saving, cost saving or client outcome.", "A missing permission entry in the notes does not establish whether permission exists elsewhere; no lawful-basis decision was evaluated.", "An internal handover summary cannot establish suitability or replace source verification." ] }, "training": { "label": "Deliberately flawed training example — fabricated errors, not the recorded AI output", "flawed_summary": "Client A will retire at 60. Their current workplace pension is £168,000. Essential spending is £2,200 a month, including the £3,000 annual travel budget. Client A gave permission to process the notes using AI. Adviser A recommended pension consolidation. Adviser A will request pension charges and holdings by 11 September 2026, and Client A will send the mortgage statement by 15 September 2026.", "exercise": [ "Read the fictional source before the flawed summary. Underline claims that are contradicted, unresolved or unsupported by the source.", "For each issue, quote the relevant source text and rewrite the claim without adding a fact.", "List the unresolved matters that need clarification and assign only the owners or deadlines actually recorded.", "Compare your findings with the answer key. Treat the example as an exercise in record checking, not a suitability assessment." ], "answer_key": [ { "issue": "An agreed retirement age was invented.", "correction": "Retirement at 60 and 63 was discussed; no age was agreed." }, { "issue": "An old pension value was presented as current.", "correction": "The notes report £198,000 from a statement dated 31 August 2026; £168,000 came from 2024 preparation material." }, { "issue": "One spending figure was selected despite a contradiction.", "correction": "Essential spending is recorded as both £2,200 and £2,600 monthly and needs clarification." }, { "issue": "The treatment of travel spending was invented.", "correction": "It is not recorded whether £3,000 annual travel spending is included in or additional to monthly spending." }, { "issue": "AI-processing permission was invented.", "correction": "Permission to process the notes with an AI tool is not recorded here; do not assume it." }, { "issue": "A client question was converted into an adviser recommendation.", "correction": "The client asked about consolidation; no transfer or investment recommendation was made." } ], "marking_rubric": "For training discussion, award one point for identifying each of the six listed errors and one for each source-faithful correction, up to 12. Accept equivalent accurate wording. The two recorded action deadlines are supported and should not be marked as errors. Other omissions can be discussed separately; this rubric is not a competence certification.", "review_checklist": [ "Label the record as a draft and state its intended internal use.", "Check every amount, date, owner and deadline against the source.", "Separate current information, old information, estimates and evidence actually inspected.", "Carry contradictory figures and missing information into an explicit query list.", "Distinguish questions, options discussed and recommendations actually made.", "Do not infer recording or AI-processing permission from silence in the notes.", "Have an accountable reviewer resolve or retain the open questions before downstream use.", "Keep source, prompt, output and corrections together so the draft can be traced." ] }, "second_review": { "method": "Separate AI reviewer with original checklist visible; not blinded or human validation.", "findings": [ "The draft omits that exploring retirement at 60 was conditional on affordability, and that 63 was mentioned first.", "The consolidation question concerned an old pension and simplifying administration; those details are lost.", "“Protect time with family” becomes “more family time”, a small shift beyond what the source says." ], "date": "2026-09-08" } }