A synthesis that checks its own work.

• 01No hidden judgementEvery rule the critic applies is a paragraph of plain language in a config file a team can edit.
• 02No silent approvalA critic that returns nothing fails the check. This rule was learned, not designed — section 05.
• 03No deletion as repairAn insight the critic objects to is fixed or downgraded, never quietly removed.

• 01Nine of my eleven matched theirsA five-transcript pass validated against the full corpus.
• 02Five themes lived in transcripts I hadn’t readCopyright, political economy, the arts vocabulary gap, DMP templates. A full-corpus run has to find those.
• 03Twelve trapsPlaces the corpus invites a bad synthesis: one private-sector voice, one dissenter on deletion, a concept the researchers introduced that the participants didn’t.
• 04The codebook didn’t existThe dataset readme promised one; it was never deposited. Ground truth was built from the report plus the manual pass instead.

• 01Deterministic, in codeThe cited turn exists; the speaker is a participant, not the interviewer; the verbatim receipt appears in the turn; confidence matches the source count — high needs four participants and no counter-evidence. Free, instant, can’t hallucinate.
• 02Model-judgedThe turn supports the claim as stated; a dissenting participant is missing; two findings are merged; a theme present in three or more transcripts has no insight. The critic reads for these.

• 01Silence failsA critic with no verdict, a synthesis with no insights, an empty list — all hard failures. Every layer.
• 02ReceiptsEach citation carries a verbatim quote, checked against the transcript in code. Deterministic failures on round one went from three to zero. The synthesis writes more carefully when it knows the check exists.
• 03Confidence is arithmetic“Overconfident” was whack-a-mole until the threshold was a number in the config.
• 04Sources are computedNot asked for.

• 01Unsupported claims halved1.7 to 0.7 per four checked.
• 02Confidence errors went to zeroThe numeric threshold did that, not the model.
• 03Trap failures fell2.3 to 1.3.
• 04Coverage dropped eighteen points75% to 57%. Reuse, a core theme, was absent from every loop run and present in five of six others.
• 05Intake alone changed nothingThe map isn’t the value; the checking is.

• 01A recall ruleThe critic now receives the intake topic maps and must flag any topic present in three or more transcripts with no insight covering it. The intake step, which had made no measurable difference, became the input to the check that recovered coverage.
• 02No silent deletionThe reviser must list anything it drops, with a reason; a vanished insight without one is a failure.
• 03Profiles for the criticSector, career stage, method — with an instruction to look hardest at the outlier.

• 01Coverage recovered51% to 69%, against 75% for a single prompt.
• 02Reuse went from 0 of 3 runs to 3 of 3The recall rule, doing exactly what it was built for.
• 03Precision heldUnsupported claims unchanged at 0.7; confidence errors 0.7.
• 04Nothing was silently deletedIn three runs.
• 05The best single report of any condition in either evalNine of twelve themes, zero unsupported, zero miscalibrated.

Part 1 ended on a named failure, above: the model was taking one finding per turn and stopping, so themes cited constantly for one thing never surfaced as a claim in their own right. Eval 2 confirmed it — two ground-truth themes, T-11 (deletion as waste) and T-15 (method hierarchy), absent from all six reports while the turns carrying them were cited in all six for something else. That, plus a duplication problem, an outlier-dissent trap that kept failing, and two unanswered questions about the critic model and the iteration cap, became the v3 eval.

Eval 3 tested four conditions against the same five transcripts (33,870 words), the same ground truth, and condition C only: control, the frozen instrument unchanged; all-v3, control plus five candidate changes (second_finding, duplicate_insight, duplicate_receipt, critic_citation, dissent_at_intake); opus-critic, control with Opus 5 as the critic; cap5, control with max_iterations: 5. Ten runs — three each on control and all-v3, two each on opus-critic and cap5, chosen so the eval fit its $25–40 budget with a working margin. All ten valid; none crashed, none re-run. Scored blind by one rater, AI-assisted, four evidence spot-checks per report at fixed positions chosen before scoring began.

Ten pass/fail bars were written into the eval spec, with their Eval 2 baseline arithmetic beside them, before the first paid run.

These are decision rules, not significance tests: with three runs and a binary outcome, 2/3 against a 0/3 baseline is not significant at any conventional level (Fisher exact ≈ 0.4), and a one-point theme-sum move is inside the rater consistency Eval 2 recorded. Bars 1, 4 and 5 are worth acting on because the baseline is never rather than sometimes. Bars 2 and 3 are guard rails against a recall change that costs precision, not claims of improvement. Bars 6 and 7 rest on two runs, not three, and cannot report a rate.

second_finding asks, for each turn a report cites, whether that turn carries a distinct finding no insight uses. It fired 8, 8 and 9 times across the three all-v3 runs and nowhere else — 25 firings, all inside one condition. In iteration 1 of all three runs it named david:0022 and penni:0031 by turn id and said what second finding each carried. Both themes ended present as claims in all three reports, scored by a rater who did not know the rule had fired.

penni:0031 was cited in 10 of 10 reports and reached a claim in only the three all-v3 ones. Coverage rose without precision falling: all-v3 gained 1.8 theme points over control and 2.2 over Eval 2’s v2, on 5.3 more insights per report, with unsupported and miscalibrated both at zero. The other two changes that earned their place are both deterministic and cost nothing: duplicate_receipt fired 5 times across the three all-v3 runs and every instance was fixed before the final output; critic_citation caught the exact failure it was written for — the critic sending the reviser to an interviewer turn — and stripped it silently but reported it.

A bar can pass and prove nothing, and saying so is the finding.

dissent_at_intake. Bar 5 passed: P-03 passed in 3 of 3 all-v3 runs. So did 2 of 3 control runs, and 4 of 4 opus and cap5 runs — 9 of 10 reports overall. The rule cannot be credited when the condition without it fixed the same thing. Ruling: stays off. Re-test it against the closed baseline, or drop it; it costs intake output tokens for an effect this eval cannot see.

duplicate_insight. Bar 4 passed: zero near-duplicate pairs at the rater’s judgement across the three all-v3 runs. But the rule never fired, in any run, at its 0.55 threshold, and the rater found no pairs for it to fire on — Eval 2’s R4 duplication was not reproduced, so there was nothing to catch. It stays off for this release too, alongside dissent_at_intake, recorded as untested rather than shipped on faith: it is cheap and deterministic, and the failure it guards — over-splitting under missing_theme pressure — is a live risk now that second_finding is adding insights, which makes it a strong v4 candidate rather than a shipped rule this round.

The rule that did ship is a deviation from the spec worth recording, because it changed the cost: the spec asked for a “deterministic-plus-model pass after revise”; it was built instead as a deterministic critic rule — Jaccard over title and claim vocabulary, threshold 0.55 — that the existing reviser acts on like any other failure. That saves one Sonnet call per iteration, about $0.18 per run and $0.55 across the condition.

Both cap5 runs went the full five rounds, stop_reason: max_iterations, without ever reaching critic_pass or triggering no_progress. Rounds 4 and 5 cost about $1.10 and 7 minutes per run and bought +0.58 theme points on average, inside rater noise. critic_pass was reached in 0 of 10 runs, at cap 3 and at cap 5, and no_progress never fired. Every run stopped on the ceiling with objections outstanding — raising the cap does not converge the loop, it buys more rounds of the same. Eval 2’s open item, whether the cap is a ceiling or a stopping rule the loop reaches, is answered in the negative.

Opus as critic clears its own decision rule: coverage 8.67 → 10.25, +1.58, at $3.83 a run against a $5.00 ceiling. Its strongest report beat every report produced under any condition in any of the three evals. But the bar is running on two runs, and opus-critic’s two-run range overlaps control’s three-run range. Ruling: make it the documented option for a high-stakes run, not the default, and re-test at n=3 if it is ever proposed as the default.

Per run: control $2.23, all-v3 $2.51 (+13% for +1.8 theme points), opus-critic $3.83 (+72%), cap5 $3.31 (+48% for +0.58). Every condition came in under its projection.

Combined project API spend after Eval 3: $72.2047. Local spend $66.4209 across 42 runs/*/meta.json; hosted spend (part 3, on the Fly volume) $5.7838.

Eval 1

$13.3645

Eval 2

$6.8362

Eval 3

$28.4963

non-eval/dev

$17.7239 (discarded and development runs)

local total

$66.4209 · 42 runs

hosted (part 3)

$5.7838

combined

$72.2047

Source: docs/case-study-notes.md, 2026-09-06; Motif commit 341c544.