The ETOT method
Every ETOT tool follows this; every case study is comparable because of it. Written after Motif, September 2026. It will be revised as later tools teach us things, and the revisions will be dated.
Nothing ships without a number that could have said no.
principle
A tool is a claim about the world ("this loop makes fewer unsupported findings than a prompt"). The method exists to make that claim testable before the tool is built, measurable after, and honest in the write-up. Nothing ships without a number that could have said no.
why it is public
So a claim about a tool can be checked by the people who did not build it. The phases below are how every ETOT tool gets built and written up; the case studies are comparable because of them.
00
Frame
½–1 day
docs/brief.md: problem, what we're building, what "agentic" (or the mechanism) means here, research questions (3–4), hypothesis with a number in it, corpus, evaluation design, success criteria, out of scope, stack, deliverables.docs/log.md: dated, three lines per session — tried / happened / next.docs/case-study-notes.md: started on day one. One line per moment worth writing up. This file is the case-study outline.- Working name is fine. Product name gets checked for category collisions (Figma Community, npm/PyPI, domains, USPTO) before it touches a URL.
01
Ground truth
1–2 days
- Real corpus or real artefacts, ethically cleared, licensed for publication.
- The builder does the task by hand first, timed, blind to any AI output. This is the baseline and validation.
- A theme/criteria checklist and a trap list: where the material invites a bad output. Scoring rubric written before any code.
- Missing pieces (a promised codebook that doesn't exist) are logged, not hidden.
02
Build v1
3–5 days
- On
core/. Three callables: produce, check, revise. Explicit stop conditions, logged. - Rules in config, in plain language. Deterministic where possible; model-judged only for what only a reader can answer.
- Every output carries its evidence and its unresolved objections. Silence is never approval — at every layer.
- Cheap offline tests must exercise the real code path with a stubbed model.
- Freeze with a git tag before measuring.
03
Evaluate
2–4 days
- Conditions: baseline (single prompt), each mechanism added in turn, full tool. Three runs each.
- Blind pack via
scripts/eval_pack.py; key unopened until scoring is complete. - Score coverage, trap failures, spot-checked support, calibration, plus time, tokens, cost.
- Report what got worse. A null result is the most credible thing in the write-up.
- If a fix follows, re-tag and re-run only the affected conditions; say so.
04
Package
1–2 days
- README with a five-minute install that's been tried by the builder as a stranger. One-command CLI.
pyproject, license, data attribution, sample output, CONTRIBUTING. - Public repo. Bad runs kept.
05
Case study
2–3 days
- Built on the ETOT chassis (
motif-build.py): spec card, one accent, serif display, mono receipts, sections as dots, figures generated from the record, honest ledger at the end. - Every number and quote on the page traces to a file in the repo. The hero exhibit and any "trace" are pulled from run logs, not reconstructed.
- Verified against the logs before publishing. The methods paragraph states how scoring was done (human, AI-assisted, blind or not).
- Thumbnail: one computational idea, one colour, no product screenshot.
06
Surface, then part 2
- The engine is platform-agnostic. Surfaces (FigJam plugin, MCP server, CLI) are thin and call the same functions.
- A "part 2" case study is written only when there's a real interface and at least one measurable outcome with users.
07
Series discipline
- One core, N tools. New tools reuse
core/unchanged. - One product-name pattern; the family word is the mechanism (verification), not the task.
- Same rubric shape across tools so results can sit in one table.