The ETOT method
Every ETOT tool follows this. It's why the case studies can be compared with each other. Written after Motif, September 2026. We'll revise it as later tools teach us things, and we'll date the revisions.
We write down what failure looks like before we start.
principle
A tool is a claim: this works better than what you're doing now. Motif's claim: this loop makes fewer unsupported findings than a single prompt. The method exists to make that claim testable before we build the tool, measurable after, and honest in the write-up. We write down what failure would look like before we start. If that's what we get, we don't ship.
why it is public
So a claim about a tool can be checked by the people who did not build it. The phases below are how every ETOT tool gets built and written up; the case studies are comparable because of them.
00
Frame
½–1 day
docs/brief.md: problem, what we're building, what "agentic" (or the mechanism) means here, research questions (3–4), hypothesis with a number in it, the material we'll test on, evaluation design, success criteria, out of scope, stack, deliverables.docs/log.md: dated, three lines per session — tried / happened / next.docs/case-study-notes.md: started on day one. One line per moment worth writing up. This file is the case-study outline.- Working name is fine. We check the product name for collisions with existing products (Figma Community, npm/PyPI, domains, USPTO) before it touches a URL.
01
Ground truth
1–2 days
- Real material — transcripts, design files, whatever the tool reads — ethically cleared and licensed for publication.
- Whoever builds the tool does the task by hand first, timed, blind to any AI output. That hand-done version is both the baseline and the ground truth — the known-correct answer the tool gets scored against.
- A checklist of themes and criteria, and a trap list — the places where the material invites a bad output. We write the scoring rubric before any code.
- We log what's missing rather than hide it — a codebook that was promised and turns out not to exist, say.
02
Build v1
3–5 days
- On
core/. Three functions: produce, check, revise. Stop conditions written out and logged. - Rules in config, in plain language. Anything that can be checked mechanically is checked that way. The model is asked only what only a reader can answer.
- Every output carries its evidence and its unresolved objections. A check that returns no verdict has failed, at every layer.
- Tests that run without calling the API must still run the real code, with a stand-in for the model.
- Mark the exact version with a git tag before measuring.
03
Evaluate
2–4 days
- Conditions: baseline (single prompt), each mechanism added in turn, full tool. Three runs each.
scripts/eval_pack.pystrips the labels, so the scorer can't tell which result came from which condition. The key stays sealed until scoring is done.- Score coverage, trap failures, spot-checked support, and calibration — whether the stated confidence matches what's actually true — plus time, tokens and cost.
- Report what got worse. A null result — a test that found no improvement — is the most credible thing in the write-up.
- If a fix follows, re-tag and re-run only the affected conditions. Say so in the write-up.
04
Package
1–2 days
- README with a five-minute install that's been tried by the builder as a stranger. One-command CLI.
pyproject, license, data attribution, sample output, CONTRIBUTING. - Public repo. Bad runs kept.
05
Case study
2–3 days
- Built on the ETOT chassis (
motif-build.py): spec card, one accent, serif display, mono receipts, sections as dots, figures generated from the record, honest ledger at the end. - Every number and quote on the page traces to a file in the repo. The hero exhibit and any "trace" are pulled from run logs, not reconstructed.
- Verified against the logs before publishing. The paragraph describing the method says how scoring was done (human, AI-assisted, blind or not).
- Thumbnail: one computational idea, one color, no product screenshot.
06
Surface, then part 2
- The engine doesn't care what it's plugged into. The surfaces — a FigJam plugin, an MCP server, a command line — are thin wrappers that call the same functions.
- We write a "part 2" case study only when there's a real interface and at least one measurable outcome with users.
07
Series discipline
- One core, many tools. New tools reuse
core/unchanged. - One product-name pattern. What the names have in common is the mechanism — verification — not the task.
- Same rubric shape across tools so results can sit in one table.