Environment: Claude Code (Opus), Node.js 20, macOS/WSL. Everything below runs from a single git repository.
We publish English exam-prep books for the Eiken test — eight grade levels, twenty-odd titles so far, each shipping as a print paperback, a Kindle file, and sometimes an MP3 audio set. For the last stretch of that work the drafting has been done by Claude Code.
Not "done by AI" in the way that phrase usually gets used. The model does not sit in a chat window and get asked for a chapter. It runs inside a repository that constrains what it is allowed to look at, what it is allowed to write, and what has to pass before anything moves forward.
This post is the map of that setup. The specific mechanisms — the vocabulary allow-list, the pre-split word pools, the two-context review gate — each need their own post. Here I want to show the shape.
The problem that forced the structure
A language model will write you a Grade 3 fill-in-the-blank exercise that uses Grade 1 vocabulary, and it will sound completely certain while doing it.
That is the whole problem in one sentence. "Difficulty" feels like something a model should be able to judge. It reads like a soft, fuzzy, language-shaped property — exactly the sort of thing an LLM is good at. It isn’t. For a standardized test, difficulty is not a vibe. It is a published word list. A word is either on the Grade 3 list or it is not, and if it is not, the exercise is broken no matter how natural it sounds.
The failure is nasty because it is invisible at read time. A fluent sentence with one over-level word reads fine to a human reviewer skimming for typos. You find it later, in print, in a book a child has already bought.
So the rule the whole repository is built around is this: the model is never asked to judge level. It is handed the list, by path, and told to choose from inside it. Everything else is scaffolding to make that rule enforceable.
Four layers, one repository
specs/ grade-by-grade exam specifications (vocabulary range,
sentence length, question formats, CEFR mapping)
style/ house style — voice, how much a explanation explains,
Japanese translation policy, ruby text rules
knowledge/ what twenty books taught us: reader feedback,
past-question corpora, reference word lists
.claude/agents/ the four subagents
.claude/skills/ procedural knowledge (cover design, KDP copy, A+ banners)
.claude/hooks/ automatic gates that fire on save
scripts/checks/ 25 verification scripts + a combined runner
templates/ chapter, mock-exam, answer-key skeletons
publishing/ per-format submission rules (paperback / Kindle / audio)
books/<title>/ the actual working directory for one bookThe top three directories are read-only ground truth. Nothing generated is allowed to contradict them, and — this matters — nothing gets written into them by a subagent summarizing something it read. When a specification changes, a human reads the source PDF and transcribes it. We learned that one the hard way: an exploration agent reported a question structure confidently, the report went straight into specs/, and the spec was wrong for weeks.
The four subagents
Each one is defined by what it may not do. That turns out to be the load-bearing part of the definition.
| Agent | Job | Writes to | Forbidden |
|---|---|---|---|
strategist |
Decide whether a book should exist, for whom, and how it differs | strategy/, brief.md |
Planning chapters. Writing prose. Recommending a roadmap before being asked |
planner |
Chapter structure, topic allocation, question counts, difficulty curve, generation order | plan/ |
Writing a single line of final text — no answers, no example sentences, no explanations |
generator |
The actual manuscript, from the plan | chapters/ |
Working without a plan. Deviating from it. If the plan is vague, it stops and pushes back |
evaluator |
Check the manuscript against specs, style, and publishing rules | reviews/ |
Editing anything. It reports; it never fixes |
The order is enforced socially rather than technically — you can call them in any order — but calling planner before strategist reliably wastes a day, and calling generator before specs/ exists for that grade means level drift cannot be detected at all. So the repository’s instructions say so in as many words, and they say why.
evaluator has one more constraint worth calling out: it is told not to praise. A review that opens with "this chapter is well-structured and engaging" has spent its first paragraph doing nothing. Every finding has to carry evidence, a location, a quotation, and a verdict of PASS, FAIL, or WARN.
The rule that does the most work
The writer never grades its own work.
generator and evaluator run in separate contexts. Not separate prompts in the same conversation — separate contexts, with different tool permissions. generator has Write and Edit. evaluator has Bash and can write only into reviews/.
This sounds like bureaucracy until you watch what happens without it. Ask a model to review something it just produced and it will find small, safe problems. It has already committed to the choices that matter. The over-level word it picked in question 7 was picked because, in that context, it looked right — and re-reading the same context reproduces the same judgment. A fresh context with the word list in hand does not have that attachment.
What the machines check
Before any chapter moves on, scripts/checks/run-all.mjs runs against it. There are twenty-five checks; the ones that earn their keep are unglamorous:
vocab-leak— n-gram overlap against a corpus of past exam papers, because "inspired by the format" must never become "copied the text"level-drift/choices-in-list— every word in every question and every distractor must come from the pool assigned to that chapteranswer-uniqueness— a four-choice question where two options are defensible is a broken questionanswer-distribution— models love answer 3non-english-in-passage— Japanese characters leaking into English passagesphrasal-verb-pronoun— catches "pick up me"passage-wordcount/choice-wordcount— the spec gives ranges; ranges are checkable
Here is vocab-leak on a real chapter:
$ node scripts/checks/check-vocab-leak.mjs books/p1-drill/chapters/ch01.md
✓ PASS 8-gram (copyright-risk) matches: 0
✓ PASS 5-gram matches: 3 (leak ratio 0.06%)
JSON report: {
"file": "books/p1-drill/chapters/ch01.md",
"chapter_tokens": 4709,
"chapter_5grams": 4705,
"matches_5gram": 3,
"matches_8gram": 0,
"leak_ratio_5gram": 0.0006376195536663124,
"sampled_locations": [
{ "ngram": "on the night of the", "source": "2023-3-1ji" },
{ "ngram": "which of the following is", "source": "2025-1" },
{ "ngram": "of the following is true", "source": "2025-3" }
]Three five-word overlaps, all of them unavoidable test-paper boilerplate. Zero eight-word overlaps, which is the number that would actually mean something. The check does not tell me the chapter is good. It tells me one specific way it is not bad, with a number I can put in a report.
That distinction is the point. A machine check that says PASS is worth having precisely because it is narrow.
Per chapter, not per book
The gate runs after every single chapter, before the next one starts. Machine checks first, then evaluator for the things a script cannot see — whether a distractor is plausible, whether a translation reads like Japanese rather than like translated English, whether the passage actually supports the answer.
Batching this to the end is the obvious optimization and it is a trap. A systematic mistake found at chapter 10 is not one fix, it is ten, plus re-recorded audio, plus a re-typeset PDF. The temptation to skip the gate arrives exactly when skipping it is most expensive: a tight deadline, three chapters in a row going well, a sense of momentum.
What this does not do
It does not decide what to write. That is a conversation with a human, and strategist exists to have it, not to settle it.
It does not lay out a page. Typesetting is HTML and WeasyPrint, illustrations come from an image model, and both need human eyes at the end — measurement and looking are different things, and a layout that measures correctly can still look wrong.
And it does not remove the need to hand the model its sources. The single most common way this setup fails is a human writing "use Grade 5 vocabulary" instead of "use only words in jlpt5/vocab/*.csv". The first is an invitation to guess. The second is a constraint.
Next posts in this series will go into the parts that took the longest to get right: the vocabulary allow-list and the machine checks behind it, and why we now split the word list into per-chapter pools before generation instead of de-duplicating afterwards.
