The writer never grades its own work

Environment: Claude Code (Opus), Node.js 20. Two subagents defined as Markdown files with YAML front matter.

Ask a model to review something it just wrote and it will find small, safe problems. Indentation. A repeated word. Something in the third paragraph that could be phrased more clearly.

It will not find the thing that is actually wrong, because the thing that is actually wrong was a decision, and it already made that decision once. Re-reading the same context reproduces the same judgment. The sentence that looked right when it was written looks right when it is read back, for exactly the same reasons.

So in our book pipeline the agent that writes a chapter is never the agent that grades it. Not a second prompt in the same conversation — a separate context, with different tools.

Different tools, not different instructions

Both agents are Markdown files. The front matter is where the separation actually lives:

---
name: generator
description: planner が書いた計画文書に基づいて、章本文・問題・選択肢・解説を Markdown で生成する。
tools: Read, Grep, Glob, Write, Edit
---

---
name: evaluator
description: generator が書いた原稿を specs / style / publishing のルールに照らして検証し、
             reviews/ にレビュー報告を書き出す。修正は一切行わない（指摘のみ）。
tools: Read, Grep, Glob, Write, Bash
---

Look at what is missing from each list.

generator has Edit. It can change the manuscript. It has no Bash — it cannot run the checks that judge it, cannot see them pass, cannot iterate against them.

evaluator has Bash. Running the verification scripts is its first job. It has no Edit at all. It can write, but only ever a new file, and the instructions restrict that to books/<title>/reviews/.

This matters more than the prose instructions do. An agent told "please don’t edit the manuscript" will, at some point, edit the manuscript — usually while helpfully fixing a typo it noticed on the way to something else. An agent without the tool cannot.

And the moment a reviewer starts fixing things, two things are lost. The record of what was wrong, which is the only artefact anyone can learn from. And the boundary: a reviewer who fixes is a second writer, with the same attachment to its own choices that the first one had.

The reviewer is told not to praise

From evaluator.md, the section headed 絶対に守ること — "must not be broken":

2. Do not write vague impressions. Not "somehow unnatural" but "the X on line ○ is outside the vocabulary range in specs/level-2.md". Always give the evidence, the location, and the quotation.

3. State a verdict. Every item gets PASS, FAIL or WARN.

5. Do not praise. Listing good points is unnecessary. Report only problems. If there are none, write "ALL PASS" and nothing else.

The no-praise rule looks petty and is not. A review that opens with "this chapter is well-structured and engaging" has spent its opening doing nothing, and — worse — it has established a frame in which the findings that follow are exceptions to a good chapter. Removing the option to be encouraging removes a whole register of hedging along with it.

What it looks like when it works

Here is the first chapter of a Grade Pre-1 drill book. The machine checks ran first, and all eight passed. Then the evaluator read the file.

target: books/p1-drill/chapters/ch01.md
reviewed_at: 2026-04-26T07:30:09Z
reviewer: evaluator (LLM Phase 2)
phase1_machine_verify: books/p1-drill/reviews/verify-2026-04-26T07-22-05-473Z.md
verdict: NEEDS_REVISION

Summary: 3 FAIL, 5 WARN, 14 PASS. The first FAIL:

All four reading passages were over length. One of them by a fifth.

Nothing in the eight machine checks was looking at passage length. Nothing in the manuscript looked wrong either — a 631-word passage reads as a perfectly good 631-word passage. Every sentence in it justified itself at the moment it was written, which is precisely why the agent that wrote them was never going to be the one to notice.

The review didn’t stop at the number. It said which sentences to keep, because they are the evidence for the comprehension questions, and which to cut — the flock of birds, the details of the brick terraces, the scene in the room above the pub. Then:

Impact: this is the template chapter. If this passage length propagates into ch02–ch10, all forty passages will fall outside spec and the book stops matching the real exam.

That framing is the reason the gate runs per chapter rather than per book. Chapter 1 sets a shape. A defect in it is not one defect.

Findings become checks

The review is dated 26 April. check-passage-wordcount.mjs was committed to the repository on 27 April.

// check-passage-wordcount.mjs — 大問 2A/2B/3A/3B 本文の語数が specs §6 範囲内か検証。
//
// 計測：whitespace split / alpha tokens / word-boundary の 3 法でクロスチェック。
// 3 法すべてが範囲内なら PASS。1 つでも外れたら FAIL。
//
// 範囲（specs §6 / passage-word-counts.json 実測）：
//   p1: 2A 245-260 / 2B 245-260 / 3A 405-420 / 3B 510-525

Three independent counting methods, because word counting is less obvious than it looks and a disagreement between methods is itself information.

This is the loop that makes the whole arrangement worth its cost. A judgment made by a reviewer with a fresh context is expensive and slow. A script is cheap and instant. So every finding that can be reduced to a rule gets reduced to one, and the expensive reviewer stops having to look for that class of problem ever again.

Nine chapters later, passage length was no longer something anyone thought about. It was a line in the runner’s output.

The order, and why it is that order

For every chapter, before the next chapter starts:

1. node scripts/checks/run-all.mjs <chapter.md> — machine checks. Vocabulary pool membership, n-gram overlap against past exam papers, answer distribution, passage and choice word counts, phrasal-verb pronoun placement, stray Japanese in English passages.
2. evaluator — the things a script cannot see. Whether two options are both defensible. Whether a distractor is plausible or merely wrong. Whether the Japanese translation reads like Japanese or like translated English. Whether the passage actually supports the answer the key claims.
3. FAIL at either stage → back to generator with the specific findings → through the gate again.

Machines first, because they are free and because there is no point spending a reviewer’s attention on something a regular expression can settle.

Batching this to the end of the book is the obvious optimisation, and it is a trap. It is also the one everybody reaches for under deadline pressure — three chapters have gone well, momentum feels good, and the gate is the only thing standing between you and chapter four. The repository’s instructions say so in as many words, because writing it down was cheaper than relearning it.

What this does not solve

The evaluator is a language model. It misses things, and it occasionally reports a finding that dissolves on inspection. The review quoted above contains one: it flags a lemma overlap with a past exam paper as a WARN, then notes that the planner had already checked that same word and concluded otherwise. Two agents, two conclusions, and a human still has to decide.

That is fine. The claim is not that the reviewer is right. The claim is that it is differently wrong — wrong in ways uncorrelated with the writer’s errors, which is the only property that makes a second pass worth anything at all.

A reviewer that shares the writer’s context shares the writer’s blind spots, and produces a green tick that means nothing.

Next in this series: what happened when we asked OpenAI’s TTS to read a long script, and it silently stopped early.