Don’t let the model judge difficulty: a vocabulary allow-list, checked by machine

Environment: Node.js 20, Claude Code (Opus). The checks are plain ES modules in the same repository as the manuscripts — no service, no database, no API calls.

In the previous post I described the four-subagent setup we use to draft exam-prep books, and said that the whole thing is built around one rule: the model is never asked to judge level. It is handed the list, by path, and told to choose from inside it.

This post is that rule in code. What the list actually is, how it gets handed over, and — the part that took longest to get right — what a machine can and cannot verify afterwards.

The list is a file, not an idea

For each exam grade we keep a reference word list as JSON:

{
  "count": 2299,
  "items": [
    { "id": 1, "english": "as a result", "meaning": "結果として，その結果",
      "pos": "接続詞", "is_phrase": true },
    { "id": 2, "english": "however",     "meaning": "しかしながら，どんなに～でも",
      "pos": "副詞",   "is_phrase": false },
    { "id": 3, "english": "environment", "meaning": "環境，状況",
      "pos": "名詞",   "is_phrase": false },
    { "id": 4, "english": "reduce",      "meaning": "を減らす，縮小する",
      "pos": "動詞",   "is_phrase": false }
  ]
}

Four fields do real work later. english is the lookup key. pos is what lets us split the list into balanced chunks instead of handing one chapter all the nouns. is_phrase marks multi-word entries so they can be matched as units rather than as loose words. meaning is there because the Japanese gloss belongs with the entry, not in a second file that drifts out of sync with the first.

Alongside it sits idioms.json, and a corpus extracted from past exam papers. Together they define "in range" for that grade.

The important property is not the format. It is that the list is a path. A prompt can point at it.

Handing it over

Here is the difference the whole system rests on:

✗  "Write ten fill-in-the-blank questions using Grade 2 vocabulary."

✓  "Write ten fill-in-the-blank questions. Every content word in the stem,
    the answer, and all three distractors must come from
    knowledge/2q/reference/vocabs.json. Do not use a word that is not in
    that file."

The first sentence asks the model to recall what Grade 2 vocabulary is. It will answer confidently, in fluent English, and it will be wrong often enough to matter — not wildly wrong, just wrong at the edges, which is the expensive kind. A word two grades too hard does not look like an error. It looks like a good sentence.

The second sentence asks the model to filter a set it can read. That is a different task, and it is one the model is reliably good at.

The same applies to distractors, which is less obvious. It is tempting to let the wrong answers come from anywhere — they are wrong anyway. But a distractor a learner cannot even read is not a distractor. It is a giveaway.

Two checks, doing two different jobs

Once the chapter exists, two scripts run on it. They look similar and they are not.

check-choices-in-list — binary

Every multiple-choice option in the chapter has to be in the word pool assigned to that chapter. Answers and distractors alike. Inflected forms are accepted; function words are ignored. Anything else is a violation, and one violation fails the file.

$ node scripts/checks/check-choices-in-list.mjs books/p1-drill/chapters/ch01.md
✓ PASS  [p1] 章 ch01 pool (4099 items incl. inflections) vs 18 問 × 4 = 72 slots; pool 外 0 件

JSON report: {
  "file": "books/p1-drill/chapters/ch01.md",
  "level": "p1",
  "chapter_id": "ch01",
  "pool_size": 4099,
  "targeted_problems": 18,
  "total_choice_slots": 72,
  "fails_count": 0,
  "fails": []
}

The scripts report in Japanese, since that is the language of the people reading them: 18 問 × 4 = 72 slots, pool 外 0 件 — nothing outside the pool. Seventy-two options, all inside a pool of 4,099 accepted forms. This check can be binary because the thing it measures is genuinely binary. A four-option question has exactly four options; each one either appears in the pool or does not. There is no judgment call in the middle.

check-level-drift — a number, not a verdict

The chapter’s prose is a different matter. Reading passages contain proper nouns, place names, the vocabulary of whatever the passage is about. You cannot demand that every token in a passage about pottery appear on a general word list.

So this check reports coverage:

$ node scripts/checks/check-level-drift.mjs books/p1-drill/chapters/ch01.md
✓ PASS  [p1] coverage 95.6%  (1092/1142 in range, 50 out)

--- Out-of-range words (first 50) ---
accompanist, avoidance, compact, crab, curriculum, daylight, dialog, difficult,
difficulty, final, fines, finger, fourteen, gap, gaps, hayashi, heartbeat,
impulsive, inequalities, inventive, irregular, jars, keepers, lid, looms,
miners, ninety, officers, outline, overturned, pause, pauses, pianist,
placeholder, priest, rhythm, rhythms, schoolhouses, separate, sixteen, slept,
snail, snails, spinning, squirt, uneven, unfinished, unscrew, upward, weavers

Look at that list honestly. hayashi is a character’s surname. sixteen and ninety are numbers. difficult, final, finger, separate are words no learner at this level is troubled by — they are simply absent from the particular reference list this grade draws on. looms appears eight times and weavers once, because one of the passages is about weaving.

Almost none of these are the problem the check is looking for. That is fine, and it is the point. The output is a triage list for a human, not an accusation. It takes about thirty seconds to scan fifty words and notice that none of them is a genuinely over-level abstraction.

The thresholds are:

Those numbers were not chosen up front. The original cut-off was 85%, and it kept failing chapters that were fine — a chapter on culture at 85.8%, one on science at 85.6%, one on economics at 84.7%. All three were dense with subject vocabulary that no general word list would ever contain. Failing them meant a human overruling the check, and a check that gets overruled routinely is a check that gets ignored entirely.

Dropping it to 80% made the gate honest: below one word in five unaccounted for, something is actually wrong.

The unglamorous half: not crying wolf

The first version of the drift check flagged charging, comfortably, and people's as out of range. All three stems were on the list. The checker was comparing surface forms.

The fix is a stack of morphological fallbacks — the least interesting code in the repository and among the most load-bearing:

function inAllowed(w) {
  if (allowed.has(w)) return true;
  const tries = [];
  if (w.endsWith("'s"))  tries.push(w.slice(0, -2));          // people's → people
  if (w.endsWith('ies')) tries.push(w.slice(0, -3) + 'y');    // studies  → study
  if (w.endsWith('s'))   tries.push(w.slice(0, -1));
  if (w.endsWith('ing')) {
    const stem = w.slice(0, -3);
    tries.push(stem, stem + 'e');                             // charging → charge
    if (/([a-z])\1$/.test(stem)) tries.push(stem.slice(0, -1)); // running → run
  }
  if (w.endsWith('ly')) {
    tries.push(w.slice(0, -2));                               // quickly     → quick
    tries.push(w.slice(0, -1) + 'e');                         // comfortably → comfortable
  }
  // ... comparatives, superlatives, past forms
  return tries.some((t) => t.length >= 2 && allowed.has(t));
}

No stemmer library, no lemmatiser. A stemmer would also collapse derivations we deliberately want to keep separate — a -tion nominalisation is a different lexeme with a different difficulty, and treating it as its stem would quietly widen the gate.

Which is the real hazard here. There is a small whitelist of instructional terms that appear in Japanese-language explanations rather than in English passages, and the comment above it in the source reads, in effect: only add terms that never appear inside an English sample. Do not add general words to make the gate pass. That temptation arrives every single time a check fails on a deadline. Widening the definition of "allowed" until the check goes green produces a green check and an unchanged book.

What this buys

Each check runs in a second or two, against one file, before the next chapter starts. Neither one tells us a chapter is good. check-choices-in-list tells us that no option in any question came from outside the assigned pool. check-level-drift tells us what fraction of the prose is accounted for, and hands over the remainder to look at.

Both are narrow, and narrow is what makes them worth having. A check that claimed to evaluate "difficulty" would be exactly the thing this whole design exists to avoid — a confident judgment with nothing underneath it.

The pool that check-choices-in-list validates against does not exist by accident either. The next post is about why we now split the word list into per-chapter pools before generation, with a fixed shuffle seed, instead of de-duplicating after the fact.