OpenAI’s TTS silently drops the tail of long inputs

Environment: gpt-4o-mini-tts and whisper-1 via the OpenAI HTTP API, Node.js 20, ffmpeg. Measured July–August 2026.

We produce MP3 audio for exam-prep books — listening sets, vocabulary drills, read-aloud tracks. It is generated, not recorded, and for the most part that works.

Then one afternoon a six-sentence listening passage came back five sentences long. The last sentence was simply not there. No error, no truncation warning, HTTP 200, a well-formed MP3 that ends cleanly on the wrong sentence.

Regenerating it fixed it. Later the same day it happened again on a different item. Two occurrences in one build, non-deterministic, in the middle of material where the final sentence is often exactly the sentence a comprehension question is about.

Why this class of bug is worse than it sounds

Nobody listens to twenty minutes of generated audio end to end before shipping it. That is the entire reason the audio is generated.

So the defect has to be caught by a machine, and the obvious machine — transcribe the output, diff it against the script — turns out to be unreliable in both directions.

Whisper’s two failure modes

Across long silences it hallucinates structure. Our listening files have a ten-second answering pause after each question. Transcribing a whole file reported that a question had been read twice, and that a number announcement was missing. Neither was true. The waveform showed one reading, and the announcement present. Both findings were artefacts of transcribing across the pauses.

On short utterances it clips the beginning. A three-second clip reading "Number one. Govern the city." came back without the number. Below roughly eight words, the transcription is not evidence of anything.

That is the awkward part: a check producing false positives and false negatives cannot be the only check. It hands you a list in which a real defect is indistinguishable from the noise.

Two checks, two modalities

So verification runs twice, in ways that fail differently.

Structure, from the waveform alone. Each listening item has a fixed shape: number announcement → passage → the word "Question." → the question → a ten-second pause. That shape is visible purely in where the silences fall. No transcription is involved, so none of whisper’s failure modes can reach it. This check also does the cutting: it is what splits a twenty-minute file into individual items.

Content, from the transcript — but per item, never per file. Only after the file has been cut does anything get transcribed. The ten-second pauses are gone by then, along with the reason whisper was confused.

Neither is sufficient. A finding from one is not a defect until the other confirms it — and a PASS from one does not clear the item either. In that build, two of the transcript’s complaints turned out to be artefacts of the pauses, while a separate real fault was confirmed on the waveform.

"""audit-audio.py — 焼き上がった音声を、台本と突き合わせて機械検査する。

リスニング3本を「番号アナウンス→本文→Question.→設問→10秒ポーズ」の1問ずつに切り分け、
  (1) 波形：各問が上の構造どおりか（無音の入り方だけで判定・文字起こしに依存しない）
  (2) 文字起こし：本文と設問が台本どおり読まれているか（読み落とし検出）
の2本立てで確かめる。

※ whisper は 10 秒の解答ポーズをまたぐと繰り返しや取りこぼしを起こすので、
   必ず1問ずつに切ってから文字起こしする。ファイル丸ごとの文字起こしは当てにならない。
"""

Catching it at generation time

Auditing afterwards is the safety net. The cheaper place to deal with a dropped tail is immediately after the request, while the script for that one utterance is still in hand.

// 読み落としは「長い入力の末尾」で起きる。短い発話は文字起こし側が当てにならない
// （3秒の "Number one. govern the city" で whisper が頭を落とす）ので検証しない。
const MIN_VERIFY_WORDS = 8;

// 焼く → 聞き直す → 一致しなければ焼き直す（最大3回）
export async function ttsVerified(key, { text, voice, instr, outFile }) {
  const exp = words(text);
  for (let a = 0; a < 3; a++) {
    await speak(key, { text, voice, instr, outFile });
    if (exp.length < MIN_VERIFY_WORDS) return;   // 短い発話・番号アナウンス・日本語は対象外
    const heard = await transcribe(key, outFile);
    if (heard === null) return;                  // 検証できないときは素通し（生成は止めない）

    const got = words(heard);
    const pool = [...got];
    let hit = 0;
    for (const w of exp) { const i = pool.indexOf(w); if (i !== -1) { pool.splice(i, 1); hit++; } }

    const cov = hit / exp.length;
    const tailOk = got.slice(-6).includes(exp[exp.length - 1]);
    if (cov >= 0.8 && tailOk) return;
    console.warn(`  ↻ 読み落としを検出（一致 ${(cov * 100).toFixed(0)}%${tailOk ? '' : '・末尾欠落'}）`);
  }
  console.error(`✗ 3回焼き直しても台本どおりにならない: "${text}"`); process.exit(1);
}

Three decisions in there are worth pulling out.

Coverage alone is not enough. A six-sentence passage missing its final sentence still scores well above 80% word overlap. So tailOk is checked separately: the last content word of the script must appear among the last six words heard. That single condition is what actually catches this bug; the coverage ratio catches everything else.

Numbers are excluded from comparison. sixteen and 16 are the same reading and different strings, and normalising every numeric form is more work than it is worth. The words() helper drops digits and number words before comparing.

A failed verification does not stop the build. If the transcription request itself fails, the utterance passes through. Verification that can halt a twenty-minute render because a second API call timed out is worse than no verification — it gets switched off. Only a positive detection of a mismatch triggers a re-take, and only three times before the script gives up loudly.

The other quirk: long instructions stretch the vowels

Same model, unrelated symptom, found while making a Japanese track. The instructions field changes pace far more than it changes anything else — and it does so in the opposite direction from what the words say.

The same one-word Japanese line, すみません。, generated under two instructions:

The first version audibly drags. The vowels stretch until the word arrives as something closer to suuumiiimaaasen. Writing "slowly" or "Do not rush" makes it worse, not better. Leaving pace unspecified produces the most natural result.

Three things follow from this:

• Shorter inputs show it more strongly. The same instruction on a twelve-character sentence yields 4.3–5.2 chars/sec — nearly normal.
• It varies between takes. Identical input, identical instruction: 2.30 s and 2.76 s on consecutive calls. Fixing the wording reduces the problem, it does not remove it.
• Therefore, measure the output. We count characters excluding punctuation, divide by duration, and re-take anything below a floor of 4.0 chars/sec, up to three attempts, logging the rate for every segment. A natural range for Japanese sits around 4.2–6.0.

And one small thing that costs an hour if you don’t know it: each generated segment carries roughly half a second of silence on the end. Concatenate them and your carefully chosen inter-line gaps are all wrong. Running silenceremove on both ends first took one track from 60.0 seconds to 49.1.

The rule this leaves us with

For generated audio, a single transcription is not a verification.

Transcription checks content. The waveform checks structure. Duration checks pace. Each is blind in a way the others are not, and the one thing you can be certain of is that the model will fail silently — HTTP 200, valid MP3, wrong contents — rather than loudly.

That is a general property of generative APIs, not a quirk of this one. The failure mode is not an error code. It is a plausible output.

Next: running a local Japanese TTS engine on WSL to produce a podcast, and why the reading of a word gets checked in kana before anything is synthesised.