Issues recognized from each round

medium Greedy-decoding artifact

Eval generations use pure greedy decoding, no repetition control → textbook repetition loops ("the standard of proof … is the standard of proof …"). Partly a decoding pathology, not pure capability.

Fix: add no_repeat_ngram_size=3 + repetition_penalty≈1.3. Comparability-safe (doesn't change weights). Run first as a diagnostic.

high Closed-book QA is unlearnable at 125M

80% of QA is closed-book — recall a long-tail document fact from a single exposure, effectively impossible for a 125M model. The judge set is drawn mostly from these, so we grade the model almost entirely on the one thing it can't learn.

Fix: rebalance toward grounded tasks (context in the prompt) + general knowledge. Breaks the curve → restart as v2 (cheap now).

minor Rigor

Decontamination skipped (HF 503) — retry to cache the eval n-grams. Consider a stronger judge model (gemini-3-flash-preview).

high Data volume is a weak, diminishing lever

2× data → judge +0.18 (1.32→1.50), ppl −0.60 (near-flat). Extrapolated, a usable ~3/5 judge would need ~8+ more days, and it won't be linear. The mix, not volume, is the constraint — Day-1's hypothesis, now trend-backed.

Direction: evidence now favors the v2 pivot over scaling the current mix.

medium Forgetting grows monotonically with data

Retention rises every round: +6.1% → +9.5% (~+0.37 ppl per +1,000 pairs). By ~Day 5 that's "notable," ~Day 8 "severe" — scaling trades growing forgetting for shrinking judge gains.

Fix (if continuing): mix a small fraction of raw pretraining windows into SFT (replay) and/or lower the LR.

high Measurement blind spot

The judge tests mostly closed-book recall — the unlearnable task — so 1.50 is dominated by the model's worst skill while learnable grounded tasks are under-measured. (Day 2's grounded severability answer became correct; the two recall answers stayed degenerate.)

Fix: report the judge per mode (grounded vs recall). Cheap; likely shows the model is decent at what it can actually learn.

decisive Plateau CONFIRMED — stop rule fired

Judge flat across three consecutive rounds: 1.50 → 1.46 → 1.50 (2,000 → 4,000 pairs). Deltas by round: +0.32, +0.18, −0.04, +0.04. All the gain arrived by Day 2; 4× the Day-1 data bought +0.18 total and nothing since.

Action: stop scaling v1. Further rounds on this mix are a known negative result, not an open question.

high Forgetting has doubled

Retention rises monotonically every round: +6.1% → +9.5% → +11.0% → +12.2% — doubled since Day 1, while the judge gained nothing since Day 2. Each round past Day 2 is pure downside.

Extrapolated to 10k: judge ≈ still ~1.5, retention ≈ +18–20% (severe). A strictly worse model, for money and hours.

medium Perplexity is actively misleading

SFT-eval ppl improved every single round (8.60 → 8.00 → 7.70 → 7.51) while the judge sat flat. Four rounds of clean evidence that perplexity measures distribution fit, not quality.

Fix: never headline SFT-eval ppl; keep it as a diagnostic only.

decisive Plateau holds at 10× scale

Judge: 1.00 · 1.32 · 1.50 · 1.46 · 1.50 · 1.52 · 1.54 · 1.60 · 1.60 · 1.54 · 1.54. All gain arrived by round 2. From 2k → 10k (5× more data) the judge moved +0.04, wandering 1.46–1.60 and ending where it started.

Predicted before the run: judge ≈1.5, retention ≈+18–20%. Realized: judge 1.54, retention +16.3%. The plateau model called it.

decisive Forgetting tripled — the only real effect

Retention rose monotonically in all ten rounds: 11.35 → 13.20 (+16.3%), nearly tripling from +6.1%. Data volume had a large, consistent effect on exactly one metric — the one we didn't want.

Every round past round 2 was pure cost.

high Quality regressed at scale

The "standard of proof" probe was correct at rounds 3–9 ("preponderance of the evidence") — seven straight rounds — then regressed to circular nonsense at round 10. More data made a previously-correct answer worse — drift overwriting earlier wins.

Verdict: v1 CLOSED. Do not run round 11+.