Evidence: the "5% real data" claim

The original claim

5% of reality can keep an AI loop sane.

This week everyone argues about who wrote the text (thanks Anthropic for strong move).

Wrong question... the problem is qiute measured one. Train AI on AI output with nothing real underneath - you will get degradation on every generation: variance shrinks, rare events disappear, everything drifts toward the average. And thus we get "Slop".

AI slop feels slopish when it's language-wise logical but wraps and repeats some facts in a nice package (the gold plating, we will get there).

And IF one anchor the same loop to real data tadam - it heals! The fraction of reality needed is small (surprisingly one study puts it near 5%). Add verification of what goes in and the loop stabilizes.

So slop was never an authorship problem. Unsourced human text is slop too, you can easily find zillions of books written pre-ai which are pure slop: proper words, even beautiful style, but zero fact checking and the conclusions built on the author's fantasies (wiht AI we'd say hallucinations). Sourced text written with AI and checked against evidence is not slop. It is checkable.

Somewhere... years ago I stopped having feelings about code and text (I leave that for poets, no problem if they are code-poets). What matters for me - the logic, the architecture. The rest is aesthetics (disposable, except of few masterpieces).

Have you heard couples complaining that their wedding ring is only 58,5% gold? So the question was never purity of the surface. It was whether the thing is gold all the way through, or just plated and polished.

Posted 15 August 2026, LinkedIn post.

The source hunt

Provenance, traced 15 Aug: the figure came from a secondary survey's characterization of Seddik et al. (2024). The original theorem bounds the maximal tolerable synthetic fraction in a mixed corpus, not a minimal sufficient real fraction, and the authors' own reading is that the real-data proportion must remain high. Somewhere between the theorem and my post, a ceiling on synthetic data became a floor on real data. I had flagged this figure as single-source with a verify-before-publish gate in my own notes, and shipped past the gate. The gate was right.

No study supports "5% real data keeps the loop stable" as a general result. I checked the main lines of the recursive-training literature (papers listed at the bottom). No universal percentage exists in any of them. The honest verdict: I compressed a regime-dependent literature into one number, and inverted the one bound I thought I was citing.

What the papers actually say

The field distinguishes training regimes, and the answer changes by regime. This is exactly the distinction Dixon asked about.

1. Replace regime, small fixed anchor: slows collapse, does not stop it. Shumailov et al. (Nature, 2024) introduced a partial-refresh setting where a small fraction of the original real data is reintroduced each generation. Degradation slows. It does not stop.

2. Fixed anchor, large enough: stable, in theory. Bertrand et al. (ICLR 2024) prove iterative retraining is stable if the initial model is good enough and the proportion of real data stays sufficiently high. Two limits: "sufficiently high" is model-dependent (no universal number, and not 5%), and the condition is sufficient, not necessary. A 2025 position paper (Schaeffer et al.) warns explicitly against reading it as "collapse is inevitable below a threshold."

3. Accumulate regime: no collapse, even as the real share goes to zero. Gerstgrasser et al. (2024) and Kazdan et al. (ICML 2025) show that when synthetic data accumulates alongside the real data (nothing deleted), test loss stays bounded even though the real-data proportion asymptotically vanishes. Kazdan et al. add the sharpest structural finding: what matters is the interaction between the absolute quantity of real data and the synthetic share, not a proportion alone.

4. The inverse number is the one with a citation. Dohmatob et al., Strong Model Collapse (ICLR 2025): in the scaling-laws regression regime, as little as 1% synthetic data in the corpus (in one formulation, 1 per 1000) can already produce collapse, and larger training sets stop helping. So the only hard small percentage in this literature points the opposite direction from my claim.

5. Mixed-training practice sits between the extremes. Empirical work on mixed regimes (2025) finds 10 to 20 percent synthetic over one to three iterations maintains performance, with degradation accelerating as share and iterations grow. Boundaries, again, regime-dependent.

Corrected claim

There is no percentage of real data that guarantees stability across regimes. What holds up:

• If old data is deleted and replaced each generation, a small real anchor only slows the decay.
• If nothing is deleted and synthetic accumulates on top, collapse does not occur in the studied settings, even as the real share shrinks toward zero.
• In the scaling-law regime, 1% synthetic contamination is already enough to break the "more data helps" assumption.

The engineering question is not "how much real data" but "which regime is your pipeline in."