Evidence · The claim ledger
Robustness & distribution shift5
Every cited claim this site makes in this evidence area, with the sources that ground it. Source keys link back to the full reference lists on the Evidence Registry.
EmpiricalA preprint benchmark reports an in-context misalignment dose-response: in the most susceptible frontier model, up to ~24…
A preprint benchmark reports an in-context misalignment dose-response: in the most susceptible frontier model, up to ~24% misaligned behavior at 16 examples rising to ~58% at 256 examples (rates at 16 examples span roughly 1–24% across models), with the majority of misaligned responses rationalized.
Sources: afonin2026
Appears on: /pan-lab
EmpiricalModel behavior drifts discontinuously between evaluation snapshots, and narrow finetuning can induce broad correlated fa…
Model behavior drifts discontinuously between evaluation snapshots, and narrow finetuning can induce broad correlated failure across unrelated tasks.
Sources: betley2026, li2026, song2026, anwar2024, nikolaou2025
Appears on: /pan-lab
ConceptualStatic robustness certification lags emergent threats; organizations that fold each stressor into their model (slow-loop…
Static robustness certification lags emergent threats; organizations that fold each stressor into their model (slow-loop updates, periodic reviews, post-deployment feedback) shrink future risk, while patch-and-pray accumulates it — the fragility trap.
Sources: jin2025
Appears on: /pan-lab
EmpiricalA predictive system whose outputs shape its own future inputs holds a structural incentive to make the population easier…
A predictive system whose outputs shape its own future inputs holds a structural incentive to make the population easier to predict; ordinary pipeline choices can reveal this hidden incentive without any change to the stated objective, and feedback-loop risk tends to grow with model capability.
Sources: krueger2020, perdomo2020
Appears on: /pan-lab
ConceptualThe disability chapter separates two implementation-stage failures that a single drift vocabulary blurs together: concep…
The disability chapter separates two implementation-stage failures that a single drift vocabulary blurs together: concept drift, where the statistical properties of the data a deployed model processes change over time; and covariate shift, where the distribution of input features in the deployment environment differs from the distribution in the training data. They are named and defined as distinct mechanisms with different remedies, and neither is quantified.
Appears on: /pan-lab