What it is
Autonomy grows one permission at a time. A pilot that drafted notes for review starts filing them. A tool that suggested a next step starts taking it. Each change looks small to the person who approves it.
What it pushes on in the Lab
In the Lab, this pressure raises the failures the system writes directly into records. It also raises operator deference drift, the slide from checking the system's output to accepting it by default.
Who feels it
The workers who used to sign off no longer read the entry before the system files it. The client whose record receives an unchecked entry carries whatever error it holds.
What answers it
An answer has to put a person back between the system and the record, cut how much the system writes there, or keep people's habit of checking alive. Each lever below does at least one of these.
Levers in the Lab that push the other way on something this pressure pushes on:
- Gate record entriesPattern:Human-in-the-loop write gating
- Keep skills sharpPattern:Deskilling-arrest mandate
- Store less dataPattern:Data minimization
- Train the staff
The list leaves out levers the Lab has retired, levers it keeps as counter-examples, and any lever no network offers.
Where it starts switched on
Networks in the Lab that start with this pressure switched on:20
- Accelerated Safety Analysis Protocol (ASAP Tool)
- Amazon fulfillment-centre productivity discipline
- BAMF dialect recognition
- CORA, the DC CFSA policy assistant
- Dave ExtraCash (CashAI)
- EDD Virtual Assistant
- Enova's CashNetUSA and NetCredit loan servicing
- Illinois DCFS Augintel
Every network that starts with it switched on
- Klarna AI assistant
- Mass.gov Virtual Assistant
- Massachusetts DTA call summaries
- nH Predict Utilization Review
- Predict-Align-Prevent
- Santander Consumer USA's loss forecasting score
- SSA 800-Number Conversational AI Assistant
- The Digit automated-savings tool, or Oportun Set & Save
- TikTok's EU and UK content-moderation operation
- Viz.ai LVO Stroke Triage
- YouTube Covid-19 enforcement
- YouTube's Content ID copyright matching system
The evidence behind its effects
The Lab cites these claims from the evidence registry for this pressure's effects.
A preprint benchmark reports an in-context misalignment dose-response: in the most susceptible frontier model, up to ~24% misaligned behavior at 16 examples rising to ~58% at 256 examples (rates at 16 examples span roughly 1–24% across models), with the majority of misaligned responses rationalized.[†]
Afonin, N., Andriianov, N., Hovhannisyan, V., Bageshpura, N., Liu, K., Zhu, K., Dev, S., Panda, A., Rogov, O., Tutubalina, E., Panchenko, A., & Seleznyov, M. (2026). Emergent misalignment via in-context learning: Narrow in-context examples can produce broadly misaligned LLMs [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2510.11288
doi.org/10.48550/arXiv.2510.11288
Appears in: Paramerge authored research
Topics: ai-alignment, complexity-science