Over-reliance, automation bias & deskilling7

Clinical assessors bound by algorithmic allocation with limited override capacity form a documented constrained-judgment pattern in home-care assessment.

Sources: sutton2020, upturn

Appears on: /pan-lab, /practice/bounded-output-screening

In contextual inquiries with Allegheny AFST call screeners, workers calibrated reliance using contextual case knowledge unavailable to the model and reliably detected and overrode erroneous risk scores — complementary human information, not generic distrust, was the safeguard's mechanism.

Sources: kawakami2022, dearteaga2020

Appears on: /pan-lab

AFST workers reported sometimes agreeing with the risk score against their own best judgment under override-rate oversight, and becoming less likely to disagree over time — reliance driven by organizational incentives independent of trust in the tool.

Sources: kawakami2022, kawakami2026

Appears on: /pan-lab

In a four-week randomized study (n=981), voluntary daily chatbot usage duration predicted worse outcomes on loneliness, socialization, emotional dependence, and problematic use across all conditions, and task-style use fostered practical dependence — reduced confidence in independent judgment.

Sources: fang2025, gerlich2025

Appears on: /pan-lab

A validated collaborative-AI metacognition scale (planning, monitoring, evaluation of one's own reliance) predicted collaboration benefits incrementally beyond general metacognition — verification-skill training, not generic AI knowledge, is the calibrated counter to over-reliance.

Sources: sidra2025, bucinca2021

Appears on: /pan-lab, /practice/verification-training

In a randomized study (N=2,784) with objective ground truth, humans accepted incorrect AI suggestions about a third of the time, and their rate of catching AI errors was governed by verification effort, prior trust in AI, and error legibility — surface errors were caught ~82% of the time versus ~31% for errors requiring conceptual judgment — not by financial incentives or time spent.

Sources: beck2026

Appears on: /pan-lab, /practice/verification-training

Group-decision research finds that cohesive groups tend to converge on the most confident member's judgment rather than the most accurate one, so peer dissent that depends on individual courage arrives too rarely to reliably correct the group.

Sources: zarnoth1997

Appears on: /practice/structured-dissent