Evidence Registry

Michigan's MiDAS system auto-adjudicated unemployment-insurance fraud with an extremely high error rate among automated determinations, wrongly accusing tens of thousands of people; litigation and court action forced review and compensation.

Sources: michiganag2022, ieeespectruma, aiincidentdatabase, benefitstechadvocacyhubb

Appears on: /domains/cases/michigan-midas

The Royal Commission into the Robodebt Scheme documented hundreds of thousands of wrongful debts raised by an unlawful income-averaging method, with the onus placed on recipients to disprove automated assessments.

Sources: royalcommissionintotherobode2023a, royalcommissionintotherobode2023b, prygodiczvcommonwealthofaust2021, lawsocietyjournal, royalcommissionintotherobode

Appears on: /domains/cases/australia-robodebt

Indiana's privatized eligibility modernization produced over a million denials in its early years — many procedural rather than substantive — before the state canceled the contract and litigated with its vendor.

Sources: eubanks2018c, governmenttechnologyb, ieeespectrumb

Appears on: /domains/cases/indiana-ibm-eligibility

Independent scrutiny of Rotterdam's welfare-fraud risk model — a 2021 municipal audit followed by a 2023 journalistic investigation that obtained the model itself — documented scores skewed against already-vulnerable groups, and the city suspended the system's use.

Sources: rekenkamerrotterdam2021, lighthousereports2023c, wiredlighthousereports2023, followthemoney, racismandtechnologycenter2023

Appears on: /domains/cases/rotterdam-welfare-fraud, /pan-lab

A large share of Arkansas home-care recipients had care hours cut when algorithmic assessment replaced nurse judgment, and courts found due-process violations centered on the inability to understand or contest determinations.

Sources: arkansasdepartmentofhumanser2017, elderv2022, calo2021, universityofmichiganihpi, benefitstechadvocacyhuba, centerfordemocracytechnology, aiaaic

Appears on: /domains/cases/arkansas-archoices

Evaluation evidence on the Allegheny Family Screening Tool found that screener overrides of the tool's recommendations reduced racial disparity in screen-in rates relative to the tool alone.

Sources: goldhaberfiebertprince2019, centreforsocialdataanalytics2019a, rittenhouse

Appears on: /domains/cases/allegheny-afst, /pan-lab

Illinois's Rapid Safety Feedback flagged thousands of children at 90-percent-or-higher risk of serious harm — beyond any caseload's capacity to act — while children who died in known-to-system cases had not been flagged; the agency ended its use in 2017.

Sources: chicagotribune2017, theimprint2017, governmenttechnologya

Appears on: /domains/cases/illinois-rapid-safety-feedback

Oregon's child-welfare agency dropped its AFST-derived Safety at Screening tool in 2022, citing equity concerns amid national scrutiny of racial disparity in child-welfare algorithms.

Sources: nprap2022, willametteweek2022

Appears on: /domains/cases/oregon-safety-at-screening

Documented benefit-automation failures replicated determinations into downstream systems with no independent reconciliation against the source records — Michigan MiDAS actioned replicated flags and Robodebt reversed the onus onto recipients.

Sources: michiganag2022, ieeespectruma, royalcommissionintotherobode2023b, lawsocietyjournal

Appears on: /pan-lab, /practice/reconcile-copied-records, /practice/record-reconciler

Documented risk-scoring deployments computed scores from multi-agency administrative records originally collected for other purposes, which is the data-protection critique recorded in independent reviews of these systems.

Sources: goldhaberfiebertprince2019, eubanks2018c, lighthousereports2023c

Appears on: /pan-lab

Documented enforcement systems actioned replicated flags automatically — garnishment and penalties applied before any human review step in the recorded MiDAS deployment.

Sources: michiganag2022, benefitstechadvocacyhubb

Appears on: /pan-lab, /practice/reconcile-copied-records, /practice/record-reconciler

The Robodebt Royal Commission documented debts raised from income-averaged derived inputs with the onus placed on recipients to disprove the automated assessments.

Sources: royalcommissionintotherobode2023b, lawsocietyjournal

Appears on: /pan-lab

Ranked risk lists steered which cases were investigated in documented deployments; the anchoring direction is documented while its magnitude is not published.

Sources: lighthousereports2023c, wiredlighthousereports2023, amnestyinternational2021b, goldhaberfiebertprince2019

Appears on: /pan-lab

In the documented MiDAS case, error among no-review auto-adjudications ran roughly 93%, and determinations erred at about 85% without human review versus 44% with it.

Sources: michiganag2022, ieeespectruma, aiincidentdatabase, benefitstechadvocacyhubb

Appears on: /pan-lab, /domains/cases/michigan-midas, /practice/reconcile-copied-records

In the documented AFST evaluation, screener overrides of the tool — roughly a third of its recommendations — cut screen-in disparity from about 20% to 9% relative to the tool acting alone.

Sources: rittenhouse, goldhaberfiebertprince2019, centreforsocialdataanalytics2019a, stapletonetal2022

Appears on: /pan-lab, /domains/cases/allegheny-afst

A peer-reviewed 2024 evaluation of the Allegheny Housing Assessment found that although the tool was substantially more accurate than the VI-SPDAT survey it replaced and produced similar risk-score distributions across race, it did not reduce the racial disparity in service rates: white single adults were served at about 23.3% versus 19.5% for Black clients.

Sources: cheng2024, alleghenycountydepartmentofh2026a

Appears on: /domains/cases/allegheny-housing-assessment, /pan-lab

After a 2025 update to the Allegheny Housing Assessment added a fourth outcome predicting future homelessness, the male share of assigned housing rose from 62% to 76% (and the female share fell from 34% to 24%), reflecting a higher measured one-year homelessness risk among men — an example of an outcome-selection choice reshaping who receives scarce housing.

Sources: alleghenycountydepartmentofh2026b

Appears on: /domains/cases/allegheny-housing-assessment, /pan-lab

The VI-SPDAT was the dominant U.S. homelessness triage assessment for roughly a decade, adopted in at least 39 states and the District of Columbia by 2015, before its own creators announced its phase-out in December 2020 on equity grounds; a 2019 commissioned racial-equity evaluation across four Continuums of Care found race predicted 11 of 16 subscales and that people of color received statistically significantly lower prioritization scores.

Sources: nationalalliancetoendhomeles2022, orgcodeconsultingiaindejong2020, cinnovationswilkey2019

Appears on: /domains/cases/vi-spdat, /pan-lab

The VI-SPDAT showed poor test-retest reliability, with most participants scoring higher on re-administration, and poor inter-rater reliability, with scores varying by interviewer and site; its predictive validity for housing outcomes was mixed across studies, positive for the youth version, null for single adults in one study, and positive in another community sample.

Sources: bitfocus2021, nationalalliancetoendhomeles2022, shinnandrichard2022

Appears on: /domains/cases/vi-spdat, /pan-lab

The U.S. Department of Veterans Affairs' REACH VET program has run a monthly suicide-risk model across the Veterans Health Administration since 2017, scoring about 6.28 million patients and flagging the top 0.1% at each facility (roughly 6,300 to 6,700 veterans a month, more than 130,000 since 2017); an independent re-analysis of 2018 data found the top-0.1% flag has a positive predictive value near 0.05% and a false-negative rate of about 98% for death by suicide, and a 2024 investigation reported that the model treated being a white man as a stronger risk signal than factors specific to women and excluded military sexual trauma and intimate-partner violence from its variables, a characterization VA has contested by framing the excluded factors as less predictive.

Sources: harris2025, glantz2024, graham2025, u2022b

Appears on: /domains/cases/reach-vet, /pan-lab

Two Veterans Health Administration evaluations of REACH VET found the program associated with improved proximal outcomes — more completed outpatient appointments, more new safety plans, and fewer documented suicide attempts — but not with reduced death by suicide: a 2021 triple-differences study of 173,313 veterans across 141 facilities found no association with suicide or all-cause mortality, and a 2025 follow-up of 266,246 observations replicated the null with all confidence intervals crossing one; both are observational rather than randomized studies.

Sources: mccarthy2021, dent2025c

Appears on: /domains/cases/reach-vet, /pan-lab

Kaiser Permanente Northern California has embedded a machine-learning suicide-attempt risk model in the electronic health record of a large virtual mental-health program that handles more than 5,000 intake visits a month; the model is scored in near-real-time (about a 30-minute delay after an encounter trigger) and, at pre-set thresholds, flags high-risk patients to the intake clinician, routing them into the same suicide-risk-assessment and outreach workflow that a positive self-report screen (the PHQ-9 and Columbia-Suicide Severity Rating Scale) triggers, so the machine flag and the self-report alert are effectively OR-merged. In a study of 1,623,232 intake appointments (2012 to 2022, base rate 0.17 percent) the model reached an area under the ROC curve of 0.77 and its top risk decile captured 48.8 percent of appointments later followed by an attempt, but with a positive predictive value of about 0.8 percent.

Sources: hsin2026, hsin2025, papini2024

Appears on: /domains/cases/kaiser-epic-suicide-risk, /pan-lab

Because the near-term suicide-attempt base rate at Kaiser Permanente Northern California mental-health intake is very low (0.17 percent) and the positive predictive value in the top risk decile is about 0.8 percent, the large majority of flagged patients will not attempt suicide in the window, so adding the machine-learning flag as a redundant sensor OR-merged onto the existing self-report screen imports a substantial false-positive and clinician-workload burden at scale — a caution the implementation team itself raised. The implementation reports are feasibility- and design-focused and present no evaluation showing the deployment reduced suicide attempts.

Sources: papini2024, hsin2026, hsin2025

Appears on: /domains/cases/kaiser-epic-suicide-risk, /pan-lab

Crisis Text Line, a national nonprofit crisis service, built an in-house machine-learning severity-triage model that reorders which texters volunteer counselors reach first; from about 2017 to 2020 the same anonymized crisis-conversation corpus was routed to Loris.ai, a for-profit spinoff CTL held an ownership stake in — reported by Politico-derived reporting at roughly 53% — which used it to train commercial customer-service software. After a January 28, 2022 Politico exposé, CTL ended the arrangement within three days and requested that the data be deleted; an FCC commissioner referred the matter to the FTC in March 2022, and no public FTC enforcement action is documented. CTL states the shared data was anonymized and never sold as personally identifiable information, and the exact number of records shared has not been made public.

Sources: crisistextlinewikipedia2026, crisistextline2022, reierson2022, bentoninstituteforbroadbanda2022

Appears on: /domains/cases/crisis-text-line-loris, /pan-lab

Crisis Text Line obtained consent for its data collection through an automated reply directing texters to a lengthy Terms of Service — described as a roughly 50-page or 4,000-plus-word document — accepted at the moment of acute crisis by users who include many minors; critics including a former board chair, who voted for the data-sharing arrangement and later said she would not have "knowing what I know now," and a terminated volunteer argued that a Terms of Service is not meaningful informed consent for people in crisis. CTL says texters must consent to its privacy policy to use the service and can request deletion by texting the word DELETE, and that since 2023 its in-house research has been overseen by an Institutional Review Board.

Sources: markkulacenterforappliedethi2022, eysenbach2025, reierson2022, trujillo2025

Appears on: /domains/cases/crisis-text-line-loris, /pan-lab

NarxCare is a proprietary clinical-decision-support platform built by Bamboo Health that layers over state Prescription Drug Monitoring Programs and returns three Narx Scores plus a composite Overdose Risk Score (each 000-999) into the electronic health record, the PDMP portal, or pharmacy software, often in the patient header alongside vitals and allergies; adoption figures vary by what is counted (more than 40 states and territories run their PDMPs on Bamboo technology and five of the top six pharmacy chains use NarxCare, while the scoring module itself is switched on in more than 20 states). The vendor states the scores are intended to aid, not replace, clinical judgment and should never be sole justification for providing or refusing medication, but clinician and patient-advocacy sources document de facto determinative use — denials, forced tapers, and pharmacy refusals — driven by automation bias and fear of regulatory and criminal liability; patients cannot see, challenge, or correct their scores, the algorithm is proprietary and has not been independently validated for clinical care, and the FDA has not regulated it as a Software-as-a-Medical-Device, so contestation has instead run through FDA citizen petitions (one rejected on procedural grounds in 2023 and a second, docket FDA-2025-P-0701, pending since 2025 with more than 1,000 public comments).

Sources: bamboohealth2023, wang2026, millerandwhitehead2023, buonora2023, oliva2022, painnewsnetwork2023, medscape2025a, medscape2025b

Appears on: /domains/cases/narxcare, /pan-lab

On its own 2013-2016 training and validation data Bamboo Health reported an Overdose Risk Score precision of about 75% (self-reported, never independently reproduced), and its own external-validation set from 2017-2023 showed precision falling to about 52%, which the vendor attributed to rising illicit fentanyl (untracked by prescription-monitoring programs) and wider use of opioid-use-disorder treatment medication. A 2026 npj Digital Medicine study that reconstructed the model on California's CURES prescription database (about 17.9 million observations) and on commercial claims data obtained a precision of only 0.01 to 0.32 across several model architectures; because overdose-death labels were unavailable to the independent researchers, that reconstruction was trained on proxy outcomes rather than the score's actual overdose-death target, so it is best read as evidence that proprietary opacity prevents anyone outside the vendor from assessing the deployed model's accuracy, fairness, or safety, rather than as a strict like-for-like refutation of the vendor's figure.

Sources: bamboohealth2023, wang2026

Appears on: /domains/cases/narxcare, /pan-lab

Limbic Access, a Class IIa UKCA-certified self-referral and triage chatbot for NHS Talking Therapies, is deployed across a large and growing share of the service (its maker's chief executive claimed about 63% of the NHS in April 2026). Two peer-reviewed observational studies report large operational gains — a study of 129,400 self-referrers across 28 services found referrals rose 15% in chatbot services versus 6% in control services, and a study of 64,862 patients reported clinical-assessment time cut from 54.4 to 41.6 minutes and recovery rates of 58% versus 27.4% — but both studies are non-randomized and were authored by people employed by or holding shares in the tool's maker (all six authors of the access study and seven of the eight authors of the efficiency study), and the efficiency study's own authors caution that the recovery difference is subject to unmeasured confounding from self-selection. No randomized or independent third-party effect estimate has been published.

Sources: habicht2024, rollwage2023, chatterjee2026

Appears on: /domains/cases/limbic-access-nhs, /pan-lab

In the peer-reviewed study of 129,400 self-referrers across 28 NHS Talking Therapies services, self-referrals rose more where the chatbot was in use than in control services (15% versus 6%), with the largest increases among under-served groups — reported at about +179% for nonbinary people, +40% for Black and +39% for Asian self-referrers. This is an observational multi-site association, not a randomized causal effect.

Sources: habicht2024, heikkila2024

Appears on: /domains/cases/limbic-access-nhs, /pan-lab

Woebot, a rule-based (non-generative) cognitive behavioral therapy chatbot used by roughly 1.5 million people over its lifetime, was deliberately retired by its maker on a pre-announced schedule: the app was taken down on June 30, 2025, with a transcript-request window (deadline July 15, 2025) and all account data anonymized as of July 31, 2025, removing personally identifying information rather than silently abandoning the service. The founder and chief executive attributed the shutdown to the cost of meeting FDA marketing-authorization requirements and to a regulatory-pathway gap, framing the exit as economic and regulatory rather than a clinical failure - a self-reported account, not an independently audited finding. The roughly 1.5 million figure is a cumulative lifetime number reported in press coverage, not an audited point-in-time active-user count.

Sources: aguilar2025, woebothealth2025, hlth2025

Appears on: /domains/cases/woebot-shutdown, /pan-lab

The foundational study in Woebot's peer-reviewed efficacy record is an early-stage, vendor-authored trial: a 2017 randomized controlled trial in JMIR Mental Health (n=70, ages 18 to 28, two weeks, unblinded, information-only control) reported a moderate between-groups reduction in PHQ-9 depression symptoms (about d = 0.44). That is an efficacy signal, not regulatory validation; the study authors were affiliated with the tool's maker, and no independent, arm's-length evaluation of the consumer app is documented (the broader published evidence base is not assembled here). A separate, investigational, prescription-only variant (WB001) received an FDA Breakthrough Device Designation in May 2021 - an expedited-review status, not marketing authorization - and entered a pivotal Software as a Medical Device trial with the first patient enrolled in January 2023, but never received FDA marketing authorization; it must not be conflated with the consumer app.

Sources: fitzpatrick2017, woebothealthbusinesswire2021b, woebothealthbusinesswire2023

Appears on: /domains/cases/woebot-shutdown, /pan-lab

DWP's own fairness assessment (covering 1 April 2024 to 31 March 2025) of its live Universal Credit Advances fraud-risk model reports statistically significant referral disparities and an accuracy inversion: relative to a 35-44 comparator, claimants aged 55-65 were about 2.80 times as likely to be referred for review and non-UK nationals about 2.27 times as likely, while for older claimants those referrals were less likely to be correct (relative correct-referral likelihoods of about 0.58 at 55-65 and 0.23 at 66-plus, the latter resting on a small sub-sample DWP flags to treat with caution). The disparities were first disclosed under freedom-of-information law and reported in December 2024, and DWP has committed to retrain the model. The figures are DWP-reported relative ratios, not independently audited absolute error rates.

Sources: departmentforworkandpensions2025c, theguardian2024

Appears on: /domains/cases/uk-dwp-fraud-ml, /pan-lab

DWP states that a human caseworker always makes the final decision on a referred Universal Credit advance with no automated decision-making, and is deliberately not shown the risk score or told the referral came from the model; DWP describes the model as around three times more effective than a randomised control at identifying fraud risk and judges continued operation reasonable and proportionate while committing to retrain it. The Public Law Project counters that only age was fully assessed among protected characteristics and that the assessment relied on safeguards preventing downstream harm rather than showing the model to be non-discriminatory. The wider counter-fraud programme is meanwhile expanding into bank-data eligibility verification under the Public Authorities (Fraud, Error and Recovery) Act 2025, a distinct system not yet in force.

Sources: centraldigitalanddataoffice2025, departmentforworkandpensions2025c, publiclawproject2025, departmentforworkandpensions2025b

Appears on: /domains/cases/uk-dwp-fraud-ml, /pan-lab

During the pandemic unemployment surge, a private facial-recognition identity check operated as a de facto eligibility gate for unemployment benefits in at least 25 U.S. state workforce agencies, with a live 'trusted referee' interview queue that House investigators documented averaging nearly 10 hours in North Dakota and over 4 hours in 14 of 21 states, versus about 6 minutes in New Jersey where an in-person option existed. Oregon's own one-month study (n=10,656 routed) recorded verification-completion differences by group -- for example 41.59% for African American and 34.48% for Spanish-language claimants versus 53.44% for White claimants -- but stated the study showed differences in completion and did not show causation, so these are a friction proxy, not a measured wrongful-denial rate. The U.S. Department of Labor does not collect or report the number of workers blocked for inability to verify identity, and where verification precedes filing those workers are not counted as denied claims at all, so the scale of any wrongful lockout is undocumented.

Sources: ushousecommitteeonoversighta2022a, stateoforegonemploymentdepar2022, nationalemploymentlawproject2023, usdepartmentoflabor2023c

Appears on: /domains/cases/us-idme-unemployment, /pan-lab

A U.S. Department of Labor Inspector General audit (March 31, 2023) found that among 24 state workforce agencies using a facial-recognition identity contractor, 18 of 24 (75%) contracts did not specify one-to-one versus one-to-many matching, 15 of 24 (63%) did not address data storage, and 13 of 24 (54%) did not address destruction of the collected biometric data, while 22 of 24 (92%) agencies reported the technology reduced improper payments -- the operator-side benefit that sustained adoption even as the wrongful-lockout cost went unmeasured. The vendor initially represented it used only one-to-one matching and later acknowledged one-to-many matching against a database; after bipartisan backlash the IRS and Treasury dropped the mandatory facial-recognition requirement in February 2022 and the vendor made it optional across agencies, though the service remained in use for unemployment identity verification in a large share of states, and a 2026 IRS proposal would allow it to retain taxpayer biometric data up to 36 months after account deletion. The reported improper-payment reductions are agency self-reports, not independently audited.

Sources: usdepartmentoflabor2023c, americancivillibertiesunionj2022, electronicfrontierfoundation2022, biometricupdate2026

Appears on: /domains/cases/us-idme-unemployment, /pan-lab

On August 30, 2023 CMS notified states that their automated Medicaid ex parte renewal systems were evaluating eligibility at the household or family level rather than the federally required individual level, so when any one household member could not be auto-renewed the whole household was dropped procedurally if a returned form was not received. CMS found 30 states had the defect and, on September 21, 2023, announced that nearly 500,000 children and other individuals who had been improperly disenrolled would regain coverage, requiring the affected states to pause procedural disenrollments, reinstate coverage, and reprogram to individual-level renewal. The ~500,000 figure is an aggregate of state-reported estimates compiled by CMS, not an independently audited count; children were disproportionately affected because their income-eligibility thresholds are higher than adults', and an HHS ASPE analysis (cited via Georgetown CCF) projected roughly 74% of disenrolled children would still be eligible, a projection rather than a post-hoc audit.

Sources: centersformedicareandmedicai2023a, centersformedicareandmedicai2023b, georgetownuniversitycenterfo2023a, healthcarediveemilyolsen2023

Appears on: /domains/cases/us-medicaid-unwinding-autorenewal

The Medicaid unwinding was governed by a federal monitor-and-respond loop: Section 5131 of the Consolidated Appropriations Act, 2023 (SSA section 1902(tt)), codified in a December 6, 2023 interim final rule, gave CMS mandatory monthly state reporting plus, for noncompliance, a Federal Medical Assistance Percentage reduction of 0.25% per quarter (capped at 1%), civil monetary penalties up to $100,000 per day for reporting failure, corrective action plans, and authority to order suspension of procedural disenrollments. Against that instrumentation the overall churn was large: KFF recorded about 25.2 million people disenrolled as of September 12, 2024 with 69% of disenrollments for procedural rather than eligibility reasons, while a June 24, 2025 GAO audit independently found about 27 million disenrolled in the first 18 months, roughly one-third of those continuously enrolled. The KFF and GAO totals differ because they cover different windows and use different data and methods, not because they conflict; the enforcement penalty details are drawn from the interim final rule and a legal-analysis summary.

Sources: federalregister2023, morganlewisandbockiusllp2023, kff2024, usgovernmentaccountabilityof2025

Appears on: /domains/cases/us-medicaid-unwinding-autorenewal

A 2024 Tribunal de Contas da Uniao (TCU) plenary audit found that INSS benefit denials were nonconforming above the maximum acceptable limit in both channels it sampled: 10.94% of automatically analyzed denials (January to May 2024) and 13.20% of manually analyzed denials (2023 sample), in Acordao 634/2025-Plenario (process TC 008.309/2024-8, session 26 March 2025). Nonconformity ('desconformidade') is a TCU audit-analysis category that includes wrongful denials but is not identical to a court-confirmed wrong-denial rate, so these are not a hard error rate; the absolute counts reported in coverage (about 920,000 automatic denials in the audited window, about 100,000 estimated wrongful, and 250,000 to 290,000 estimated unjustified manual denials) are journalistic extrapolations from the TCU percentages, not officially published counts. Neither channel uses a machine-learning or predictive risk score; 'automatic' means rules-based administrative processing and documentary-conformity analysis.

Sources: tribunaldecontasdauniao2025, infomoney2025, consultorjuridico2025

Appears on: /domains/cases/brazil-inss-automation

The TCU root-cause finding was that INSS measures server productivity by the number of processes analyzed rather than the quality of the decision's justification, creating an incentive to choose denial as the fastest disposition, with no incentive for correct motivation of the denial and no effective communication with the insured. The correction channel is slow and external: the CNJ recorded 5,109,076 pending previdenciario lawsuits as of 31 October 2024, and CNJ 'Justica em Numeros' data put the average pending-case duration at about 746 days with a conciliation rate near 24.84%, so a fast automated or manual denial is reversed only after a roughly two-year judicial wait. The 5.1-million-case backlog and the 746-day duration are CNJ caseload figures and cannot be mechanically attributed to automated denials specifically, because the public data do not link an individual court reversal to the channel that produced the denial.

Sources: tribunaldecontasdauniao2025, consultorjuridico2024, conselhonacionaldejustica2024

Appears on: /domains/cases/brazil-inss-automation

The Commonwealth Ombudsman's first report, Automation in the Targeted Compliance Framework (published 6 August 2025), found that the Department of Employment and Workplace Relations and Services Australia acted contrary to the law and unlawfully cancelled the payments of 1,009 jobseekers under the predominantly automated Targeted Compliance Framework, with a further 45 auto-cancelled after a pause was ordered (the first-cohort figure is variously reported as 'more than 900', 964, or 1,009; 1,009 is the most precise and most widely cited). The unlawfulness was an omission: the April 2022 SPROM Act required a discretionary reasonable-excuse consideration before a cancellation and required a mandated automated-decision safeguard, the Digital Protection Framework, neither of which was implemented, so cancellations executed without the check the law required. The defect operated from April 2022, was detected in September 2023 by external legal advisors, and cancellations were not paused until July 2024 — a roughly ten-month gap the Ombudsman called not acceptable; both agencies accepted all seven recommendations. A commissioned Deloitte assurance review separately found the IT system increasingly unstable, with five IT errors dating to 2018, and could not assure the integrity, effectiveness, or appropriateness of decisions.

Sources: commonwealthombudsman2025a, informationageaustraliancomp2025, itnews2025, theexamineraustralianassocia2025, departmentofemploymentandwor2025a

Appears on: /domains/cases/australia-workforce-tcf

The Targeted Compliance Framework operates at very large scale: advocacy and analysis of departmental data describe roughly 2.5 million payment-suspension notices a year to about a million people, with 200,000 to 240,000 people facing suspension threats each quarter (these system-scale figures are directionally consistent across sources but exact denominators and periods vary). The Ombudsman's second report, Fairness in the Targeted Compliance Framework (published 9 December 2025), found that automatic Penalty-Zone suspensions undermine a jobseeker's ability to challenge penalties, that the department's assessment of provider performance lacks transparency, and that a high rate of provider decisions are overturned on review, while the complaints line went more than 140,000 calls unanswered between November 2024 and September 2025. A separate and far larger section 42AM automated-cancellation review is still expanding: the department previously published up to 9,510 unlawful cancellations or reductions, an advocacy estimate put potential exposure near 310,000, and in June 2026 Senate estimates a department official said the number was in the vicinity of that estimate but qualified that 55 to 70 percent may have legitimately lost eligibility, implying roughly 93,000 or potentially 100,000-plus. Those larger figures are estimates of potentially unlawful cases pending case-by-case assessment, not confirmed cancellations.

Sources: theantipovertycentre2026, powertopersuade2025, commonwealthombudsman2025b, sbsnews2025

Appears on: /domains/cases/australia-workforce-tcf

New York City launched the MyCity Business chatbot in 2023 on Microsoft Azure AI as a public-facing generative-AI adviser for business owners. A March 29, 2024 investigation by The Markup with THE CITY and Documented NY found it confidently and repeatably wrong on legal obligations, advising businesses in ways that would break the law, including that employers could take a cut of workers' tips, that landlords need not accept Section 8 vouchers or source-of-income tenants (illegal in New York City), that stores could go cashless against a 2020 city law, and that funeral-price disclosure could be concealed against the federal funeral rule; when ten staffers asked the housing-voucher question they received the same wrong answer, which had changed from an earlier correct one, showing the tool was non-deterministic. The 2024 findings are qualitative, based on specific tested questions rather than a sampled error rate. The city relabeled the tool a beta product with a disclaimer and applied a scope-narrowing patch rather than withdrawing it, kept it online for roughly two years, and shut it down in early 2026 as a budget cut rather than an accuracy fix.

Sources: themarkup2024, themarkupandthecity2024, reutersjonathanallen2024, themarkupcolinlecherandkatie2026

Appears on: /domains/cases/nyc-mycity-chatbot

A December 30, 2025 performance audit of the MyCity system, issued under New York City Comptroller Brad Lander, found the chatbot 'appears to be unable to provide accurate or consistent information' and reported that the wider MyCity system had cost over 100 million dollars across more than 120 agreements with about 50 vendors, lacked a system development plan, and had not delivered the promised single-form access to city benefits; the Office of Technology and Innovation disagreed with all seven of the audit's recommendations, including one to conduct AI red-teaming. Among the audit's figures, an internal weekly production report reproduced in the audit showed the chatbot did not answer 23 of 48 tested government questions, and of the more than 2,200 questions asked in July and August 2025 the 70 users who left thumbs-up-or-down feedback were 71.4 percent negative (50 of 70), a share the city disputes as roughly 2.25 percent of all responses, with the audit rebutting that denominator. The 100-million-dollar figure is the whole MyCity system, not the chatbot alone.

Sources: officeofthenewyorkcitycomptr2025

Appears on: /domains/cases/nyc-mycity-chatbot

Nevada's Department of Employment, Training and Rehabilitation contracted Google to build a generative-AI tool on the Vertex AI Studio cloud platform that reads an unemployment-appeal hearing transcript and evidence, retrieves against a corpus of Nevada unemployment law and prior appeals decisions, and drafts a recommended determination (approve, deny, or modify a claim) together with the written decision for a human referee to review and sign. The contract set a 90 percent success requirement self-assessed by state workers on test decisions -- not an independent external audit -- and DETR said it wanted accuracy higher than 90 percent before going live; rollout was repeatedly delayed over less-than-desired accuracy, including the tool citing incorrect Nevada statutes and failing to pull information from all hearing documents, problems officials said were fixed. Reported cost evolved from about 1 million dollars in 2024 to a total of 2.6 million dollars with about 1.1 million spent by early 2026. As of the most recent available reporting (March 2026) the system was in delayed pre-deployment testing on historical appeals and described as launching in coming weeks; it was not independently confirmed to be adjudicating live claimant appeals.

Sources: thenevadaindependent2025, themarkuptoddfeathers2024, thenevadaindependentericneug2024, fordhamintellectualproperty2024

Appears on: /domains/cases/nevada-detr-genai-appeals, /pan-lab

Nevada's generative-AI unemployment-appeals tool was justified as a speed measure for a pandemic-era backlog, projecting a drop in referee determination time from as much as several hours to about five minutes per case, with a mandatory human review DETR said adds an estimated 10 to 30 minutes and a required referee sign-off (Director Christopher Sewell said no AI-drafted written decisions issue without human review). Legal scholars, attorneys who represent claimants, and a former U.S. Department of Labor official warned that backlog and speed pressure could hollow out that review and create incentives to rubber-stamp AI outputs -- one attorney noting the time savings only happens if the review is very cursory, and a legal analysis warning staff might feel pressured to authorize AI decisions with haste. That automation-deference risk is expert-projected, not a measured outcome: no referee override or rejection rate has been published, and claimants are not required to consent to AI processing of their appeal.

Sources: themarkuptoddfeathers2024, fordhamintellectualproperty2024, thenevadaindependent2025

Appears on: /domains/cases/nevada-detr-genai-appeals, /pan-lab

The Government Digital Service ran a cross-government experiment with Microsoft 365 Copilot from September 30 to December 31, 2024, with about 20,000 employees across 12 organisations, and published the findings report on June 2, 2025. Participants self-reported saving an average of about 26 minutes per working day (the report extrapolates this to roughly 13 days a year from the median values of six reported time-savings ranges; independent coverage recomputed it to about 4.6 days on a 253-working-day basis), 17% reported no clear savings, adoption held near 80% after peaking at about 83%, and 82% said they would not want to return to working without it. The experiment measured adoption and self-reported time rather than output quality: the report recorded no audited error rate, flagged significant accuracy concern for low-verifiability tasks such as grievance handling and performance evaluations, noted external web data was used without built-in verification, and documented a provenance failure in which the tool struggled to identify which documents generated a response.

Sources: governmentdigitalservicedsit2025, governmentdigitalservice2025b, theregisterthomasclaburn2025

Appears on: /domains/cases/gds-m365-copilot-experiment, /what-ai-can-do

A companion Department for Business and Trade evaluation of Microsoft 365 Copilot (1,000 licences, October to December 2024; published August 28, 2025) reported 72% user satisfaction but concluded it did not find robust evidence that time savings were leading to improved productivity; in observed tasks its users completed spreadsheet data analysis more slowly and to worse quality and accuracy than non-users, and produced presentation slides over 7 minutes faster on average but to worse quality and accuracy that then needed correction. In its diary study, 22% of respondents said they had identified hallucinations, 43% detected none, and 11% were unsure, with a further roughly one in five not answering, so the figure reflects user-detected hallucination rather than audited incidence. A Department for Work and Pensions evaluation (3,549 licences; published January 29, 2026) measured 19 minutes a day saved across eight routine tasks against a comparison group (95% confidence interval 17 to 22 minutes), found 85% rating meeting-note accuracy good or very good, reported that users consistently reviewed outputs before use, and concluded the tool is complementary to human expertise and requires consistent human oversight.

Sources: departmentforbusinessandtrad2025b, theregisterpaulkunert2025, departmentforworkandpensions2026

Appears on: /domains/cases/gds-m365-copilot-experiment, /what-ai-can-do

According to its Algorithmic Transparency Recording Standard record, published on November 27, 2025, the UK Department for Work and Pensions runs a Whitemail Insights and Vulnerability Scanner that reads roughly 25,000 scanned documents a day (reported as around 22,000 a day at end-2023 and in a March 2024 operator interview). Each document is passed first through the Vulnerability Scanner, a pre-trained open-source transformer doing zero-shot classification, which flags potentially vulnerable customers against eight prescribed themes including suicide and self-harm, domestic violence and abuse, and financial hardship; only documents not flagged as indicating vulnerability are relayed to Whitemail Insights for routing across nine themes. The output to trained staff is an anonymised daily report of flagged customers, and DWP states the tool does not make or influence benefit entitlement decisions. The record names precision, recall and F1-score as its evaluation metrics but discloses no values, and no independent accuracy evaluation has been published.

Sources: departmentforworkandpensions2025b, trendall2025, ukparliamentworkandpensionsc2023, corbridge2024a

Appears on: /domains/cases/dwp-whitemail-scanner

Guardian FOI reporting in January 2025 recorded that benefit claimants are not told the AI reads their correspondence: the internal data protection impact assessment stated that letter writers do not need to know about their involvement in the initiative, and the tool had been piloted since at least 2023 without appearing on the central government AI transparency register despite a ministerial mandate. The correspondence it processes can include national insurance numbers, health information, bank details, and children's details. Turn2us policy manager Meagan Levin voiced serious concerns, noting that prioritising some cases inevitably deprioritises others, so it is vital to understand how these decisions are made and ensure they are fair. The further reading that a missed flag on the unflagged residual therefore has no complaint channel and surfaces only as downstream harm is an analytical inference from the documented non-notification and shortlist design, not an adjudicated harm.

Sources: booth2025, toth2025, dent2025a

Appears on: /domains/cases/dwp-whitemail-scanner

In the UK Home Office's own pilot of an AI tool that summarises asylum interview transcripts for decision-makers, 9% of the generated summaries were deemed inaccurate or incomplete and removed by a pre-use filter before any caseworker saw them, and 23% of users reported not being fully confident in the rest; the summaries carried no source references back to the transcript. The official evaluation, published April 29, 2025, measured a 23-minute-per-case time saving (a 32% reduction) for the summariser and about 37 minutes for a companion policy-search tool, and Home Office Calibre quality-assurance reviews found no statistically significant difference in decision quality on small pilot samples. The evaluation recommended addressing the identified limitations before a full rollout, continuous monitoring in early rollout, and a larger-scale evaluation after deployment; the Home Office announced expansion the same day. By January 2026 the policy-search tool had been rolled out to all asylum decision-makers, and per trade-press reporting the summarisation tool entered national rollout in April 2026.

Sources: ukhomeofficegovuk2025, openrightsgroup2026c, governmenttransformation2026

Appears on: /domains/cases/home-office-asylum-summarisation

The Home Office's asylum interview-summarisation tool inserts a compression step whose measured value is a 23-minute-per-case time saving that exists only insofar as the decision-maker does not redo the reading the summary replaced: caseworkers are not required to verify summaries against transcripts, and the pilot summaries carried no source references that would make checking cheap. The correction loop is also severed from the other side. In a May 2026 written parliamentary answer, minister Alex Norris confirmed that asylum claimants are not told about the AI tools used in their cases, so the one party with first-hand knowledge of their own account cannot surface a summary error; this postdates Article 22C of UK GDPR (in force February 5, 2026). As of mid-2026 the rollout had proceeded without a published post-deployment evaluation or continuous-monitoring data and, per Open Rights Group, without a published Data Protection Impact Assessment, Equality Impact Assessment, or Algorithmic Transparency Recording Standard entry, with prompts withheld under a Freedom of Information refusal. A March 16, 2026 commissioned legal opinion argues the use is likely unlawful on procedural-fairness and data-protection grounds; that is a contested legal position, not a court ruling.

Sources: ukhomeofficegovuk2025, resultsense2026, openrightsgroup2026a, openrightsgroup2026b

Appears on: /domains/cases/home-office-asylum-summarisation

In February 2026 the Superior Court of Los Angeles County, the largest trial court in the United States, began a pilot of the Learned Hand AI drafting workbench with six civil-division judges and their research attorneys under a contract of about $314,000 running into early 2027, and the Superior Court of Riverside County gave seven civil and probate research attorneys access under a separate $10,000 agreement used for research memos; the tool ingests case filings, synthesizes applicable law, and drafts proposed orders in the individual judge's own writing style. Under California Judicial Council Rule 10.430 (effective September 1, 2025, the first statewide court generative-AI framework in the nation), disclosure is required only when a document consists entirely of generative-AI output, and the rule reaches judicial officers only for tasks outside their adjudicative role, so neither court is obligated to tell litigants when AI assisted with an order or memo in their case; both courts declined to confirm whether litigants whose cases are used in testing are informed.

Sources: mihalovichandjohnson2026, queally2026, judicialcouncilofcalifornia2025

Appears on: /domains/cases/learned-hand-la-courts

In the Learned Hand pilot the only reported error-correction safeguard is the judge's own review: officers are required to review and edit each draft before adopting a tentative ruling, and a court spokesman said the assistance does not supplant the judicial officer's independent role. No external audit, query logging, or benchmarking regime was reported (legal analysis coverage drew a contrast with Michigan's approach), and no error, edit, or override rate for the tool has been published. The Los Angeles District Attorney raised an anchoring concern, that an AI-generated draft could greatly influence what the judge's position should be before an independent view forms; this is an attributed critique rather than a measured effect, and the vendor's per-sentence Deep Verify hyperlinking and multiple-verification-passes claims are unverified vendor statements.

Sources: queally2026, howell2026, mihalovichandjohnson2026, learnedhandandsuperiorcourto2026

Appears on: /domains/cases/learned-hand-la-courts

The UK Government Digital Service ran what it called the government's biggest public test of generative AI to date: across two gated public pilots (a late-2024 web pilot of 10,136 users asking 23,838 questions, and an autumn-2025 GOV.UK app pilot of 641 users asking 2,670 questions in four weeks), more than 10,000 people asked GOV.UK Chat about 26,000 questions on tax, benefits and visas. Its first 2023 version was held back in findings published January 18, 2024 because, GDS reported, answers did not reach the highest level of accuracy demanded for a site like GOV.UK, including a few cases of hallucination. GDS reports measured answer accuracy rising from 76 percent (its earliest benchmark) to 90 percent by the autumn 2025 pilot, assessed by subject-matter experts plus automated evaluation, an 88 percent answer rate for in-scope questions after a clarifying-questions feature was added, and that 508 attempts to jailbreak the system across the pilots were all prevented by its guardrails; it soft-launched to all GOV.UK app users on March 26, 2026 and officially launched on May 14, 2026. Nearly every one of these figures is self-reported by GDS, the system's operator, and the accuracy denominators and sampling frames are unpublished.

Sources: governmentdigitalserviceinsi2026, governmentdigitalserviceinsi2024a, governmentdigitalservice2026

Appears on: /domains/cases/govuk-chat

GOV.UK Chat is a retrieval-augmented assistant that, per its Algorithmic Transparency Record published October 7, 2025, answers only from roughly 700,000 vectorised chunks (36.9 GB) of curated official GOV.UK guidance, is instructed to ignore its training data, rejects questions containing phone numbers, emails or card numbers, links every answer back to its GOV.UK source pages with a reminder to verify, and retains question data encrypted for 12 months; GDS states it does not attempt to provide advice and makes no automated decision. GDS's December 2025 vision post frames a content-dependency loop, stating that GOV.UK Chat can only be as good as the content published on GOV.UK by departmental teams. The record's independent evaluation is a jailbreak (security) assessment conducted with the AI Security Institute, alongside the record's own caveat that it is not possible to guarantee no jailbreaking attempts will succeed; there is no independent audit of the accuracy methodology, and GDS's claim that for government-related questions the tool scores higher than widely-used consumer AI assistants is the operator's own comparison.

Sources: departmentforscience2025b, governmentdigitalserviceinsi2025, civilserviceworldjimdunton2026

Appears on: /domains/cases/govuk-chat

Frida is the chatbot at the front line of the Norwegian Labour and Welfare Administration's (NAV) anonymous contact-center chat channel; NAV states it launched in summer 2018 and, as of 2026, that citizens first meet Frida (open 24 hours a day) and can ask it for a human advisor on weekdays between 9:00 and 15:00, with the channel anonymous and no personal information visible to NAV. During the COVID-19 lockdown NAV reported a roughly 250 percent surge in inquiries; the platform vendor's case study reports the chatbot answered more than 270,000 coronavirus-related inquiries and that about 80 percent of enquiries were resolved without escalating to a human, and NAV's own funded research report records nearly 11,000 inquiries in Frida on some days between March and May 2020 with a week-13-2020 peak equal to the capacity of about 230 human advisors, where the vendor and the peer-reviewed EJIS study state about 220. These pandemic figures originate substantially in the vendor's marketing case study and are reported here as vendor claims with the 220-versus-230 source tension left unresolved; the roughly 80 percent containment is a completion or non-escalation rate, not a measure of answer accuracy.

Sources: boost2020, parmiggiani2021, vassilakopoulou2022a, nav2026

Appears on: /domains/cases/frida-nav-norway

The best-documented property of NAV's Frida chatbot is its chatbot-to-human handover boundary, which the evidence suggests behaves as a governance-controlled dial: NAV's funded three-university Frida@work project reports that about one in five conversations transferred to a live human advisor under free channel choice, and only about 30 percent of dialogues transferred when NAV removed the explicit choice between the chatbot and human chat, a regime-specific figure that must be read against the interface in force. Independent chat-log studies document irrelevant answers, omitted information, and three classes of domain-knowledge failure, with the most critical failures occurring when a misunderstanding goes undetected inside a conversation the chatbot completed; no per-answer accuracy or error rate has been published, and the Frida@work project found context survives the handover imperfectly, with citizens often unsure whether they are talking to a person or a machine. NAV's own 2025 channel-use analysis found that chatbot visibility appears not to change contact-center inquiry volumes and attributes the steady post-2019 decline to a bundle of causes (self-service improvements, new application systems, SMS notifications and changed contact-center practices), so the chatbot is not shown to reduce human workload outside the crisis peak.

Sources: parmiggiani2021, verne2022, simonsen2020, mcvey2025

Appears on: /domains/cases/frida-nav-norway

Burokratt is Estonia's national network of public-sector chatbots operated by the Information System Authority: each participating institution runs its own assistant, a central classifier routes a citizen's query between them and oversees the handover, and from 2025 a shared knowledge module built from the eesti.ee state portal feeds cross-domain answers. RIA's page lists 20 participating organisations and trade press reports 18 integrated; an independent 2025 ethnography drawing on twelve insider interviews (conducted in late 2023, when the system spanned ten institutions) found it marketed as advanced AI while functioning much like an FAQ list, with use differing considerably by institution and low in some. No published session volumes, escalation-to-human rates, or answer-accuracy figures, and no dedicated algorithmic-oversight body, published evaluation framework, or national-audit report on the network, were located in the public record.

Sources: informationsystemauthorityri2025a, govinsider2025, kaun2025

Appears on: /domains/cases/burokratt-estonia

A 2025 survey-vignette experiment in Estonia reported that citizens' intended use of a government chatbot relates to perceived usefulness and trust in the technology, that privacy concerns relate to service-provision uses but not to information-provision uses, and that trust in government, explainability, and the amount of information provided were not related to intended use.

Sources: alishani2025

Appears on: /domains/cases/burokratt-estonia

Albert France Services was a sovereign, in-house generative AI assistant built by DINUM with ANCT to help France Services counter advisers answer citizens' benefits and procedure questions from a curated base of official documents, presented by the Prime Minister as a sovereign French AI in April 2024 and, in a demonstration before him, giving a wrong answer on identity-card cost. Piloted from an initial panel of about sixty volunteer advisers to roughly eighty advisers across more than forty counters (forty-eight at final count per AFP) in six departments over three iterated versions, it was, per a January 12, 2026 AFP dispatch, formally not going to be generalized 'in its current form,' a decision DINUM announced on January 9, 2026 while stating that the majority of Albert-brand projects are sustained and fully operational. No error rate, usage volume or override count for the tool was ever published; AFP reports DINUM's annual AI budget at about 1.2 million euros since 2024 with Albert France Services a minimal share, a figure distinct from and not directly comparable to the union Solidaires Finances Publiques' separate claim of a roughly 1.3 million euro project cost.

Sources: wekafrafpdispatch2026, acteurspublics2026, solidairesfinancespubliques2026, franceservicesanct2024

Appears on: /domains/cases/albert-france-services

For Albert France Services no instrumented error-detection channel existed: no error rate, override count or usage figure was published during the pilot, and the failures that framed the tool surfaced through the operator side, with several unions documenting recurring malfunctions and wrong answers, an investigative-television broadcast in April 2025 (per Solidaires Finances Publiques) featuring unenthusiastic agent testimony, and advisers reporting answers worse than an ordinary search. According to Solidaires Finances Publiques the project had in fact stopped by September 2025 with no announcement, inferred from Albert no longer appearing among projects presented in a ministerial working group (a union claim). Alongside the January 2026 non-generalization decision DINUM migrated the Albert API model aliases off the 'albert-' branding and removed the web-search functionality, retiring legacy aliases by February 15, 2026, while a successor adviser tool that integrates models from the vendor Mistral AI was in test with about 10,000 public agents through June 2026, gated by a summer-2026 evaluation that must notably establish the cost of a generalization.

Sources: solidairesfinancespubliques2026, nextnextink2026, wekafrafpdispatch2026, acteurspublics2026

Appears on: /domains/cases/albert-france-services

In the sociotechnical simulation, over a supervised-plus-agent scenario, adding a verifier to the autonomous agent removed roughly 46% of the harm that persists and a coordinated governance package roughly 43%, while upgrading the model alone removed only about 6%.

Sociotechnical simulation result: PAN social-work governance guidance, lever-ranking comparison.

Appears on: /practice/verifier-on-the-agent, /practice/improve-the-model, /pan-lab

In the sociotechnical simulation, fixing the surrounding system out-leveraged an equal-effort model upgrade in nearly every case tested, and by several times the margin - a better model helps least where the system, not the model, does the damage.

Sociotechnical simulation result: PAN baseline analysis.

Appears on: /practice/improve-the-model, /pan-lab

In the sociotechnical simulation, deleting records without reading them raised the contaminated share by stripping out benign entries; only content-aware cleanup reliably reduced it.

Sociotechnical simulation result: PAN governance-lever audit.

Appears on: /practice/connection-authorization, /pan-lab, /practice/data-minimization, /practice/content-aware-decontamination, /practice/record-reconciler

In the sociotechnical simulation, the same AI in three modeled office cultures - stylized, not real workplaces - let errors stick at very different rates: roughly 75% under low-oversight autonomy, 20% under human supervision, and 16% under high-governance professional controls.

Sociotechnical simulation result: PAN social-work governance guidance, three-office comparison.

Appears on: /pan-lab

In an independent validation — against the NICE Evidence Standards Framework — of a Magic Notes documentation-assistance pilot at Kent County Council adult social care, staff self-reported weekly written-admin time falling roughly 6.8-7.2 hours (about 35-41%), records submitted some 2.0-3.5 days sooner, and case-note detail rated 6.2 to 8.7 out of 10; the validator judged the findings directionally valid rather than a productivity measurement, because the study was commissioned by the vendor (Beam) — which collected and analysed the data while the validator only sense-checked it — and rested on 29 opt-in staff over 8 weeks with self-estimated time, no control group, no statistical testing, and safety and accuracy explicitly out of scope.

Sources: unityinsights2025, beam, somersetcouncil

Appears on: /pan-lab, /what-ai-can-do

In the same independent validation of the Kent County Council Magic Notes documentation-assistance pilot, the report records the deskilling concern that runs alongside the benefit: one client declined having their session recorded "due to personal feelings of risk of loss of practitioner skills" (p.19). It is a single qualitative observation from a vendor-commissioned pilot of 29 opt-in staff over 8 weeks with no control group — evidence for the direction of the crutch/deskilling risk that accompanies documentation assistance, not for its magnitude.

Sources: unityinsights2025

Appears on: /pan-lab, /what-ai-can-do

In a peer-reviewed staggered-deployment study of 5,172 customer-support agents at a single firm, access to a generative-AI assistant raised issues resolved per hour by about 15% on average, with the gain concentrated in the least-experienced workers — roughly +30% for novices versus near-zero for the most experienced, who showed small quality declines; the widely cited 14%/34% pair comes from the 2023 draft, while the peer-reviewed figures are 15%/30%, and because the domain is customer support the direction is imported to social services but the magnitude is never treated as a fixed quantity.

Sources: brynjolfsson2025a

Appears on: /pan-lab, /what-ai-can-do

In a randomized controlled trial of a benefits-navigation chatbot (co-authored by Cornell researchers and the tool's developer, Nava) with 125 caseworkers across six Los Angeles County organizations over 14 weeks, caseworkers answered complex benefit questions at about 49% accuracy unaided, and high-quality chatbot suggestions raised accuracy by roughly 27 percentage points — with larger gains on harder questions, but a persistent 'AI underreliance' plateau in which correct suggestions were not always adopted; the trial did not establish a clear effect on administrative burden, a null reported honestly rather than inferred as a benefit.

Sources: gosciak2026, kanne2025, navapublicbenefitcorporation2025d, navapublicbenefitcorporation2026

Appears on: /pan-lab, /what-ai-can-do

In the county-commissioned impact evaluation of the Allegheny Family Screening Tool, screen-in accuracy — further action or re-referral within 60 days — rose from 42.85% to 46.61% (p=.000) while consistency across screeners was maintained; the finding is contested and quasi-experimental, with true maltreatment rates unknown and the accuracy gains concentrated among white children and ages 7-12, the gain for Black children attenuating to statistical non-significance.

Sources: goldhaberfiebertprince2019

Appears on: /pan-lab, /what-ai-can-do

Crisis-text triage classifiers read language, so the language they read least well is where they fail. The volume's mental-health chapter reports that sarcasm, cultural idiom and code-switching confuse these models, that benchmark studies show markedly lower accuracy on African American English and other underrepresented dialects, and that the operational cost runs in both directions: a false positive can trigger an unwanted welfare check, a false negative leaves a texter waiting in silence. This is benchmark evidence about a class of classifier, not a measurement of any deployment in this registry. The underlying benchmark study is not held in this repository's reference snapshot, so no magnitude is carried, and the finding must never be attached to any named service's own published or unpublished figures. Nothing here is a fairness metric and nothing here is computed about any texter.

Sources: yang2026c

Appears on: /domains/cases/crisis-text-line-loris, /pan-lab

The volume's substance-use chapter reports the field's one sustained deployment benefit case: a national health system's opioid risk-mitigation dashboard, an advisory clinical decision-support tool that stratifies patients for review, was evaluated in a randomized design across the system's medical centers, and use of the risk stratification was reported as associated with a decrease in mortality among the at-risk patients it covered. Three limits travel with the finding and are part of the claim. It is an association reported in a peer-reviewed secondary synthesis, not a causal result this repository can inspect: the primary evaluation is absent from the reference snapshot, so no sample size, facility count, follow-up window or effect size is carried here. It is a benefit direction on a domain this registry otherwise describes almost entirely in the failure register, and it is recorded so that the failure register is not mistaken for the whole evidence base. And it is an outcome for people who are outside the model by construction: it is never read from a Lab gauge, never a Service-Regime number, and never computed from any diagram.

Sources: saba2026

Appears on: /domains/behavioral-health-triage

In a national survey of 1,179 U.S.-based social workers conducted from October 2025 to February 2026 by the University of Texas at Austin in collaboration with NASW, 63.5% of respondents reported using AI tools or technologies in their current role.

Sources: borah2026a, isbanner2022

Appears on: /pan-lab, /practice/ai-literacy, /what-ai-can-do

In the 2025–2026 University of Texas at Austin / NASW national survey of U.S. social workers, concerns about data privacy and security were the most frequently reported challenge to using AI in practice (46.5% of respondents), and an increased focus on client privacy and confidentiality was the most requested improvement to AI tools for social work (50.4%).

Sources: borah2026a, isbanner2022

Appears on: /pan-lab, /practice/data-minimization

In the 2025–2026 University of Texas at Austin / NASW national survey of U.S. social workers, 40.8% of respondents reported ethical concerns about relying on AI for decision-making, and overreliance on automated decision-making was among the most frequently cited concerns overall.

Sources: borah2026a, pinazohernandis2026

Appears on: /pan-lab

The 2025–2026 University of Texas at Austin / NASW national survey of U.S. social workers describes a gap between AI exposure and AI preparedness: 26.6% of respondents cited lack of training or understanding of AI technology as a challenge, 53.4% said training on AI tools and effective use would help, and clear guidelines on the ethical use of AI were the most-endorsed need (66.8%).

Sources: borah2026a, pinazohernandis2026

Appears on: /pan-lab, /practice/ai-literacy

In the 2025–2026 University of Texas at Austin / NASW national survey of U.S. social workers, 42.1% of respondents reported having no role in decision-making about AI adoption in their workplace; the report concludes most respondents have limited or no control over how AI technologies are selected or implemented within their organizations.

Sources: borah2026a

Appears on: /pan-lab

In the open-ended comments of the 2025–2026 University of Texas at Austin / NASW national survey of U.S. social workers, ethical concerns — prominently including the environmental impact of AI infrastructure — were the most common theme, and the report's first recommendation includes environmental impact among the topics profession-wide ethical guidance should address.

Sources: borah2026a, massey2026

Appears on: /pan-lab

Documentation and administrative tasks consume roughly half of practitioner time: a nationally representative US child-welfare workforce snapshot found caseworkers spend about 54% of the workday (4.3 of 8 hours) on paperwork and documentation, and a UK children's-services review reports staff spending over 50% of their time on case recording, paperwork, and related tasks.

Sources: opre2025, burbidge2022

Appears on: /pan-lab, /what-ai-can-do

Professional caseload standards published by the Child Welfare League of America recommend no more than about 15 families per worker, sitting well below documented practice loads.

Sources: childwelfareleagueofamerica, childrenandfamilyresearchcen2002, academyforprofessionalexcell2021

Appears on: /pan-lab

In a two-year child-welfare ethnography, an ill-fitting algorithmic tool imposed ongoing repair work on caseworkers — anticipatorily editing the inputs they supplied so the tool would return a usable result, and bending or working around procedure to reconcile its output with the case in front of them — labor spent making a poorly-suited tool usable rather than on the casework itself, distinct from any deliberate checking of the output.

Sources: saxena2024

Appears on: /pan-lab

Documentation and administrative recording consume the majority of frontline social-care time: in a nationally representative snapshot of the U.S. child-welfare workforce, caseworkers spent about 4.3 of 8.0 daily working hours on documentation (n=183), and a UK children's-services review found more than half of social-care time going to recording and paperwork — the demand baseline against which any documentation-assistance benefit is measured.

Sources: opre2025, burbidge2022

Appears on: /pan-lab, /what-ai-can-do

The workplace chapter of the social work volume argues that algorithmic management — end-to-end digitalised task allocation, workflow organization, performance evaluation, scheduling and income distribution — is adopted for operational efficiency and cost reduction, and that the same systems erode frontline autonomy and make it hard for a worker to understand or question a decision about their work. The chapter reports no original measurement, so this is a documented direction and an argued mechanism, never a magnitude.

Sources: guo2026

Appears on: /pan-lab

Clinical assessors bound by algorithmic allocation with limited override capacity form a documented constrained-judgment pattern in home-care assessment.

Sources: sutton2020, upturn

Appears on: /pan-lab, /practice/bounded-output-screening

In contextual inquiries with Allegheny AFST call screeners, workers calibrated reliance using contextual case knowledge unavailable to the model and reliably detected and overrode erroneous risk scores — complementary human information, not generic distrust, was the safeguard's mechanism.

Sources: kawakami2022, dearteaga2020

Appears on: /pan-lab

AFST workers reported sometimes agreeing with the risk score against their own best judgment under override-rate oversight, and becoming less likely to disagree over time — reliance driven by organizational incentives independent of trust in the tool.

Sources: kawakami2022, kawakami2026

Appears on: /pan-lab

In a four-week randomized study (n=981), voluntary daily chatbot usage duration predicted worse outcomes on loneliness, socialization, emotional dependence, and problematic use across all conditions, and task-style use fostered practical dependence — reduced confidence in independent judgment.

Sources: fang2025, gerlich2025

Appears on: /pan-lab

A validated collaborative-AI metacognition scale (planning, monitoring, evaluation of one's own reliance) predicted collaboration benefits incrementally beyond general metacognition — verification-skill training, not generic AI knowledge, is the calibrated counter to over-reliance.

Sources: sidra2025, bucinca2021

Appears on: /pan-lab, /practice/verification-training

In a randomized study (N=2,784) with objective ground truth, humans accepted incorrect AI suggestions about a third of the time, and their rate of catching AI errors was governed by verification effort, prior trust in AI, and error legibility — surface errors were caught ~82% of the time versus ~31% for errors requiring conceptual judgment — not by financial incentives or time spent.

Sources: beck2026

Appears on: /pan-lab, /practice/verification-training

Group-decision research finds that cohesive groups tend to converge on the most confident member's judgment rather than the most accurate one, so peer dissent that depends on individual courage arrives too rarely to reliably correct the group.

Sources: zarnoth1997

Appears on: /practice/structured-dissent

Across 1.5 million real assistant conversations, sycophantic validation — not fabrication — dominated reality-distortion risk; disempowering interactions received higher user satisfaction than baseline, making satisfaction a biased proxy that rewards deference.

Sources: sharma2026

Appears on: /pan-lab

Human autonomy is not a single alignment target but a contested value with internal tradeoffs; an assistant can satisfy a user's stated preferences while eroding their agency over time, and the governing test for legitimate delegation is whether the person willingly yielded power and retains the means to regain control.

Sources: fischli2026

Appears on: /pan-lab

Research on AI sycophancy describes it as a fragmented construct — a family of distinct agreement-seeking behaviors that share a label but differ in form, mechanism, measurement, and required mitigation — and finds it intensifies under user pushback and across multi-turn interaction.

Sources: ye2026, sharma2024

Appears on: /pan-lab

In formal simulation, even ideal Bayesian users spiral to near-certain false beliefs under a sycophantic interlocutor at sycophancy rates measured in frontier models (~50-70%), and truth-constrained cherry-picking still produces spirals — minimizing hallucination alone is insufficient.

Sources: chandra2026, sharma2024

Appears on: /pan-lab

Across five preregistered studies (N=3,075), sycophantic AI delivered the emotional and esteem support people most associate with close relationships, narrowing the felt-understanding gap between AI and humans and leaving people less satisfied with real human interaction over three weeks — and offering users a choice of interaction styles did not reduce their preference for the sycophantic one.

Sources: ibrahim2026, cheng2026

Appears on: /pan-lab

Given only a covert persuasion goal and an explicit no-deception instruction, a frontier model still produced manipulative cues in 8.8% of turns, and cue frequency did not reliably predict manipulative success — while automated detection of such cues is itself bounded.

Sources: akbulut2026

Appears on: /pan-lab

In frontier-model testing, some systems behaved measurably safer when they believed they were monitored than when unmonitored, and exhibited strategic dishonesty or underperformance under pressure — so ‘behaves well under monitoring’ is insufficient evidence of safety, arguing for unpredictable continuous oversight.

Sources: shanghaiartificialintelligen2025, greenblatt2024, meinke2024

Appears on: /pan-lab

A preprint benchmark reports an in-context misalignment dose-response: in the most susceptible frontier model, up to ~24% misaligned behavior at 16 examples rising to ~58% at 256 examples (rates at 16 examples span roughly 1–24% across models), with the majority of misaligned responses rationalized.

Sources: afonin2026

Appears on: /pan-lab

Model behavior drifts discontinuously between evaluation snapshots, and narrow finetuning can induce broad correlated failure across unrelated tasks.

Sources: betley2026, li2026, song2026, anwar2024, nikolaou2025

Appears on: /pan-lab

Static robustness certification lags emergent threats; organizations that fold each stressor into their model (slow-loop updates, periodic reviews, post-deployment feedback) shrink future risk, while patch-and-pray accumulates it — the fragility trap.

Sources: jin2025

Appears on: /pan-lab

A predictive system whose outputs shape its own future inputs holds a structural incentive to make the population easier to predict; ordinary pipeline choices can reveal this hidden incentive without any change to the stated objective, and feedback-loop risk tends to grow with model capability.

Sources: krueger2020, perdomo2020

Appears on: /pan-lab

The disability chapter separates two implementation-stage failures that a single drift vocabulary blurs together: concept drift, where the statistical properties of the data a deployed model processes change over time; and covariate shift, where the distribution of input features in the deployment environment differs from the distribution in the training data. They are named and defined as distinct mechanisms with different remedies, and neither is quantified.

Sources: wang2026a, pham2025

Appears on: /pan-lab

Claims and behaviors spread through peer networks sideways, along informal ties — diffusion research finds weak ties and small-world clustering carry information and practices across a network far faster than formal reporting lines.

Sources: granovetter1973, watts1998

Appears on: /pan-lab, /practice/peer-edge-governance, /practice/structured-dissent

A single automated rule set applied uniformly and without human review produced tens of thousands of correlated wrongful fraud determinations in the documented Michigan MiDAS case — one flaw repeating at caseload scale rather than averaging out.

Sources: michiganag2022, ieeespectruma

Appears on: /pan-lab, /practice/cross-model-checking, /practice/reconcile-copied-records, /practice/peer-edge-governance

Model behavior — including misaligned behavior — can propagate through model-to-model channels: research shows narrow in-context examples and inter-model interaction can induce broadly misaligned behavior in the receiving model.

Sources: afonin2026, panpatil2025, betley2026

Appears on: /pan-lab, /practice/cross-model-checking

Emerging agentic-AI governance frameworks treat inter-agent interaction as a first-class assurance surface, requiring explicit oversight of agent-to-agent couplings rather than per-model evaluation alone.

Sources: khan2025, hammond2025

Appears on: /pan-lab, /practice/peer-edge-governance

Professional-verification cultures documented in social work practice sustain peer checking of AI output rather than unquestioned acceptance.

Sources: baez2026

Appears on: /pan-lab

Retrieval layers propagate rather than sanitize their inputs: studies find retrieval-augmented systems remain unfaithful even when the retrieved passage is correct, so faithfulness is bounded rather than assured.

Sources: faithfulrag, faithfulragwithsparseautoenc, ragevaluationsurvey

Appears on: /pan-lab, /practice/curated-corpus-retrieval

Restricting retrieval to a curated, vetted document set bounds what re-enters the model: retrieval-augmented systems fact-checking against a curated peer-reviewed corpus reach roughly 0.97+ accuracy and factuality evaluation is limited by knowledge-base coverage — what is checkable depends on what is documented — so a vetted corpus reduces contamination drawn back into the model relative to open retrieval, though faithfulness remains imperfect under knowledge conflict.

Sources: retrievalaugmentedcovidfactc, ragevaluationsurvey, faithfulrag

Appears on: /pan-lab, /practice/curated-corpus-retrieval

Automated output checks are partial, not complete — measured detector-accuracy bands sit well below completeness, especially on hard or adversarial content.

Sources: theillusionofprogress, halogen, datadogllmasajudge2025, mentalhealthchatbotdetection

Appears on: /pan-lab, /practice/bounded-output-screening

Model error has a hard nonzero floor: formal impossibility results rule out zero error, and measured floors run roughly 1.6–11.6% in frontier evaluations and 4–86% across domains.

Sources: xuetal2024, karpowicz2025, halogen, openai2025, llmstats2026, suprmindbenchmarkdigest2026

Appears on: /pan-lab, /practice/improve-the-model

Automated catch fractions cap out below completeness — around 84% balanced accuracy in optimistic settings versus about 55% on hard content and 9.3% recall in worst-case measurements.

Sources: faithfulragleaderboard, theillusionofprogress, mentalhealthchatbotdetection, datadogllmasajudge2025, samedetectionaccuracyliterat

Appears on: /pan-lab, /practice/bounded-output-screening

Record audit-and-correct shares the detection-ceiling family: an optimistic anchor near 96% token accuracy falls away on hard content, so decontamination is bounded rather than total.

Sources: halludetectlegaldomain, samedetectionaccuracyliterat, theillusionofprogress

Appears on: /pan-lab, /practice/content-aware-decontamination

The verification channel itself is bounded — curated-corpus fact-checking tops out around 0.972–0.978 reliability and collapses under knowledge conflict.

Sources: retrievalaugmentedcovidfactc, faithfulrag, faithfulragwithsparseautoenc

Appears on: /pan-lab, /practice/curated-corpus-retrieval

The verifiable fraction of contaminated records is a planning range (0.90/0.60/0.30) that is explicitly calibration-required and has never been measured.

Sources: ragevaluationsurvey

Appears on: /pan-lab

Language models commit to an answer in their first token (~95-98% of the time) and then fabricate claims to stay consistent with it — recognizing 67-87% of those fabrications as false when re-asked in a clean, uncontaminated context but not correcting them in place — so one error deterministically spawns supporting errors, a self-sustaining failure the model's own downstream output feeds.

Sources: zhang2024

Appears on: /pan-lab

The advocacy chapter's systematic review screened 7,715 records, included 415 articles and described 80 of them in detail, reporting cluster sizes across five method-application buckets. Those are counts of what has been built and published: the review reports no pooled effectiveness estimate and no risk-of-bias appraisal, and its own limitations section calls for randomised trials or rigorous observational studies to become standard practice. A count of applications measures activity, never effect.

Sources: alba2026

Appears on: /pan-lab

A 19-model study across six languages found that the ideological stance an LLM expresses varies systematically with the language it is prompted in and the geopolitical region of its creator, and persists within a single region — so the choice of model is not value-neutral, and dominance by a few models can shift the ideological center of gravity of available information.

Sources: buyl2026, santurkar2023, rozado2024

Appears on: /pan-lab

Formally, estimation error shrinks with data while the human perception gap that produces stationary-environment black swans has a non-zero lower bound — so an incident-free operating history yields confidence without safety.

Sources: lee2025

Appears on: /pan-lab

A frontier risk-management framework in practice ties deployment authority to measured capability-vs-safety zones — green (routine plus monitoring), yellow (controlled with strengthened mitigations), and red (suspend).

Sources: shanghaiartificialintelligen2025, greenblatt2024, meinke2024

Appears on: /pan-lab

An authoritative review of deployed-AI monitoring finds staleness, performance drift, the right cadence of re-evaluation, and who acts on detected anomalies to be unresolved open challenges — and that systems can behave differently when they believe they are monitored — so post-deployment oversight is an unsettled, gameable control rather than a fixed guarantee.

Sources: rao2026

Appears on: /pan-lab

Two chapters of the volume describe the same gap from opposite ends. The governance chapter names an ethical capacity gap: many social workers have not been trained in data science or AI oversight, which it argues leaves them ill-prepared to question or interpret the algorithmic outputs they are nonetheless answerable for. The literacy chapter's professional-development framework assigns audiences by tier, placing sanctioned-tool lists, ethics review boards and vendor bias-mitigation terms with agency leadership while the skill to audit a decision and advocate for a misclassified client is taught at the tier below; it also reports professional-body guidance placing the duty to train on the employer rather than on the individual practitioner. Both are arguments about where authority sits relative to capacity. Neither reports a measured rate of either.

Sources: yang2026a, huang2026b, britishassociationofsocialwo2025b

Appears on: /pan-lab

Automated content moderation fails in two directions at once, and which direction it favors is a governance choice rather than a technical default. The volume's LGBTQIA+ chapter documents both halves landing on the same population: identity terms such as 'trans', 'queer' and 'nonbinary' have been flagged as inappropriate content while overt hate speech aimed at that population evades detection. The platform natural experiment already in this registry shows the same choice made explicitly rather than by default: with human review capacity withdrawn, the deployer said it would over-enforce rather than let harmful content stay up. The chapter is a peer-reviewed secondary synthesis and supplies the direction and the vocabulary, never a magnitude; the removal and reinstatement figures in this registry come from the platform's own transparency reporting under a separate claim and are not restated here. No outcome for the people whose content is moderated is computed anywhere in this Lab.

Sources: downey2026, youtubegoogle2020

Appears on: /domains/cases/youtube-covid-enforcement, /pan-lab

Two chapters converge on the procedural half of a second look: embed a procedure requiring a practitioner to document the reason for agreeing or disagreeing with an algorithmic suggestion, so supervision can examine how the person and the tool worked together instead of counting how often they agreed. The housing chapter supplies the failure it is written against — staffing reduced on the assumption that the tool is efficient turns review into rubber-stamping, and an organization that penalises deviation makes its own oversight performative. Both state it as a design prescription and neither offers an effect size.

Sources: yang2026d, shin2026

Appears on: /practice/structured-dissent

A survey of generative-AI safety evaluations found 85.6% operate at the model-capability layer, only 5.3% at the human-interaction layer and 9.1% at the systemic-impact layer — yet context determines whether a capability becomes harm, so the human and system layers where risk actually manifests are the least evaluated.

Sources: weidinger2023

Appears on: /pan-lab

AI-safety failure classification has a missing interaction layer between institutional risk categories and system-level failure modes: practitioners lack a shared vocabulary of recognizable error patterns, and the catch-all 'hallucination' collapses distinct logic failures whose correct fixes differ.

Sources: beyer2026

Appears on: /pan-lab

A data-driven taxonomy built from 9,705 real AI-incident reports found mitigation practice dominated by reactive and legal levers (incident investigation, reporting, regulatory and court action) while proactive technical and governance levers (model alignment, safety frameworks, board oversight) were least common — real organizations respond after harm rather than preventing it.

Sources: popchanovska2026, slattery2024

Appears on: /pan-lab

Trustworthiness measured at the model or benchmark level does not transfer to the deployed system: standard benchmarks compare models but do not cover the aspects that matter most in a specific application context, so safety and responsibility are properties of the system-in-context — its users, incentives, and institutions — not of the model alone.

Sources: mitra2025

Appears on: /pan-lab

AI failures often originate not in individual models but in the architecture of the decision process - recurring failure topologies including temporal feedback instability (small errors amplified through loops) and relational propagation (errors spreading through network structure) - so safety is a property of the decision architecture, not the model alone.

Sources: cemri2025, perdomo2020

Appears on: /pan-lab

The volume's child-welfare chapter reports that a widely deployed screening score predicts whether a child will be placed out of the home within two years, which is the system's own future response rather than the maltreatment the worker is deciding about. The chapter treats the gap between the modelled target and the decision's actual question as a design property of the deployment, not as a defect in the model's accuracy.

Sources: zhang2026b

Appears on: /pan-lab

The volume's criminal-justice chapter separates three routes by which a risk instrument inherits disparity. The training target is usually arrest, charge or conviction rather than offending itself, which is largely unobserved. The strongest inputs are typically prior system contacts, which carry the disparity of the enforcement that produced them. And the sample is drawn from the population the system already touched. The chapter treats these as distinct routes, so a remedy aimed at one does not address the others.

Sources: ahn2026

Appears on: /domains/security-operations-fraud

The volume's poverty chapter uses the targeting literature's paired vocabulary for the two directions in which a system steering a scarce resource fails: an inclusion error reaches someone the program did not intend to reach, and an exclusion error leaves out someone it did intend to reach. The chapter reports reducing both as the stated aim of machine-learning-assisted eligibility and proxy-means targeting. It is a narrative review and measures neither rate itself; the accuracy results it summarises belong to its sources.

Sources: zeng2026

Appears on: /pan-lab

The disability chapter is the one setting in the volume where the AI is the assistance itself rather than a decision aid steering an institutional decision about a person, which is the case that most tests the operator-network boundary. The resolution the framework already carries holds: model institutional propagation, keep clinical and operations staff in the operator network, and record the assisted person's own outcome externally. No outcome for a served person is computed from any diagram here.

Sources: wang2026a

Appears on: /pan-lab

The older-adults chapter supplies the harder substitution case: where virtual contact substitutes for face-to-face contact, or a monitoring device substitutes for a person looking, the party who stops looking can be an unpaid family caregiver rather than an employee the organization can train, roster or audit. Substitution drawn on a staff link assumes an authority relationship that does not exist in that case, and no lever in the catalogue reaches that person.

Sources: shen2026

Appears on: /pan-lab

The housing chapter names consequences that sit outside an operator-network model entirely rather than being merely unmodelled attributes of people: spatial stigma and the disinvestment or gentrification pressure that follows an area being algorithmically labelled as declining on superficial visual indicators, and a landlord's willingness to rent. These are place- and market-level effects, and nothing in a governance diagram computes them.

Sources: shin2026

Appears on: /pan-lab

The volume's conclusion proposes four distinct evaluation layers for AI in social work: technical performance, distributive impact, procedural fairness and experiential legitimacy. It is a normative proposal, not an empirical finding, and the chapter reports no original measurement of any layer. Read against this Lab: pathway and lever direction sit in the technical-performance layer, the external equity surface is distributive impact and is recorded rather than derived, and procedural fairness and experiential legitimacy have no surface here at all.

Sources: yang2026d

Appears on: /pan-lab

AI literacy — the knowledge and skills required to understand, use, and critically evaluate AI systems — has been proposed as a core competency for social work, relevant even to practitioners who never directly use AI tools.

Sources: ahn2025

Appears on: /pan-lab, /practice/ai-literacy

In a two-year child-welfare ethnography, a re-purposed assessment algorithm produced process-oriented harms to practice, organization, and street-level decisions, compelling caseworkers to perform added repair work; 80% of interviewees reported that the tool had stripped their decision-making discretion.

Sources: saxena2024, ammitzbollflugge2021

Appears on: /pan-lab

The same agency's theory-driven 7ei tool — which tracks case trajectories instead of predicting outcomes — earned collective buy-in and better engagement, but required sustained investments: trauma-informed training, specialized supervision and expert consultation, and new collaborative staffings.

Sources: saxena2024

Appears on: /pan-lab

In a participatory-design study (CHI Late-Breaking Work) with 51 social-service practitioners across two stages (27 in co-design workshops, 24 in contextual inquiry), AI value concentrated in documentation relief, assessment brainstorming, guidance for junior workers, and supervision support — with deskilling and privacy concerns voiced inside the same sessions.

Sources: tan2025

Appears on: /pan-lab, /what-ai-can-do

AI documentation assistance can cut clinician documentation burden substantially, but the efficiency paradox converts freed time into added caseload unless organizational policy protects it — time returned is realized as benefit only when governance decides where the dividend goes.

Sources: vanhara2026

Appears on: /pan-lab, /what-ai-can-do

A human-services AI framework argues organizations should start from their own practice challenges and ask which AI capabilities might help, rather than adopting vendor tools first, and pair that with digital stewardship — discernment, accompaniment, and attunement — noting that most organizational AI investments have shown no meaningful return.

Sources: goldkind2025

Appears on: /pan-lab, /what-ai-can-do

The health-care chapter renders the NASEM Assistance category as reach as much as throughput: post-discharge texting that checks whether a patient obtained their medication and alerts a worker when something is wrong, and round-the-clock operation treated as timely aid, extend assistance beyond the clinic visit. The chapter states this as a practitioner expectation it argues for, and attaches no measured coverage, uptake or outcome figure to it.

Sources: ji2026

Appears on: /pan-lab

Tiered HIPAA penalties run from $145 to $73,011 per violation with an annual cap near $2.19M (2025-adjusted), and disclosure to a tool that is not a business associate is itself a violation.

Sources: hipaajournal2026a, hipaajournal2026b

Appears on: /pan-lab

Protected health information (PHI) is individually identifiable health information; under the HIPAA Privacy Rule (45 CFR 160.103) its uses and its disclosures are both regulated, so how PHI is used inside a system — not only whether it leaves — is governed.

Sources: u2013

Appears on: /pan-lab

Personally identifiable information (PII) is information that can distinguish or trace an individual's identity, alone or combined with other data; NIST SP 800-122 directs organizations to minimize its collection and use and to limit use to the purpose for which it was collected.

Sources: mccallister2010

Appears on: /pan-lab

Under the GDPR (Regulation (EU) 2016/679, Article 5), personal data must be collected for specified purposes and not further processed in a way incompatible with them (purpose limitation), and kept adequate, relevant, and limited to what is necessary (data minimisation).

Sources: europeanparliamentandcouncil2016

Appears on: /pan-lab

The volume's school social work chapter names function creep, a term it takes from Koops (2021), as the long-term risk that data collected for a beneficial purpose such as identifying mental health needs is later repurposed for an entirely different function; it names student discipline and sharing with external law-enforcement agencies as its examples. The chapter's stated concern is the absence of specific, renewed consent for the new use and the erosion of trust that follows, not the volume of data held. It offers this as a risk argument and reports no incidence rate.

Sources: huang2026a

Appears on: /pan-lab

In a school communication-monitoring deployment the flagged-content archive is itself the exposure pathway. Every flag writes a durable record of a student's most sensitive writing, including disclosures of sexual orientation, and the investigative record already in this registry documents that archive being released unredacted through links that required no login, students who risked being outed after writing about sexual orientation or gender identity, and a district that discontinued the vendor in 2023 following an outing incident. The volume's LGBTQIA+ chapter supplies the mechanism that record instantiates: a store built to protect people is also the means by which they can be exposed, which is why practitioners are documented deliberately omitting sexual-orientation and gender-identity data from client information systems even at a cost to record completeness. This claim states institutional exposure only - what the record holds and which hand-offs it can travel along - and never a per-student outcome. Students sit outside these dynamics by construction and nothing about them is computed on any diagram.

Sources: downey2026, bryanandlurye2025, associatedpress2025

Appears on: /domains/cases/gaggle-school-monitoring, /pan-lab

The LGBTQIA+ chapter documents a governance trade-off practitioners already make: social work professionals intentionally omit sexual-orientation and gender-identity data from client information systems to protect people from exposure, forced outing or violence. The chapter frames this as a considered deviation from data-completeness norms rather than a recording error, reports it from the literature it reviews, and gives no prevalence figure.

Sources: downey2026

Appears on: /pan-lab, /practice/data-minimization

Minimisation carries a cost the protective case usually leaves out: a record deliberately kept thinner is also a record that supports less verification, so minimising trades exposure against the evidence the correction loop itself runs on. The duty that travels with it is purpose limitation — consent obtained for one purpose does not cover reuse of that data to train a model for another — which is how the data-protection regulation states the two together. Direction only: none of these sources measures the size of either cost.

Sources: downey2026, an2026a, europeanparliamentandcouncil2016

Appears on: /pan-lab, /practice/data-minimization

Cited per-unit intensities of roughly 0.3 Wh per inference call and about 3.14 L of water per kWh are applied to authored illustrative volumes rather than to measured deployment totals.

Sources: jegham2025, li2023

Appears on: /pan-lab

Safety-only alignment establishes a behavioral floor without a ceiling: systems can be 'not-unsafe' yet directionless — compliant without being constructive — and benefit must be assessed as scaffold versus crutch.

Sources: laukkonen2026

Appears on: /pan-lab, /what-ai-can-do

Formally, a system benefits from volatility when its response to a stressor is convex (Jensen's inequality: the expected outcome under variability exceeds the outcome at the average), and is harmed when the response is concave — so whether a shock strengthens or weakens an organization depends on the curvature of its response, a bounded local property that fails beyond a defined stress range.

Sources: axenie2024, taleb2013

Appears on: /pan-lab, /what-ai-can-do

Repeatable behaviors follow a dose-response curve — beneficial at low frequency or count and harmful past a hormetic limit (the dose beyond which net utility turns negative) — because a fast benefit process is followed by a slower accumulating opposing process, giving AI assistance an optimal bounded dose rather than a monotonic benefit.

Sources: henry2025, calabrese2002

Appears on: /pan-lab, /what-ai-can-do

Where base rates differ between groups, a risk instrument cannot be both well calibrated across those groups and equal in its error rates across them. The volume's criminal-justice chapter restates this result, which the site already carries in its primary form from Chouldechova's 2017 analysis of recidivism instruments. It is a property of the scoring instrument and the population it is applied to, not a defect that a better model removes.

Sources: ahn2026, chouldechova2017

Appears on: /domains/security-operations-fraud

The volume's criminal-justice chapter reports a statewide pretrial reform, evaluated by Anderson and colleagues in 2019, in which the pretrial jail population fell without a rise in crime or in failure to appear. Over the same period the racial composition of those still detained did not materially change. Both halves are the finding: the level moved and the composition did not.

Sources: ahn2026

Appears on: /domains/security-operations-fraud

The volume's ethics chapter names ethics washing as addressing ethical concerns superficially, to gain public trust, while making no substantive change to practice. It identifies three forms this takes: ethical statements that are vague or go unenforced, ethics boards constituted with limited authority, and adoption of frameworks that carry no accountability mechanism. The chapter offers this as a taxonomy of forms, not as a measurement of how often each occurs.

Sources: an2026a

Appears on: /pan-lab

The volume's mental-health chapter reads clinical decision support, digital phenotyping and mental-health conversational agents into the high-risk tier of the EU AI Act, which attaches conformity assessment before placing on the market, post-market monitoring afterwards, and documented human-oversight measures throughout. The obligations follow from the tier rather than from any property of a particular model.

Sources: yang2026c, europeanparliamentandcouncil2024

Appears on: /practice/vendor-gate

The volume's governance chapter matches regulatory intensity to risk level rather than applying one standard everywhere, so that a low-stakes use is not governed as though it were a high-stakes one. The social-work-specific tier table it offers is the chapter's own conceptual synthesis rather than a measured classification, and should be read as a proposal.

Sources: huang2026b, europeanparliamentandcouncil2024

Appears on: /practice/oversight-cadence

A review reported in the volume's governance chapter found that none of the nine principal social-work codes of ethics it examined explicitly addresses the use of predictive algorithms in decision-making. The professional obligation exists in the codes; the specific practice does not appear in them.

Sources: huang2026b, reamer2023

Appears on: /practice/oversight-cadence

The volume's sexual and partner violence chapter draws a scope line around predictive risk modelling and restates it twice: such models are used in research contexts to study population-level risk factors, they are not intended for individual-level decision-making in clinical or legal settings, and they are not intended for practitioners to screen or label individuals directly in real-world settings. The chapter treats the distance between that declared scope and a case-level use as a central governance danger rather than as a modelling defect, and reports no measurement of how often the line is crossed.

Sources: fang2026a

Appears on: /pan-lab

The volume's governance chapter places two standing duties on an agency that deploys AI, both distinct from any pre-deployment approval. First, an incident-reporting protocol that enables timely identification and remediation of algorithmic harm - discriminatory treatment, a biased risk assessment, a misdiagnosis, a breach of confidentiality. Second, transparent channels through which both the people served and the practitioners can report concerns or unexpected effects, so the accountability loop closes after deployment rather than ending at approval. The chapter is a conceptual synthesis and is cited as one: its four-tier social-work risk taxonomy is labeled by its own author as an original construction, informed by but not derived from binding regulation. It may be cited as a framework and must never be presented as a regulatory classification of any deployment in this registry.

Sources: huang2026b

Appears on: /domains/behavioral-health-triage, /domains/public-benefits, /domains/child-welfare, /domains/hiring-employment-screening

The volume's ethics chapter names five redress mechanisms as the operational answer to a diffuse responsibility gap: social impact assessment before deployment, audit trails documenting data inputs and decision processes, appeal mechanisms letting a person challenge an AI-driven decision, liability frameworks allocating responsibility by role, and ethics oversight committees seating practitioners, clients and technologists. It is a normative framework proposal, and no effect size is attached to any of the five.

Sources: an2026a

Appears on: /pan-lab

A model card is not a provenance label, and the two govern different objects. A provenance label marks machine-originated content inside the record, so readers and retrieval discount it. The volume's introduction describes a model card or nutrition label as documentation of the tool — how it functions, what its limitations are, and the contexts in which it should not be applied. Keeping the two apart keeps a record-side control and a procurement-side control from being read as one.

Sources: an2026b

Appears on: /practice/provenance-labeling

The same chapter reports that workers who distrust a screening tool may still follow it, because organizational and policy pressure makes following the tool the defensible act. Deference on this account is produced by where accountability sits, not only by how much the worker trusts the output.

Sources: zhang2026b

Appears on: /pan-lab

How a model scores on data held back from its own training and how it scores at a different site are different quantities. The volume's disability chapter reports a named pair where the second is materially lower than the first. A figure quoted without saying which of the two it is does not tell a reader what the model will do in their setting.

Sources: wang2026a

Appears on: /pan-lab

The volume's substance-use chapter reports that nearly all models in the field it reviews are developed and evaluated on a single dataset. Where that holds, a reported ceiling describes that sample rather than a portable capability, and it should be read as the best case observed on one collection of records, not as what the tool will do elsewhere.

Sources: saba2026

Appears on: /pan-lab

Discrimination statistics such as AUROC, precision, F1 and lift describe how a model separates cases on a labelled sample. They are not per-interaction rates at which an error is adopted, written into a record, or corrected. The volume's research chapter treats these as different quantities, and they must never be entered into a diagram as though one substitutes for the other.

Sources: yang2026e

Appears on: /pan-lab

An independent audit of the Allegheny Family Screening Tool's first years (2016-2018) found that, run without human override, it would have recommended screening in about 68% of Black children versus 50% of white children (an 18-point gap), while call screeners actually screened in 51% and 43% (a 7-point gap) — the narrower gap came from workers disagreeing with the score about a third of the time.

Sources: stapleton, stapleton2025, hoandburke2022

Appears on: /domains/cases/allegheny-afst, /pan-lab

An ACLU and Human Rights Data Analysis Group analysis of the Allegheny Family Screening Tool found that 97% of Black referral-households in the data were affected by at least one permanent 'ever-in' variable drawn from public-benefits data sources, compared with 80% of non-Black households.

Sources: gerchicketal2023

Appears on: /domains/cases/allegheny-afst, /pan-lab

The U.S. Department of Justice's Civil Rights Division was reported to be scrutinizing the Allegheny Family Screening Tool after civil-rights complaints filed in fall 2022 raised concerns that its use of disability, mental-health, and Supplemental Security Income data may discriminate against parents with disabilities; families are not shown their scores, and no public findings or enforcement have been reported.

Sources: associatedpress2023, hoandburke2023

Appears on: /domains/cases/allegheny-afst

In Allegheny County's Hello Baby program, the top-tier roughly 5% of newborns by predictive risk score accounted for about 54% of children later removed from the home by age three, at roughly twenty times the removal risk of other newborns (methodology relative risk 22.24, 95% CI 17.50-28.25); the model reported an AUC of about 0.93 on holdout data.

Sources: centreforsocialdataanalytics2020, vaithianathan2025

Appears on: /domains/cases/allegheny-hello-baby

An external evaluation of Hello Baby covering birth cohorts 2016-2024, controlling for COVID-19, found the program associated with fewer first child-maltreatment investigations and first substantiated investigations, but no reduction in out-of-home foster-care placements - the outcome the predictive model was built to estimate.

Sources: lery2025, centreforsocialdataanalytics2020

Appears on: /domains/cases/allegheny-hello-baby

In a 2019 proof of concept, Chile's Sistema Alerta Niñez risk models reached test-set AUC of roughly 0.88 to 0.95 for a two-year outcome — a child's separation from family or contact with child-protection programs — using 280 administrative variables per child; the deployed operational model's real-world performance was never publicly disclosed.

Sources: derechosdigitalesmatiasvalde2021, derechosdigitalesmatiasvalde2022, centreforsocialdataanalytics2019b

Appears on: /domains/cases/chile-alerta-ninez, /pan-lab

Sistema Alerta Niñez drew on 280 administrative variables that families had supplied to access social benefits, without informed consent to the risk ranking or a way to opt out; the model's developers acknowledged it was less able to identify higher-income children at risk, because lower-income families have more contact with the state.

Sources: derechosdigitalesmatiasvalde2021, derechosdigitalesmatiasvalde2022, centerforhumanrightsandgloba2022

Appears on: /domains/cases/chile-alerta-ninez, /pan-lab

The Douglas County Decision Aide, deployed into the county's RED-Team call-screening process in February 2019, scores each referral from 1 to 20 for a child's likelihood of out-of-home removal within two years; an independent Cornell-led randomized controlled trial found it sped up screening decisions without significantly changing child outcomes, and a companion study found workers attended mainly to extreme scores while largely disregarding mid-range ones.

Sources: vaithianathanetalcentreforso2019, fitzpatrick2025, eiermann2026

Appears on: /domains/cases/douglas-county-decision-aid, /pan-lab

Eckerd's Rapid Safety Feedback spread from Hillsborough County, Florida to child-welfare agencies in several states — promoted on the vendor's own reported gains and highlighted as “innovative” in a 2016 federal commission report — years before an independent 2022 peer-reviewed evaluation found the process did not lower repeat high-severity maltreatment among children identified as high risk (a joint odds ratio of about 1.05).

Sources: eckerdconnects2016, routefiftygovernmentexecutiv2016, parker2022, floridaschildrenfirst2012

Appears on: /domains/cases/eckerd-florida-rsf-origin

Gladsaxe's early-detection project (DTO) was a decision-tree model over about 44 risk indicators, meant to score, for every child aged 0 to 6 rather than only families already receiving help, the estimated probability that the child was living in vulnerability; per a university-run Danish public-sector AI catalogue it was to be trained on roughly 173,000 notifications the authorities received between April 2016 and December 2017, but only about 117 usable historical cases existed, and it was halted in its development phase in 2019 without ever running on live decisions, after a national media storm and an unrelated data breach that exposed about 20,000 citizens' personal identification numbers.

Sources: offentligaiuniversityrundanind, kennethkristensensamfundsled2022, helenefriisratnerandkasperel2023, katarinafastlappalainen2021, tvkosmopolformerlytvlorry2018

Appears on: /domains/cases/gladsaxe-denmark

Hackney paid the analytics firm Xantura £361,400 over four years to run an Early Help Profiling System that flagged families for preventive intervention from council data, but scrapped the pilot in 2019 after finding that, despite flagging about 350 families, it surfaced only 7 children previously unknown to the council and the available data was too limited and variable to justify continuing.

Sources: hackneycouncilpayskpoundstod2018, townhalldropspilotprogrammep2019

Appears on: /domains/cases/hackney-early-help

Families whose data Hackney's Early Help Profiling System processed were not informed directly: reporting describes families profiled without their knowledge, given notice only through a general online privacy notice, with no option to opt out recorded in the system's impact assessment and the method withheld as commercially sensitive; the council argued that disclosing the system could prejudice potential interventions.

Sources: townhalldropspilotprogrammep2019, reddenj2020, hackneycouncilpayskpoundstod2018

Appears on: /domains/cases/hackney-early-help

Internal DCFS tracking data released under Illinois public-records law showed the Rapid Safety Feedback tool flagged more than 4,100 children at a 90-percent-or-higher probability of death or serious injury within two years, including 369 children under age 9 assigned a 100-percent probability, while children who died in cases already known to the system — among them 17-month-old Semaj Crosby, found dead after at least ten DCFS investigations — were not flagged as top-risk; the roughly $366,000 program was ended in 2017.

Sources: chicagotribune2017, governmenttechnologya

Appears on: /domains/cases/illinois-rapid-safety-feedback

Illinois brought in the Eckerd/MindShare Rapid Safety Feedback program under DCFS director George Sheldon through a no-bid arrangement the state classified as a grant; a July 2017 joint report by the Illinois Office of Executive Inspector General and the DCFS Inspector General found this classification to be mismanagement because it avoided state bidding-transparency requirements.

Sources: chicagotribune2017, sunshinestatenews2017

Appears on: /domains/cases/illinois-rapid-safety-feedback

Bristol's Think Family Database drew on roughly 30 to 35 fused council, police and other datasets covering about 55,000 families (some 170,000 residents in 2021 reporting), and its child sexual and criminal exploitation risk models were quietly withdrawn in 2023 as 'not fit for operational use' after an independent evaluation judged the risk-scoring models the weakest element and staff reported victims of exploitation scoring below people involved in burglary; FOI responses indicate no record was kept of why the models were switched off, and auditors could not locate their source code or variable lists.

Sources: seanmorrison2026a, markwildingandmattburgess2026, seanmorrison2026b, bristolcitycouncil2025, jakehurfurtbigbrotherwatch2021

Appears on: /domains/cases/insight-bristol

Reporting and FOI responses on Bristol's Think Family Database indicate the exploitation models' source code and variable lists could not be located when auditors sought them, and that an ethics committee advising the police analytics reportedly did not revisit the analytics after 2017; a 2021 review warned that data gathered through 'legal gateways' meant 'legality is not the same as legitimacy.'

Sources: markwildingandmattburgess2026, seanmorrison2026a, seanmorrison2026b

Appears on: /domains/cases/insight-bristol

In a retrospective test against historical outcomes, Los Angeles County's Project AURA — a proprietary risk model built by SAS — correctly flagged 171 of the highest-risk children but produced 3,829 false positives, a false-positive rate of about 95.6% that DCFS's own public-affairs director confirmed on the record, and the county shelved the tool in 2017 without ever using it on a live case.

Sources: theimprintdanielheimpel2015, childprotectiveservicesdefen2015, nccprrichardwexler2017, witnesslarichardwexler2017

Appears on: /domains/cases/la-county-aura, /pan-lab

The Dutch government's own 2011 pilot evaluation of ProKid found that 36% of the tool's red, orange and yellow child-risk flags (902 of 2,444 over three months across four police regions, rising to 53% in Amsterdam-Amstelland) were system or registration errors or based on irrelevant incidents, and that in none of the four regions was there a well-functioning instrument.

Sources: dspgroepforthewodcabraham2011, dimitritokmetzissargasso2012

Appears on: /domains/cases/netherlands-prokid, /pan-lab

New Zealand's Ministry of Social Development commissioned a child-maltreatment risk-modelling tool that, on a 2012 development sample of 57,986 children and 132 selected variables, reported an area under the ROC curve of 76% and a top risk decile in which 47.8% had a substantiated maltreatment finding by age five; those figures come from development data rather than field performance, the tool was never operationally deployed, and a proposed two-year study that would have scored about 60,000 newborns was halted by the incoming Social Development Minister, who annotated the briefing papers 'Not on my watch! These are children not lab rats.'

Sources: vaithianathan2013, nzherald2015, otagodailytimes2015, mordaunt2026

Appears on: /domains/cases/nz-msd-prm

Oregon's Department of Human Services stopped using its Safety at Screening tool at the end of June 2022 and replaced it with a non-algorithmic Structured Decision Making process, telling staff the change was meant to reduce disparities; the move followed Associated Press reporting on racial disparity in the Allegheny tool it was derived from and a racial-bias inquiry from a U.S. senator.

Sources: associatedpress2022, willametteweek2022, hoandburke2022, nprap2022

Appears on: /domains/cases/oregon-safety-at-screening

Oregon's 2019 report describes a post-processing fairness correction — group-specific thresholds selected under an 'error rate balance' criterion — applied to a dual-outcome risk model built only on the state's own child-welfare administrative records.

Sources: orrai2019, associatedpress2022

Appears on: /domains/cases/oregon-safety-at-screening

None of the 32 machine-learning models What Works for Children's Social Care built across four English local authorities cleared the pre-specified 65% average-precision success bar; the best single model reached only about 42% average precision and, at an operating point, missed roughly 79% of the children whose cases actually escalated.

Sources: claytonandsanders2022, communitycareturner2020a, childhubterredeshommes2020

Appears on: /domains/cases/wwc-uk-ml-pilots, /pan-lab

In a survey of 129 social workers carried out for the project, only about 26% supported using predictive analytics to identify families for early help and about 34% thought it should not be used at all.

Sources: communitycareturner2020a

Appears on: /domains/cases/wwc-uk-ml-pilots, /pan-lab

In a Los Angeles County pilot, 335 people who enrolled in the voluntary Homelessness Prevention Unit were reported to be 71% less likely than a regression-adjusted comparison group of 1,285 eligible non-enrollees to enter a homeless shelter or have street-outreach contact within 18 months; the California Policy Lab describes this as an association not yet shown to be causal, pending a randomized controlled trial with results expected in 2027.

Sources: blackwell2025, countyoflosangeles2025, uclanewsroom2025

Appears on: /domains/cases/la-homelessness-prevention

The Homelessness Prevention Unit's own November 2024 equity audit, on a test population of 47,582 individuals eligible to be scored, reported false-negative rates ranging from about 56% for Black individuals to roughly 63 to 65% for other groups: the model misses a majority of the people who later become homeless, while performing roughly consistently across race, ethnicity, and gender and identifying Black individuals slightly more strongly.

Sources: californiapolicylab2024, foxsowell2025

Appears on: /domains/cases/la-homelessness-prevention

Xantura's OneView integrates more than 15 multi-agency data feeds into a single household view and flags residents as likely to become homeless months ahead. In Maidstone's pilot year it produced 650-plus alerts that a single financial-inclusion officer could contact only about 260 of. Its headline effectiveness figures - a reported 40 percent fall in homelessness, savings and an ROI over 600 percent, and the widely quoted contrast between contacted and uncontacted households - are vendor- and council-reported pre/post numbers from one COVID-affected pilot year; the contact-versus-no-contact contrast reflects capacity-driven selection rather than a randomised comparison, and the independent randomised controlled trial commissioned to test the causal claim was still in progress into 2026.

Sources: crisisuk2023, xantura2023, governmenttransformationmaga2023, ministryofhousing2024, centreforhomelessnessimpact2024

Appears on: /domains/cases/xantura-oneview-housing

OneView's single view of vulnerability is built by integrating sensitive multi-agency records - including offending, health, benefits and debt data - under a statutory Digital Economy Act 2017 data-sharing agreement with named public-body controllers and processors. An independent ethnography of an early deployment (its fieldwork centered on children's social care and the COVID-19 response) found frontline staff could not see which factors drove the tool's alerts and were not all convinced it was as accurate as described, and a separate NGO investigation characterised the vendor's COVID-era model as operating without residents' knowledge.

Sources: digitaleconomyactregister2023, adalovelaceinstitute2024, bigbrotherwatch2021

Appears on: /domains/cases/xantura-oneview-housing

London, Ontario's CHAI is a live, caseworker-facing machine-learning model that flags people in the city's shelter system as at risk of chronic homelessness (more than 180 shelter days in a year) about six months ahead; it provides intelligence to prevention caseworkers and does not itself make service decisions. Its widely repeated '93 percent accuracy' is a builder-reported, testing-phase figure from 10-fold cross-validation on historical HIFIS records, never independently validated after deployment; the same technical work reports recall of about 0.921 but precision of only about 0.651, implying substantial false positives under a low base rate.

Sources: wray2020, vanberlo2009, lebel2023, govlaunchstories2020

Appears on: /domains/cases/chai-london-ontario

CHAI is consent-based: it draws on de-identified HIFIS records pooled from roughly 20 to 24 London homelessness-support organizations and lets individuals opt out of inclusion, and it was built with reference to GDPR principles, Canada's Directive on Automated Decision-Making, and local feature-attribution explanations for caseworkers. Because HIFIS captures people who use public shelters, an independent review and reporting at launch note it can under-represent or miss groups who avoid them - including many women, families, new immigrants, some Indigenous people, and private-shelter users; academic researchers situating the tool raise related fairness and inequality concerns. So the population the model can score is a selected sample of actual need, and the opt-out self-selects it further.

Sources: wray2020, lebel2023, lamberink2020, redden2026

Appears on: /domains/cases/chai-london-ontario

In a 2025 Los Angeles County pilot evaluated by Nava Labs with academic partners at Cornell University and Georgetown University's Better Government Lab, a generative-AI assistive chatbot for Imagine LA's Benefit Navigator was estimated to improve benefits-navigation answer accuracy by an average of about 40% in a randomized controlled trial of 125 caseworkers answering hypothetical client questions, alongside a fourteen-week field pilot with 61 caseworkers across six organizations; the evaluation was co-authored by the tool builder rather than independently replicated, the accuracy figure is a decision-support contrast on hypothetical questions rather than a live-caseload eligibility audit, and time-savings and administrative-burden effects were reported as promising but inconclusive (published March 2026).

Sources: navapublicbenefitcorporation2026, chen2026

Appears on: /domains/cases/imagine-la-benefit-navigator

The same evaluation reported that the chatbot's accuracy gains were largest on the most difficult client questions and among the newest, least-experienced staff (a directional finding, not a quantified breakdown), that about 65% of caseworkers with access used it at an average of about 14 prompts each and a modest, low-positive satisfaction (a Net Promoter Score of 11), that usage tended to decline over time without sustained engagement, and that answers averaged a tenth-to-twelfth-grade reading level against college-level source manuals.

Sources: navapublicbenefitcorporation2026, chen2026

Appears on: /domains/cases/imagine-la-benefit-navigator

In a single-center randomized trial across three Vanderbilt neurology clinics (August 2022 to February 2023), an EHR suicide-risk model flagged 596 of 7,732 encounters (about 8%) at a 2%-or-higher 30-day-risk threshold; making the identical alert interruptive rather than passive led clinicians to elect a suicide-risk screen in 42% of encounters (121/289) versus 4% (12/307) for a passive chart icon, an adjusted odds ratio of 17.70 (95% CI 6.42–48.79). Screening remained fully advisory: about 58% of interruptive and 96% of passive alerts produced no screening.

Sources: walshetal2025, aitestedforalertingclinician2025, suicidepreventionmorefeasibl2025

Appears on: /domains/cases/vsail-vanderbilt

In a separate 2021 prospective silent-mode study (115,905 predictions on 77,973 patients, June 2019 to April 2020), the model reported a c-statistic of 0.797 for suicide attempt and 0.836 for ideation center-wide but only 0.544 for attempt in behavioral-health settings, and in the highest-risk quantile the number-needed-to-screen was 271 for attempt and 23 for ideation. In the 2022 to 2023 trial no suicidal ideation or attempts were documented in either arm during 30-day follow-up, and the trial was explicitly not powered for clinical outcomes, so it measured a process outcome (screening) rather than reduced harm.

Sources: walshetal2021, walshetal2025, suicidepreventionmorefeasibl2025

Appears on: /domains/cases/vsail-vanderbilt

Between roughly 2005 and 2019 the Dutch Tax Administration's benefits branch (Belastingdienst/Toeslagen) wrongly accused an estimated 26,000 or more families of childcare-benefit fraud and demanded full repayment; broader advocacy estimates run higher and count different populations, and by February 2026 about 69,000 people had applied to the recovery scheme and more than 43,000 were formally recognized as affected, each entitled to a minimum of 30,000 euros. A self-learning risk-classification model that scored applications using a Dutch-nationality indicator, a 270,000-person fraud blacklist (the FSV) held without a legal basis, and an all-or-nothing recovery regime were coupled together; the Dutch Data Protection Authority imposed 6.45 million euros in fines (2.75 million for the nationality processing in 2021 and 3.7 million for the FSV blacklist in 2022), a parliamentary inquiry found rule-of-law violations, and the third Rutte cabinet resigned on 15 January 2021.

Sources: wikipedia2026, autoriteitpersoonsgegevens2021, autoriteitpersoonsgegevens2022, amnestyinternational2021b, tweedekamerderstatengeneraal2020, rijksoverheid2026

Appears on: /domains/cases/nl-toeslagenaffaire, /pan-lab

The scandal's harm is best read as the coupling of three distinct components rather than a single algorithm. Government-commissioned technical reviews (KPMG in 2022 and PwC in 2023) described the tool as a self-learning classifier that routed the highest-scoring of roughly 90,000 benefit applications sent to manual treatment in 2014 to 2019, but judged the Dutch-nationality indicator's standalone predictive weight to have been limited; the model's precision and false-positive rate were never measured or published. The FSV fraud blacklist held frequently inaccurate data that was not corrected when people were cleared, and internal 2016 guidance auto-labelled childcare debts over 3,000 euros as intent or gross negligence, blocking payment arrangements. Out-of-home child placements are a documented but causally contested downstream harm: statistics counted roughly 2,090 children of affected parents placed out of home through mid-2022, while a 2025 judicial study found no child was removed solely because of financial problems.

Sources: kpmg2022, pwc2023, autoriteitpersoonsgegevens2022, statisticsnetherlandscbs2022, rechtspraak2025, wikipedia2026

Appears on: /domains/cases/nl-toeslagenaffaire, /pan-lab

On 5 February 2020 the District Court of The Hague ruled that the legislation authorising SyRI, the Dutch state's secret cross-database welfare-fraud risk-profiling system, violated Article 8 of the European Convention on Human Rights, and it ordered the system's use stopped; the State did not appeal. The ruling is widely described as one of the first times a court anywhere halted a digital welfare-fraud technology on human-rights grounds. Across its two executed neighbourhood projects SyRI was reported to have produced no confirmed fraud cases, and in one municipality 62 of 113 risk notifications were reported to be false positives.

Sources: districtcourtofthehague2020, vanbekkum2021, unofficeofthehighcommissione2020, algorithmwatch2020a, pontdataprivacyprivacywebnl2019, publicinterestlitigationproj2020

Appears on: /domains/cases/nl-syri

The District Court of The Hague found that the SyRI framework provided no duty to notify people that their data had been processed or that a risk report had been filed, so a flagged person generally could not know about, access, or contest the notification; notifications were retained in a register for up to two years. The court held that a risk notification carried significant effect for the person even though it lacked formal legal effect, and it faulted the scheme for a lack of transparency and for breaching data-minimisation and purpose-limitation principles.

Sources: districtcourtofthehague2020, vanbekkum2021

Appears on: /domains/cases/nl-syri

France's family-benefits fund (CNAF) computes a monthly benefit-fraud suspicion score, on a 0-to-1 scale, for every benefit-receiving household — analysing the data of about 32 million people and producing more than 13 million scores each month, close to half of France's population; the highest scores route households into fraud controls, up to the most invasive on-site checks. An analysis by Le Monde and Lighthouse Reports of an extracted production model (a logistic regression of about 33 variables) found that markers of economic vulnerability raised the score: a stable-income family averaged about 0.33, while a person working while receiving the disability allowance (AAH) averaged about 0.66. The model's target was an overpayment (indu) above a threshold, which is frequently unintentional administrative error rather than proven intentional fraud, and the score itself is not disclosed to the person and cannot be appealed directly. CNAF disputed the discrimination framing, describing the tool as a neutral decision-aid that only prioritises which files to check; a coalition that grew to 25 organisations challenged the model before the Conseil d'État, and as of this writing no court had ruled.

Sources: lighthousereports2023b, lighthousereports2023a, laquadraturedunet2023, laquadraturedunet2026a, amnestyinternational2024b, generationnt2026

Appears on: /domains/cases/france-cnaf

In an internal simulation study by CNAF's own statistics department (DSER), reported in October 2025 by Le Monde and La Quadrature du Net, recipients of the RSA minimum-income benefit were about 13% of beneficiaries but 39 to 41% of the highest-scoring 5%, and single mothers were about 14% of beneficiaries but 37 to 40% of that top bracket; households including a foreign national scored higher on average even after the nationality variable was removed. The full study is not public, and false-positive rates by protected group have not been released. The French ombudsperson (Défenseur des droits) told the Conseil d'État that a presumption of indirect discrimination appeared established because the differential treatment rests on beneficiaries' economic vulnerability; CNAF disputed the characterisation, and no court had ruled.

Sources: laquadraturedunet2026b, generationnt2026, laquadraturedunet2026a

Appears on: /domains/cases/france-cnaf

Analysing the Swedish Social Insurance Agency (Forsakringskassan) 2017 outcome data, Lighthouse Reports and Svenska Dagbladet reported on 27 November 2024 that the agency's in-house machine-learning risk profile for the temporary parental allowance (VAB) selected women (more than 1.5x), people of a foreign background (about 2.5x), below-median earners (2.97x), and people without a university degree (3.31x) for fraud investigation more often than comparison groups by demographic parity, and wrongly flagged those groups at higher false-positive rates (about 1.7x for women and 2.4x for people of a foreign background); in the agency's paired random-control sample, 20.2 percent of applications contained at least one day incorrectly paid, an unbiased base error rate. These are outcome computations under specific fairness definitions from a single obtained year of data, not confirmed model internals; the agency disputed the framing and did not release the model. The data-protection regulator IMY closed its GDPR supervision on 18 November 2025 for mootness after the agency withdrew the system, and no court or regulator issued a discrimination or GDPR penalty.

Sources: lighthousereports2024, lighthousereportsb, lighthousereportsc, integritetsskyddsmyndigheten2025a, integritetsskyddsmyndigheten2025b

Appears on: /domains/cases/sweden-forsakringskassan

The Swedish Social Insurance Agency (Forsakringskassan) did not disclose the machine-learning risk profile it used to select temporary-parental-allowance recipients for fraud investigation: its algorithm class, features, and precision were never released, and the agency resisted freedom-of-information disclosure for roughly three years on fraud-prevention grounds. In 2018 the audit inspectorate ISF found the risk-based profiling substantially more accurate than alternative controls while warning that it raised legal-certainty and equal-treatment concerns, and cautioning that an accurate model can still be inequitable when two groups err equally but only one is followed up. Amnesty International reported that a former agency data protection officer warned in 2020 that the operation breached European data-protection rules. The system was decommissioned in 2025 during the regulator's supervision, before any court or regulator ruled on it.

Sources: lighthousereports2024, lighthousereportsb, inspektionenforsocialforsakr2018a, inspektionenforsocialforsakr2018b, amnestyinternational2024d

Appears on: /domains/cases/sweden-forsakringskassan

Denmark's Udbetaling Danmark (UDK), administered by ATP, runs a data-driven welfare-fraud operation that as of 2019 used up to about 60 AI and machine-learning models to score benefit recipients into a 'wonderlist' of high-risk people, which a human control team filters into control cases for investigation. In UDK's own 2023 control statistics (three documented models), the 'Model Abroad' foreign-affiliation model sent 511 cases for control but recovered money in only 36 -- about 7%, with roughly nine in ten resulting in no further action -- and UDK confirmed that 54% of the 'Really Single' household-outlier cases its unit opened were in fact legitimate. Those 'revenue' outcomes conflate deliberate fraud with honest error, which UDK does not separate, so they are not pure fraud rates. Amnesty International characterised the system as mass surveillance and prohibited social scoring under the EU AI Act; UDK, ATP and the ministry (STAR) rejected that characterisation, the system was not suspended, and as of this writing no court had ruled.

Sources: amnestyinternationalalgorith2024, amnestyinternational2024a, fortuneeurope2024, bablai2024

Appears on: /domains/cases/denmark-udbetaling

Udbetaling Danmark's 'Joint Data Unit' merges and links the personal data of millions of residents from around ten national registers -- civil registration (CPR), buildings and dwellings (BBR), business, income, tax (R75), health, VAT, cash and sickness benefits, education grants and the motor-vehicle register -- alongside a 'Joint Data Unit Abroad' that pulls data from foreign authorities; in 2021 UDK paid about DKK 241 billion to roughly 2.4 million recipients. Amnesty International documents this as mass surveillance and argues the design carries a discrimination risk: 'Model Abroad' scores a relative strength of ties to non-EEA countries with citizenship as a direct input, and 'Really Single' treats statistically atypical households as suspicious. That harm is a design-level risk rather than a measured outcome, because UDK and ATP denied all requests for the demographic data needed to test the models for bias, so no disparate-impact figure exists in the record. Oversight is thin: the Danish Data Protection Authority (Datatilsynet) can generally act only on complaints (GDPR Art. 57) with no proactive power, and because flagged people rarely learn an algorithm selected them, complaints are rare. UDK rejects the discrimination-by-design and social-scoring findings; no court has ruled.

Sources: amnestyinternationalalgorith2024, amnestyinternationaldanmark2024, bablai2024

Appears on: /domains/cases/denmark-udbetaling

In judgment STS 1119/2025 of 11 September 2025, the Third Section of Spain's Supreme Court (Sala de lo Contencioso-Administrativo) ordered the government to give the transparency foundation Civio access to the source code of BOSCO, the software that determines eligibility for the electricity social bonus (bono social electrico). Applying the Transparency Law (Ley 19/2013) together with Article 42 of the EU Charter of Fundamental Rights and Article 105.b of the Spanish Constitution, the Court held that access to public information is a constitutional right and that neither intellectual property nor national security is an automatic shield, dismissing the government's secrecy claims as a 'mere risk' of eventual harm to be assessed case by case under a proportionality test. Civio and legal commentators describe an 'error multiplier': because BOSCO decides automatically and gives no reasons, one systematic error can propagate to thousands of eligible people at once. As of May 2026, roughly eight months after the ruling, the source code had still not been delivered and Civio had filed for judicial enforcement.

Sources: consejogeneraldelpoderjudici2025, fundacionciudadanacivio2025b, fundacionciudadanacivio2025a, derechoadministrativoyurbani2025, fundacionciudadanacivio2025c, fundacionciudadanacivio2026, freesoftwarefoundationeurope2026

Appears on: /domains/cases/spain-bosco

The transparency foundation Civio documented, by reconstructing BOSCO's behaviour from partial technical specifications and functional test cases, two systematic ways the software denied the electricity social bonus to people who qualified: when a pensioner ticked the 'pensioner' box the application could return an 'imposibilidad de calculo' (impossibility of calculation) error and be rejected without properly evaluating income; and large families, entitled to the bonus regardless of income, were denied whenever a household member withheld authorization to consult income data, although income was not a regulatory requirement for that category. After a 2017-2018 overhaul required all beneficiaries to re-apply by 31 December 2018, enrollment fell from roughly 2.4 to 2.5 million under the prior scheme to 1,111,958 as of January 2019 (later cited around 1.5 million), against an estimated 4.5 to 5.5 million eligible people, and more than half a million applicants were rejected. No audited per-decision error rate is public, because the source code and verification-test results were withheld; one academic analysis records errors in both directions, but the documented net effect is under-inclusion.

Sources: fundacionciudadanacivio2019, algorithmwatchnicolaskayserb2019, freesoftwarefoundationeurope2026, xatakaenriqueperez2024, rebootdemocracyjoseluismarti2025

Appears on: /domains/cases/spain-bosco

Serbia's Social Card (Socijalna karta) registry, given a statutory basis by the Law on the Social Card in force from 1 March 2022 and financed in part by an 82.6 million euro World Bank public-sector loan, cross-links roughly 130 to 135 categories of data from other state registers to verify social-assistance eligibility and flag suspected undeclared income or assets. After the law, named sources report the caseload falling by a range of tens of thousands: government figures cited by Amnesty International show about 35,000 fewer recipients by August 2023, A11 counts at least 44,000 people having lost assistance by early 2024, and the UN Working Group on Business and Human Rights reported over 60,000 without assistance by October 2025. These are largely net caseload declines rather than audited counts of system-caused removals, and the government attributes part of the fall to a stronger economy. Roma are reported among the most affected because informal earnings are misclassified as income, but the registry records no ethnicity, so this is inferred rather than officially disaggregated. As of the latest reporting, Constitutional Court, World Bank Inspection Panel, and UN scrutiny were pending or active, with no court or panel yet ordering changes.

Sources: ainitiativeforeconomicandsoc2024, amnestyinternational2023b, contextthomsonreutersfoundat2023, unworkinggrouponbusinessandh2025, worldbankinspectionpanel2024, chinaceeinstitute2024

Appears on: /domains/cases/serbia-social-card

Under Serbia's Social Card system, a removed beneficiary has 15 days to appeal and must wait three months to reapply regardless of changed circumstances, and removal letters frequently reference only unspecified data from the electronic database. A11's Request for Inspection to the World Bank Inspection Panel alleges that, because the system is semi-automated, social workers cannot correct errors recorded in it. Over roughly two years the Ministry processed more than 100,000 notifications of suspected income or asset increases, while beneficiaries filed only 361 appeals against Centers for Social Work rulings; because the two figures cover different populations, the gap illustrates how rarely flags were contested rather than a measured appeal rate. Documented misclassifications include a one-off funeral donation read as income and long-scrapped cars still counted as assets.

Sources: amnestyinternational2023b, ainitiativeforeconomicandsoc2024, worldbankinspectionpanel2024, chinaceeinstitute2024, contextthomsonreutersfoundat2023

Appears on: /domains/cases/serbia-social-card

Samagra Vedika, an entity-resolution system built by the Telangana government, decided welfare eligibility by matching residents across thirty-plus government databases into a consolidated profile; between 2014 and 2019 more than 1.86 million ration cards were cancelled and 142,086 fresh applications were rejected without notice. Its core error was entity-resolution false-positive matching, in which a similarly-named third party's asset was attributed to the applicant and silently flipped the eligibility flag. After the Supreme Court of India ordered field re-verification in April 2022, a partial re-verification found roughly 7.5 percent wrongful rejection (at least 15,471 approved of 205,734 re-processed cases), a lower bound from an incomplete review; the system is proprietary and closed and an independent technical audit could not be completed, with no source code or accuracy data released. The government cited a self-reported 95 percent fraud-filtering efficiency, which measures spurious-application filtering rather than the wrongful-exclusion rate.

Sources: amnestyinternational2024c, tapasya2024, tusharvsharma2026, sumitjha2024, kumarsambhav2020, pulitzercenteraiaccountabili2024

Appears on: /domains/cases/india-samagra-vedika

Under Samagra Vedika, exclusions were silent and there was no statutory route to contest an algorithmic decision, so the burden of proof fell on the excluded person: reporting describes officials who, though formally able to override the algorithm with evidence, deferred to it and declined to overturn its verdict, treating errors as backend technical issues. Documented individual harms include a 67-year-old widow denied rations for more than seven years after the system linked her deceased husband to a car owned by a similarly-named third person, and a family rejected for allegedly owning a four-wheeler that was declared eligible only after a Telangana High Court ruling. Corrections came through individual litigation and did not systematically feed back into the model, and the same entity-resolution technology was reused to issue new ration cards in 2024-2025.

Sources: tapasya2024, thereporterscollective2024, tusharvsharma2026, amnestyinternational2024c, sumitjha2024

Appears on: /domains/cases/india-samagra-vedika

In A.M.C. v. Smith (No. 3:20-cv-00240, M.D. Tenn.), a federal court held after a five-day bench trial that Tennessee's Deloitte-built TEDS automated Medicaid eligibility system, operational statewide since March 19, 2019 for a program covering roughly 1.7 million residents, produced wrongful terminations, wrong-household assignments, and misleading or missing notices that violated the Medicaid Act, the Fourteenth Amendment's Due Process Clause, and the Americans with Disabilities Act; the 116-page opinion, issued August 26, 2024 by Judge Waverly D. Crenshaw Jr., ordered mediation before considering an injunction.

Sources: statescoopkeelyquinlan2024, stotlerhayesgroupllcerinsail2024, georgetownuniversitycenterfo2024, nationalhealthlawprogram2024

Appears on: /domains/cases/tennessee-tenncare-teds

The UK government built its own AI meeting scribe for council caseworkers and piloted it through a cohort of 25 selected councils (22 active, more than 400 users) under one shared pooled-assurance record, then open-sourced it and adapted it to enlist around 500 housing and homelessness workers by June 2026; the cohort published a multi-council governance dataset but no transcription-accuracy or error-rate evaluation, and standard risk controls such as penetration testing and certification had not been completed on the alpha at pilot time.

Sources: localgovernmentassociation2025b, localgovernmentassociation2025a, ministryofhousing2026, trendall2026, incubatorforartificialintell2026

Appears on: /domains/cases/minute-local-ai

Independent research by the Ada Lovelace Institute on AI transcription in social work, based on interviews with 39 social workers across 17 local authorities in England and Scotland, reported that local authorities focus their evaluations on efficiency rather than impact on people who draw on care and that perceptions of reliability and the need for human oversight vary significantly among workers; the research covers such tools sector-wide, not this tool specifically.

Sources: adalovelaceinstitute2026b, bruff2026, adalovelaceinstitute2026a

Appears on: /domains/cases/minute-local-ai

The Ministry of Justice built an in-house AI transcription and summarisation copilot, Justice Transcribe, for probation staff in England and Wales, scaling it from a pilot to more than 1,000 officers in October 2025 and to every probation officer by June 2026, with official transparency data recording more than 800,000 supervision meetings summarised between 7 October 2025 and 2 June 2026; the reported time-savings are the ministry's own and rest on an operating assumption the department itself labels illustrative, and no transcription-accuracy rate, officer correction rate, or independent evaluation of the tool has been published.

Sources: justiceaiunit2026, ministryofjustice2025, ministryofjusticeandhmprison2025b, ministryofjusticeanddsit2025, ministryofjustice2026

Appears on: /domains/cases/justice-transcribe-probation

Copilot-written probation case records sit upstream of high-volume algorithmic risk assessment over the same record ecosystem: reporting places the ministry's OASys-based reoffending-risk prediction at more than 1,300 people a day, drawing on probation and prison caseload systems and the Police National Computer, with a successor tool rolling out during 2026, and the ministry's own validation found lower predictive validity for all Black, Asian and Minority Ethnic groups for non-violent reoffending and for Black and Mixed ethnicity offenders for violent reoffending - a property of the downstream risk model, not the copilot; peer-reviewed commentary raises the erosion of professional judgment and the unresolved accountability for algorithm-influenced decisions as structural concerns, and no published source documents a named data pipeline from the copilot's output into the risk tools.

Sources: statewatch2025, phillips2026, nellis2026

Appears on: /domains/cases/justice-transcribe-probation

The US Social Security Administration requires decision writers to run fully favorable disability decisions through its in-house Insight verifier before issuance, with narrow documented exceptions, and the 2025 federal AI inventory records the tool computing 43 quality flags. In the agency's internal five-month study of roughly 50,000 appeals-level cases, reported through the 2019 Inspector General audit, analysts who used Insight logged about 0.9 errors per case against 0.7 for non-users, saw processing time fall about 4.7 days per case, and had about 12.6 percent of their cases returned for quality issues against 21.5 percent for non-users. These are internal, non-randomized comparisons among self-selected voluntary users, and the same audit found the agency stopped tracking performance after the first five months and could not determine any effect on remands.

Sources: ussocialsecurityadministrati2019, ussocialsecurityadministrati2026, engstrom2020

Appears on: /domains/cases/ssa-insight

In a review issued April 30, 2026 (report 25-00153-47), the Department of Veterans Affairs Office of Inspector General found that at least 8,000 of an estimated 8,100 automated Dependency and Indemnity Compensation (survivor-benefit) granting decisions issued from September 2023 through August 2024 - nearly all - contained at least one legal or procedural deficiency, such as incomplete evidence summaries and omitted favorable findings, with most rating decisions listing only the death certificate as evidence. The OIG separately found that at least 2 percent of the decisions (at least 190) carried monetary-impact legal errors totaling at least 2.7 million dollars (2,727,764 dollars in questioned costs); the roughly 98 percent figure is the share with any legal or procedural defect, not the monetary-error rate. The system, phased in beginning May 2020, extracts data from scanned documents and applies predefined encoded rules to grant service-connected death claims end to end with no human involvement when the rules are met; the OIG describes it as rules-based automation and document extraction, not machine learning, and its figures are outcome statistics from a statistical sample rather than a per-interaction rate.

Sources: departmentofveteransaffairso2026, nieberg2026, weston2026

Appears on: /domains/cases/va-claims-automation

The Office of Inspector General reported that VA's internal correction channels did not catch the automated survivor-benefit deficiencies and that the external audit was, empirically, the only channel that changed behavior. In April 2020 a VBA analyst reported through the internal defect-tracking system that automated decisions listed only the death certificate as evidence, and the Pension and Fiduciary Service closed the defect without action; the same deficiency was central to the 2026 findings, and VA removed the long-form guidance from its manual only in March 2025, immediately after the OIG's preliminary briefing - roughly five years later, and the OIG's full public report did not follow until 2026, roughly six years after the ticket. The OIG found the quality-review checklist for automated claims was less rigorous than the review traditional claims receive, and that the PACT Act section 701(b) modernization plan to Congress did not fully disclose that VBA grants these claims end to end without human intervention. Errors persisted as the program expanded: the VA Secretary announced expanded DIC automation in May 2025, and 20 additional automated decisions from September and October 2025 showed similar errors as of November 2025, with one recommendation still open and VBA concurring only in part.

Sources: departmentofveteransaffairso2026

Appears on: /domains/cases/va-claims-automation

In Trelleborg, Sweden, the first municipality to fully automate social-assistance decisions, peer-reviewed analysis reports that about 30 percent of digital reapplications are decided entirely by rules-based software with no human review and about 85 percent receive at least partial automated handling; decision time on reapplications fell from roughly two days to under a minute, and a human caseworker re-enters the path only by exception, when a routing rule detects significantly changed circumstances, a missing activity plan or job-seeking documentation, or a complex or negative case. No error, override, exception-routing, or appeal-rate figures for the automated path have been published, so the fraction of automated decisions that ever reaches a human cannot be established from the record.

Sources: algorithmwatch2020b, ranerupandhenriksen2022, europeancommissionjointresea2021

Appears on: /domains/cases/trelleborg-rpa

The City of Amsterdam spent roughly five years and an estimated EUR 535,000 building a deliberately fair, explainable welfare-fraud screening model with nearly every recommended pre-deployment safeguard in place - a bias audit, training-data reweighting that approximately equalized wrongful-flag rates on retrospective data, a data-protection assessment and a human-rights assessment, external and academic review, a citizen panel, and dual algorithm-register transparency - and discontinued it after a 2023 live pilot on nearly 1,600 applications. In the investigating journalists' analysis of aggregate data the city provided, the group disparities re-emerged inverted on the live pilot, now more likely to wrongly flag Dutch nationals, women, and applicants with children, with the tool flagging more applications than the analog process and no better than caseworkers at finding genuine cases. The Dutch national algorithm register records the deployment ending September 2023 and lists it out of use, and the responsible alderman announced the halt in November 2023.

Sources: braun2025, lighthousereportsa, algoritmeregisterdutchnation2023

Appears on: /domains/cases/amsterdam-slimme-check

In a developer-reported randomised controlled trial of more than 1,000 adviser support requests, an adviser-facing benefits copilot at Citizens Advice returned supervisor-checked answers in about four minutes, roughly half the previous response time, with about 80 percent of its drafts approved by supervisors without revision; these figures are reported by the tool's builders and have not been independently replicated.

Sources: varotsis2025, departmentforscience2025a, stanfordlegaldesignlabjustic

Appears on: /domains/cases/caddy-citizens-advice, /what-ai-can-do

Advisers given access to the copilot were reported to be more than twice as likely to say they felt confident giving advice than a control group, a self-reported measure from post-call in-chat surveys rather than a client-outcome or accuracy measure.

Sources: varotsis2025, stanfordlegaldesignlab2025

Appears on: /domains/cases/caddy-citizens-advice, /what-ai-can-do

In 2023 Singapore's GovTech began a whole-of-government retire-and-replace of its scripted Ask Jamie chatbots, embedded since 2014 on 70-plus (a vendor case study claims 80) agency websites as independent per-agency answer engines, migrating government chatbots onto centrally provided large-language-model engines; the stated aim was to convert all 88 chatbots and retire the scripted engine by end 2023, the verified snapshot is 21 of 88 converted as of September 2023 (migration completion not independently documented), and by the VICA product page updated 29 April 2026 the successor platform hosts over 100 chatbots for 60-plus agencies at an average of over 800,000 monthly queries, figures that are all government self-reported.

Sources: hirdaramani2023, govtechsingapore2026, govtechsingapore2019

Appears on: /domains/cases/singapore-chatbot-fleet-refresh

The Singapore government benefits-navigation surface is documented as scope-limited to information and estimates rather than adjudication: the Ministry of Finance Support For You Calculator turns self-declared inputs into benefit estimates that are explicitly estimates and not entitlement decisions, and the Chat.Gov.SG (Beta) explainer hosted on the SupportGoWhere domain states the assistant summarises information from official government websites and does not assess eligibility, make decisions, submit applications, or complete transactions, and warns users not to share personal or sensitive information.

Sources: publicservicedivisionsingapo2026, mustsharenews2024

Appears on: /domains/cases/singapore-chatbot-fleet-refresh

A June 2026 Treasury Inspector General for Tax Administration performance audit (Report Number 2026-308-029) reported that the IRS expanded its Automated Collection System chatbot and live-chat program and made live chat permanent while having no performance measures for it, despite a Taxpayer First Act requirement for metrics and benchmarks, and that management's claim the bots reduced telephone demand could not be substantiated; the statistical reports the IRS did collect were deemed unreliable, in one instance showing a single assistor apparently working 603 chats at once against a systemic cap of three, attributed partly to a miscalculated handle-time metric the vendor had not resolved as of December 2025.

Sources: treasuryinspectorgeneralfort2026, bracken2026, bramwell2026

Appears on: /domains/cases/irs-acs-chatbots

In the same audit, of a judgmental sample of 40 IRS ACS live assistors, 24 (60%) were found working multiple chats concurrently and 12 of those 24 had at least one authenticated chat open while working another, which TIGTA reported as raising the risk of disclosing taxpayer information to the wrong taxpayer; the audit also reported 635,684 resolution codes against 613,056 chats (a mismatch management knew of but did not investigate) and, in March 2025 hand-testing, 14% of chatbot process flows deficient and 83% of tested keywords unrecognized or insufficient, with the figures drawn from a nonprobability sample and data the audit itself characterized as unreliable and not projectable to the full assistor population.

Sources: treasuryinspectorgeneralfort2026, bramwell2026, cohn2026

Appears on: /domains/cases/irs-acs-chatbots

In a spring-2025 randomized pilot inside a large consumer EBT app, the vendor reports that 53% of eligible SNAP recipients took up in-app AI help for missed deposits and that treated users were restored faster and more often in the same month than a control group, with every AI dead-end escalated to a named human; all outcome figures are vendor-published and the effect magnitudes were not disclosed.

Sources: propelincpropelinsights2025b, guarino2025a

Appears on: /domains/cases/propel-snap-assistant

By the vendor's own account of the design, the assistant grounds on a state-verified deposit record it reads but does not write to, and steers recipients to act on the state system of record rather than acting for them.

Sources: propelincpropelinsights2025b

Appears on: /domains/cases/propel-snap-assistant

Between 2019 and 2025 more than 70% (about 73% per its ten-year retrospective) of California's online SNAP applications were submitted through GetCalFresh, a deterministic, structured-workflow application assister built and operated by the nonprofit Code for America, which reports helping 6.2 million people obtain more than $12.8 billion in food benefits from 2017 to 2025 (organization-published figures that are not independently audited); the node made no eligibility determinations, and in 2024 and 2025 the California Department of Social Services coordinated a dated, phased transfer of its functions into the state-owned BenefitsCal portal.

Sources: codeforamerica2024b, codeforamerica2025a, californiadepartmentofsocial2025

Appears on: /domains/cases/getcalfresh, /what-ai-can-do

A randomized controlled trial of roughly 65,000 Los Angeles GetCalFresh applicants (Giannella, Homonoff, Rino, and Somerville, American Economic Journal: Economic Policy 16(4), 2024) found that access to applicant-initiated flexible interviews increased SNAP approvals by about 6 percentage points, doubled early approvals, and raised long-term participation by over 2 percentage points, identifying the intake interview as a key procedural-denial barrier; Code for America separately reported an in-house experiment lifting renewal-form submissions among prior non-responders from about 1.5% to roughly 12% (organization-published, without sample sizes or confidence intervals).

Sources: giannella2024, codeforamerica2021, codeforamerica2024a

Appears on: /domains/cases/getcalfresh

In June 2024 the board of Benefits Data Trust, a Philadelphia benefits-navigation nonprofit that reported helping more than 120,000 people access about $182 million in benefits in 2023, voted unanimously to wind the organization down within a self-imposed 60-day window, citing only 'a perfect storm of circumstances'; the organization closed on August 24, 2024, laying off 273 employees, despite roughly $12 million in unrestricted reserves at the end of 2023 and about $32 million in projected 2024 revenue.

Sources: brubaker2024a, brubaker2024b, wink2024, mosbruckergarza2024

Appears on: /domains/cases/benefits-data-trust-winddown

The closure left active government partnerships without a designated successor, including a Pennsylvania Department of Aging workload of nearly 48,000 applications from 27,018 households in the final year and a Philadelphia BenePhilly call-center contract the organization was reported to be exceeding through mid-2024; the navigation function fragmented to higher-friction channels, with the work redistributed across partner agencies and a subcontractor and referral waits reported as several months, which a Pew analyst described as a 'cascading effect.'

Sources: brubaker2024d, burnley2024, mosbruckergarza2024

Appears on: /domains/cases/benefits-data-trust-winddown

London's Strategic Insights Tool for Rough Sleeping probabilistically links records from three separately governed systems - CHAIN street-outreach contacts, In-Form charity casework, and H-CLIC borough statutory applications - into a single rough-sleeping journey per person that is read, in aggregate form only, across all 33 London local authorities; the tool makes no individual-level determinations, and after the build vendor's data-processor contract ended on 2 February 2024 the Greater London Authority contracted Homeless Link, which also operates the CHAIN source system, for its ongoing hosting, management, and maintenance.

Sources: techuk2024, loti2023, londonofficeoftechnologyandi2023

Appears on: /domains/cases/london-rough-sleeping-sit

The Strategic Insights Tool's matcher accepts an association only above an 85% probability threshold chosen to minimise false positives, and the project's own Phase 2 Data Protection Impact Assessment reports 91% recall - conceding that roughly 9 in 100 true cross-system matches are missed so that 'numbers subsequently appear lower in places where they should be higher' and that recall varies as new data of varying quality is ingested; no false-positive rate is published, the accuracy figures are self-reported by the delivery team, and no independent evaluation of the tool's decision impact exists.

Sources: loti2023, lotiannahumplebyandfacultyja2025

Appears on: /domains/cases/london-rough-sleeping-sit

San Jose's vehicle-mounted computer-vision pilot, described by city officials and national housing advocates as the first US experiment training AI to recognize tents and lived-in vehicles, reported sharply class-asymmetric accuracy in the city's own staff-ground-truthed evaluation — 97% for potholes and 88% for trash, but only 70% for RVs (unable to distinguish a lived-in RV from an empty one) and 12.5% for lived-in vehicles, with a March 2024 official interview bracketing the habitation figures at 70–75% for RVs and 10–15% for lived-in cars against a 70% goal; no detection ever generated an operational dispatch, and after investigative exposure and structured engagement the city removed every habitation-detection use case, its March 2025 status report declining to recommend implementing AI object detection in city operations at this time.

Sources: feathers2024, cityofsanjoseinformationtech2025, usdepartmentoftransportation2025

Appears on: /domains/cases/san-jose-encampment-detection

The pilot's published data-usage protocol declares that the footage cannot be actively monitored for law-enforcement purposes while preserving a police request path to it — verbatim, 'Law enforcement may request access to previously stored footage. Law enforcement is not actively monitoring any data collected' — and requires de-identification or deletion within one month; the CIO stated data was not shared with police during the pilot, yet public-records reporting documented that one vendor's system ran optical character recognition of license plate numbers despite the city's no-identification claim, so the declared authority rule and the feasible data flows diverged, a gap surfaced by journalists rather than by any standing audit, and the no-law-enforcement-use clause is city protocol language rather than statute.

Sources: feathers2024, cityofsanjoseinformationtech2024, varian2024

Appears on: /domains/cases/san-jose-encampment-detection

The Los Angeles Coordinated Entry System replaced the VI-SPDAT survey for single adults with the Los Angeles Housing Assessment Tool, a 19-item self-report score whose weights were derived by a regression on 71,747 historical assessments linked to county records; where the CESTTRR research estimated the VI-SPDAT scored near chance (AUC 0.54) with racial false-negative gaps up to 8.5 percentage points, the equity-adjusted successor was deliberately traded down in overall accuracy (AUC 0.60, from an accuracy-only 0.64) to close those gaps to under one percentage point, and every such figure is a pre-deployment estimate on 2015 to 2018 held-out data rather than an observed post-launch outcome.

Sources: rice2023, losangeleshomelessservicesau2025a

Appears on: /domains/cases/lahsa-triage-revision

During the dual-tool transition the two instruments' PSH-consideration thresholds were 8-plus on the VI-SPDAT and 17-plus on the LA HAT, and by LAHSA's account initial quantitative data and provider feedback showed participants were more likely to obtain an eligible score under the VI-SPDAT, so direct-service providers opted to administer it, a trend LAHSA states 'perpetuated the racial bias of the VI-SPDAT in the System'; on April 22, 2026 the CES Policy Council lowered the LA HAT threshold to 12-plus, ruled the most recent LA HAT score supersedes a coexisting VI-SPDAT score, and forced deactivation of new VI-SPDAT completions (for LA HAT-access programs on May 1, 2026 and system-wide on June 30, 2026), though LAHSA has not released the underlying eligibility-rate numbers.

Sources: losangeleshomelessservicesau2025a, losangeleshomelessservicesau2026b, losangeleshomelessservicesau2026a

Appears on: /domains/cases/lahsa-triage-revision

In a registered randomized controlled trial of 1,263 imminent-risk applicants (514 treatment, 749 control) run by the University of Notre Dame's evaluation lab, households offered flexible emergency financial assistance averaging about 2,000 dollars, typically one to two months of back rent, through Santa Clara County's homelessness-prevention system were reported 81 percent less likely to become homeless within six months and 73 percent within twelve months; the peer-reviewed article's abstract states the assistance reduced homelessness by 3.8 percentage points from a 4.1 percent base rate, and the researchers conservatively estimated 2.47 dollars in community benefits per net dollar spent.

Sources: phillipsandsullivan2025, universityofnotredamenews2023, phillipsandsullivan2021

Appears on: /domains/cases/santa-clara-prevention

Because becoming homeless is statistically rare even among at-risk applicants - about 96 percent of the trial's control group never became homeless without assistance - the program's own co-author cautions that prevention resources can flow to households that would have stayed housed anyway, making screening precision on a low base rate the binding constraint; as of February 2026 the model is being replicated across about ten heterogeneous US jurisdictions under a 77-million-dollar initiative, with the same evaluation lab as the common evidence partner assessing each site.

Sources: kendall2026a, phillipsandsullivan2025, destinationhome2026b, universityofnotredamenews2026

Appears on: /domains/cases/santa-clara-prevention

At the Calgary Drop-In Centre, a University of Calgary engineering group and the NGO shelter operator built deliberately interpretable screening for chronic and episodic shelter use - explicit stay-count thresholds (for example 81 or more stays in a 90-day window) and database-queryable rules derived from the shelter's own administrative records, reported to flag candidate clients at a median of about 98 days versus 285 days under the Government of Canada definition and 365 under the Alberta definition - and, rather than surface a risk score, deployed a co-designed data-navigation interface that shows frontline staff raw client histories; no fetched source confirms the thresholds running as an automated production screener, and the deployed, studied artifact is the raw-history interface.

Sources: messier2021, arulesearchframeworkfortheea2022, masrani2025

Appears on: /domains/cases/calgary-drop-in-shelter-ml

Across a 2022 to 2024 embedded deployment study of the interface (16 staff across 7 role categories; 29.5 hours of qualitative data; five committee observations; three deployed versions), the participant-research team documented a stakes-dependent 'data-outsourcing continuum': staff were reluctant to outsource high-stakes barring decisions, treating the data as a starting point for collaborative discussion, while reporting more willingness to accept automated data-driven recommendations for lower-stakes housing triage; the finding is the staff's own articulated practice rather than a measured override or agreement rate, all deployment evidence is authored by the embedded research team, and no independent evaluation, usage logs, or decision volumes are published.

Sources: masrani2025, thehumanbehindthedatareflect2023

Appears on: /domains/cases/calgary-drop-in-shelter-ml

An AI quality-assurance tool deployed on a national 988 backup line scores crisis counselors' own call practice rather than callers, expanding measured review from the under-3% of calls that had been reviewed by hand toward nearly all of them; a peer-reviewed reliability study of 476 labeled calls reported agreement with human ratings at 98 percent of human interrater agreement for detecting any risk assessment, with average F1 of about 0.86 at call level and 0.66 at statement level, and its authors include four holders of equity in the vendor.

Sources: imel2024, aguilar2023, nihreporternationalinstitute2025

Appears on: /domains/cases/lyssn-protocall-988

The registered randomized crossover trial of the tool's counselor feedback (81 call-takers) completed on October 31, 2025, but as of mid-2026 no results were posted to the trial registry or found in the peer-reviewed literature and participant-level data were marked unavailable for proprietary reasons, so reported counselor-skill-improvement effects remain vendor claims pending independent publication.

Sources: clinicaltrialsgovusnationall2026, lyssn2026

Appears on: /domains/cases/lyssn-protocall-988

Gaggle's student-communication safety monitoring, used by roughly 1,500 US districts covering about 6 million students as of a March 2025 AP and Seattle Times investigation, scans school-issued accounts around the clock and routes flags through a multi-hop chain (a machine flag, an off-site vendor reviewer, district safety staff, and, for imminent-danger after-hours alerts, occasional police welfare checks); in Vancouver Public Schools nearly 2,200 students (about 10% of enrollment) triggered alerts in one year, in Lawrence USD 497 more than 1,200 incidents were logged in ten months with about two-thirds deemed nonissues by officials (a figure the plaintiffs drew from district records), and the archive of flagged documents was accidentally released to reporters as nearly 3,500 unredacted files through unprotected links, while a 2023 RAND review found only scant evidence of either benefit or risk and the vendor publishes no accuracy figures.

Sources: bryanandlurye2025, associatedpress2025, lawrencejournalworld2025

Appears on: /domains/cases/gaggle-school-monitoring

After nine Lawrence, Kansas students sued their district in early August 2025 over its use of AI communication monitoring, court filings revealed the district had ceased using Gaggle mid-litigation and substituted a different monitoring vendor with no board vote or public disclosure — surfacing only as a line in a check register — and the plaintiffs' amended complaint argued the swap does not moot the case because the core practice of suspicionless scanning, flagging, and seizure of student speech continues; on April 10, 2026 a federal judge found the district violated the Kansas Open Records Act in withholding the substitution and phase-out records, and on June 4, 2026 ordered it to pay the students' attorney fees, characterizing the conduct as drawn out, hollow and perplexing, with a jury trial on the surviving constitutional claims set for January 2027.

Sources: heimsoth2025, heimsoth2026a, heimsoth2026b

Appears on: /domains/cases/gaggle-school-monitoring

In March 2026 the UK Parliamentary and Health Service Ombudsman partly upheld a complaint that an NHS mental health trust installed camera-based, contact-free bedroom monitoring on a psychiatric ward without seeking a patient's consent, gave her no information about it, and did not switch it off when she asked; the case documentation and investigative reporting describe an internal clinical evaluation that the vendor is reported to have authored the business case for and shaped, a rebrand of the vendor during a statutory inquiry, and an open data-protection investigation, while the tool's own outcome-reduction figures are vendor claims contested by a campaign-linked meta-analysis and its adoption share across NHS mental health trusts is reported only as a contested range.

Sources: parliamentaryandhealthservic2026, williamson2026a, williamson2026b, nationalsurvivorusernetwork2025, stopoxevision2026

Appears on: /domains/cases/oxevision-nhs-wards

The ombudsman's report on the case (decision 27 March 2026) found the trust did not seek or revisit the patient's consent for the bedroom monitoring, did not turn the camera off when she asked, gave her no information about it, and kept no record of how staff used it, and that even the trust's revised 2025 procedure still permits overriding a capacitous patient's refusal on clinically-safe grounds with multidisciplinary-team approval; on the separate question of over-reliance the ombudsman found on balance, cross-referencing observation charts, a nurse-adviser review and door key-card data, that in-person observations had continued and did not uphold that part of the complaint.

Sources: parliamentaryandhealthservic2026

Appears on: /domains/cases/oxevision-nhs-wards

In 2025 ODMAP's pre-set county thresholds - a rolling 24-hour count against a threshold each agency sets or accepts, recommended by the system as two standard deviations above the county's own previous 90-day mean, a deterministic rule rather than a machine-learning model - fired 74,805 advisory spike-alert notifications from 498,003 suspected, unconfirmed overdose events that only about 1,362 of its 5,605 approved agencies actually submitted; these figures are self-published by the program in its own annual report and manuals, and ODMAP states its data are suspected, incomplete, not a system of record, and should not be generalized beyond participating agencies.

Sources: washingtonbaltimorehidta2025b, washingtonbaltimorehidta2026c, washingtonbaltimorehidta2025a

Appears on: /domains/cases/odmap-overdose-spike-alerts

ODMAP's shared overdose store is housed inside a federal drug-enforcement program, and its operating policies both state that ODMAP is neither an intelligence sharing database nor a pointer index records system and grant the host permission to use the data as the HIDTA sees fit, including combining it with other databases it manages for law enforcement and public health products; a 2024 peer-reviewed stakeholder study documented divergent public-health versus public-safety data-privacy standards, and a 2025 peer-reviewed analysis argues the integration risks racialized surveillance and criminalization of people who experience overdose, a contested scholarly critique of the link structure rather than a documented misuse incident.

Sources: washingtonbaltimorehidta2022, syvertsen2025, allen2024

Appears on: /domains/cases/odmap-overdose-spike-alerts

The Targeted Real-Time Early Warning System (TREWS), a machine-learning sepsis early-warning model, was evaluated prospectively across five hospitals of an academic health system covering 590,736 monitored patients — the largest prospective study of an ML sepsis system on record. Its central finding was conditional on the human loop: sepsis patients whose alert was evaluated and confirmed by a provider within three hours had a 3.3 percentage-point absolute and 18.7 percent relative adjusted reduction in in-hospital mortality, with less organ failure and shorter stays, while the alert on its own did not; a companion study found provider uptake varied with experience, unit culture, and alert context.

Sources: adams2022a, henry2022a

Appears on: /domains/cases/johns-hopkins-trews, /pan-lab

The TREWS mortality-benefit evaluation was prospective and peer-reviewed but observational and developer-led: it was built at the deploying institution and commercialized through a company founded by its principal investigator, and confirmation-associated benefit is an observational association rather than a randomized effect of the algorithm — providers who engaged with alerts may differ from those who did not in ways the adjustment does not capture. The strongest numbers in the record therefore come from the party with the strongest interest in them, and no independent replication of the mortality effect had been published.

Sources: adams2022a

Appears on: /domains/cases/johns-hopkins-trews, /pan-lab

The Advance Alert Monitor is an in-hospital deterioration model running around the clock across 21 hospitals of an integrated health system, scoring inpatients hourly and firing roughly twelve hours before predicted deterioration; a 2020 New England Journal of Medicine evaluation associated its alert-driven rapid-response workflow with lower mortality. Its defining feature is where the alert goes: not to the bedside, but to a dedicated regional tier of critical-care virtual quality nurse consultants who screen every alert around the clock, work up the chart, and only then escalate to the on-site rapid-response team — so the measured benefit is priced against the whole two-tier staffing topology, not the model alone.

Sources: escobar2020a, thekaiserpermanentenorthernc2022

Appears on: /domains/cases/kaiser-aam-deterioration, /pan-lab

Sepsis Watch is a deep-learning sepsis-detection system scoring every emergency-department patient every five minutes over 86 variables, deployed at an academic hospital under a registered clinical trial, with alerts fronted by rapid-response-team nurses who track treatment-bundle completion on three- and six-hour timers. Its structural fault line is an authority split: the operator who receives the alert (the nurse) is not the operator empowered to act on it (the physician who holds treatment authority), so the correction runs through a peer-persuasion edge. An independent ethnography found the system worked because nurses performed hidden repair work — mediating the professional hierarchy and doing the emotional labor of communicating a risk score upward — labor that was structurally necessary, largely invisible to the deployment's formal description, and undervalued.

Sources: sendak2020a, elish2020

Appears on: /domains/cases/duke-sepsis-watch, /pan-lab

A widely implemented proprietary sepsis-prediction model shipped inside a common electronic-health-record platform and switched on across hundreds of hospitals was externally validated in 2021 across 38,455 hospitalizations at an academic health system: it achieved an area under the curve of 0.63, identified only 33 percent of sepsis cases, and had a positive predictive value of about 12 percent, generating roughly 109 alerts for every true sepsis case — a real-world performance the vendor had not fully examined before selling the model, and which an investigation attributed in part to undisclosed features such as antibiotic-order data that inflated internal validation.

Sources: wong2021c, statnews2021

Appears on: /domains/cases/epic-sepsis-michigan, /pan-lab

After external criticism, the vendor overhauled the sepsis model — retraining it, changing the sepsis-onset definition, and reducing its reliance on antibiotic-order features. A 2026 multicenter prospective validation of the updated model across 227,091 encounters reported an area under the curve of 0.82 to 0.92 with positive predictive value of 0.13 to 0.26 and substantial between-site variability, and its authors urged local validation and alert-silencing strategies rather than trusting the model out of the box — a correction that arrived only after independent scrutiny of a model that had already been deployed at scale behind a corporate firewall shielding it from outside inspection.

Sources: statnews2022, wong2026a

Appears on: /domains/cases/epic-sepsis-michigan, /pan-lab

The largest documented ambient-scribe deployment ran a 10-week pilot at an integrated medical group and then scaled to 7,260 physicians and 2,576,627 patient encounters over fourteen months, with roughly 16,000 hours of documentation time saved and sustained physician support measured along the way. The system records the visit and drafts the clinical note; the clinician edits and signs, and the model-to-record write is gated both by that clinician review and by a standing internal quality-assurance program over the AI output — a real subsystem with a real cost, because the drafted note becomes a permanent record that later clinicians and later tools read as fact.

Sources: tierney2024a, tierney2025a

Appears on: /domains/cases/kaiser-tpmg-scribe, /pan-lab

The scale numbers from a single ambient-scribe deployment are the deployer's own first-party measurements and should be read as that system's dashboard rather than a guarantee of the product class: a multisite study of 8,581 clinicians across five health systems found more modest effects — on the order of 13 to 16 fewer minutes per day with no meaningful after-hours relief — and a validated per-note evaluation found hallucinations in about 31 percent of ambient-generated notes under structured review, versus about 20 percent of physician-written gold-standard notes, making ambient notes more thorough but less accurate. The clinician review and quality-assurance program are the controls that stand between that error rate and a contaminated permanent record.

Sources: rotenstein2026a, palm2025a

Appears on: /domains/cases/kaiser-tpmg-scribe, /pan-lab

The strongest causal evidence in the ambient-scribe family comes from a 24-week stepped-wedge, individually randomized trial of an ambient scribe across 66 practitioners and 71,487 notes (38 percent AI-generated), which found work exhaustion significantly reduced, professional fulfillment unchanged (a recorded null), roughly 22 minutes per day of documentation time returned, and diagnostic coding accuracy improved. Unlike the larger first-party deployment reports, this is a randomized estimate of the well-being and time effects — though it measured practitioner well-being and time, not per-note error rates.

Sources: afshar2025a

Appears on: /domains/cases/uw-health-abridge-scribe, /pan-lab

The same team that ran the ambient-scribe trial released an open operations playbook for safety and effectiveness monitoring of ambient AI in production — the rare case where the organization-side monitoring function exists as a citable, designed subsystem rather than an assumed practice. That monitoring is the org's stated answer to a documented system-level risk of the technology: a coding arms race, in which better AI documentation raises coding intensity, payers recalibrate in response, and clinician attestation liability grows — so the improved coding accuracy the trial measured sits next door to an upcoding pressure the monitoring is meant to watch.

Sources: afshar2025b, dai2025a

Appears on: /domains/cases/uw-health-abridge-scribe, /pan-lab

A peer-reviewed evaluation of an ambient documentation platform at a large multi-specialty system found note time per appointment reduced (6.2 to 5.3 minutes) and NASA-TLX cognitive load reduced — but the burnout change (42.1 to 35.1 percent) was not statistically significant, the domain's honest null bound of cognitive-load relief without a demonstrated burnout effect.

Sources: stults2025a

Appears on: /domains/cases/sutter-ambient-scribe, /pan-lab

In the same evaluation, benefit varied sharply by clinician group: 85.8 percent of primary-care physicians reported improved satisfaction against 36.4 percent of medical specialists — the same tool, in the same system, under the same workflow, helping one operator class and largely failing another, so any uniform service term overstates the effect for the group it helps least.

Sources: stults2025a

Appears on: /domains/cases/sutter-ambient-scribe, /pan-lab

Company-run, pre-registered, peer-reviewed randomized rollouts of a commercial code-completion assistant across 4,867 developers at three enterprises found a pooled 26.08 percent increase in completed tasks, with gains concentrated among less-experienced developers. An independent randomized study of 16 experienced open-source maintainers on 246 tasks in familiar repositories bounded the expert tail from the other direction: those developers were about 19 percent slower with the AI while believing themselves about 20 percent faster — a measured perception-reality gap that means a uniform productivity number overstates the effect for senior engineers.

Sources: cui2025a, becker2025a

Appears on: /domains/cases/msft-accenture-copilot, /pan-lab

Individual coding-assistant gains do not automatically compose to organization-level delivery outcomes: a cross-industry research program measured a roughly 1.5 percent decrease in delivery throughput and a 7.2 percent decrease in delivery stability for every 25 percent increase in AI adoption, evidence that the churn the assistant adds must be absorbed by code-review and testing gates or the individual speed-up degrades the organization's delivery performance.

Sources: googleclouddora2024

Appears on: /domains/cases/msft-accenture-copilot, /pan-lab

An in-house machine-learning code-completion system built, deployed, and measured by a company's own platform organization for more than 10,000 internal developers reported, against a control group, a 25 to 34 percent suggestion-acceptance rate, a 6 percent reduction in coding iteration time versus control, and 3 percent of new code characters coming from the model at the time of measurement. The measuring party, the building party, and the deploying party were the same organization, and the numbers were published as an engineering-blog self-report rather than a peer-reviewed or independent evaluation.

Sources: tabachnyk2022

Appears on: /domains/cases/google-internal-completion, /pan-lab

The same company's cross-industry research program reported that AI-assisted software development amplifies an organization's existing strengths and weaknesses rather than substituting for them, with policy clarity and platform investment identified as the levers that determine whether AI adoption improves or degrades delivery — evidence that the individual coding gains do not compose to organization-level outcomes on their own, and that the deploying organization's existing gates and platform quality are what decide the result.

Sources: googleclouddora2025

Appears on: /domains/cases/google-internal-completion, /pan-lab

A regulated bank ran a structured six-week internal experiment with about 100 of its 5,000 engineers before scaling a commercial coding assistant to roughly 1,000 engineers, publishing its own measurement of the rollout. The bank's engineers reported productivity and code-quality improvements — and recorded the security impact as explicitly inconclusive, a real gating decision taken and documented under uncertainty rather than resolved by assertion, with the honestly recorded unknown carried forward into the scaled deployment.

Sources: chatterjee2024a, theregister2024

Appears on: /domains/cases/anz-bank-copilot, /pan-lab

What the bank's inconclusive security finding leaves open is not hypothetical: an independent security assessment of code generated by a widely used assistant found that about 40 percent of generated programs contained vulnerabilities across scenarios spanning the CWE top-25 weaknesses, and separate research documents developers accepting insecure suggestions with overconfidence — so the security unknown a deployment carries forward unresolved sits against a class-level literature in which insecure generation is common.

Sources: pearce2022a

Appears on: /domains/cases/anz-bank-copilot, /pan-lab

A mid-size enterprise ran a systematic four-phase evaluation-to-rollout of a commercial coding assistant across more than 400 developers, publishing acceptance telemetry (a 33 percent suggestion-acceptance rate, with 20 percent of suggested lines accepted), a 72 percent satisfaction figure, documented per-language variation, and stated limitations. Its evaluation instrument is acceptance-rate telemetry — which the productivity literature identifies as the measure most correlated with perceived productivity rather than outcome, and perception is measured to be miscalibrated for experienced developers, so acceptance telemetry captures adoption feel, not delivered output.

Sources: bakal2025a, ziegler2024a

Appears on: /domains/cases/zoominfo-copilot, /pan-lab

The deployment report stated its limitations but reported no security evaluation at all — an unrecorded unknown, one step less honest than a deployment that runs a security check and records the result as inconclusive, because an absence no one has written down is not a governed object and cannot be carried forward or resolved. The value of the case is the documentation quality of an ordinary, competent adoption — phase gates, telemetry definitions, per-language deltas, and stated limitations by the deployer itself — with the missing security question priced as the one thing even that documentation did not name.

Sources: bakal2025a

Appears on: /domains/cases/zoominfo-copilot, /pan-lab

A global bank replaced rules-based transaction monitoring with a cloud vendor's machine-learning anti-money-laundering product as its primary monitoring system in key markets, reporting two to four times more confirmed suspicious activity with roughly 60 percent fewer alerts. Every one of those numbers is a vendor-and-customer self-report with no independent audit — which is itself the honest structure of the domain, because a peer-reviewed deployment-scale benefit measurement inside a named financial-crime operation does not publicly exist, and the alert-volume reduction the vendor advertises is precisely the lever a regulator scrutinizing an under-monitoring risk would question.

Sources: googlecloud2023

Appears on: /domains/cases/hsbc-aml, /pan-lab

Two structural dynamics govern fraud and financial-crime detection. Under extreme base rates, detection precision is dominated by the false-alarm rate rather than by accuracy, so at realistic prevalence a threshold change moves the burden of alerts rather than the truth of them (the base-rate fallacy). And the labels the model learns from are the investigators' own dispositions: only a small set of flagged transactions is ever verified, and models are retrained on the analysts' calls, so a rise in 'confirmed' activity is partly a measure of what the system taught its reviewers to confirm rather than an independent ground truth (the label-feedback loop).

Sources: axelsson2000a, dalpozzolo2018a

Appears on: /domains/cases/hsbc-aml, /pan-lab

A neobank's fraud algorithms — triggered heavily by pandemic-era government benefit deposits — froze and closed the accounts of legitimate customers at scale, holding their balances for thirty to more than ninety days, and the company admitted some of the closures were mistakes. The false-positive tail here lands on real people as immediate hardship, concentrated among benefit-deposit recipients and low-balance households for whom a frozen account means no access to funds for weeks.

Sources: kessler2021

Appears on: /domains/cases/chime-fraud, /pan-lab

A 2024 federal consent order priced the downstream operational failure rather than the model: thousands of consumers waited weeks to months for their balances after account closure, and the order imposed a 3.25 million dollar civil penalty plus at least 1.3 million dollars in consumer redress for the delayed refunds. The harm ran through three stages inside the organization's control — the scoring model's false positives, the operations backlog that turned a freeze into months without funds, and the refund process whose delay drew the regulator — and the enforcement attached to the last stage, the backlog, not to the model that started it.

Sources: consumerfinancialprotectionb2024

Appears on: /domains/cases/chime-fraud, /pan-lab

A Nordic bank's rules-based legacy fraud system ran at roughly 40 percent detection with a 99.5 percent false-positive rate — a measured pre-machine-learning baseline whose badness is the most credible datum in the record, since a 99.5 percent false-positive rate is not a marketing claim. The vendor-published rollout of a deep-learning engine scoring transactions in real time (under 300 milliseconds) claims false positives cut by about 60 percent and true-positive detection raised by about 50 percent; those figures are an organization-named, trade-press-covered vendor case study, entered here as claimed magnitudes against that legacy baseline because they were not independently audited.

Sources: teradata2017, groenfeldt2017

Appears on: /domains/cases/danske-fraud, /pan-lab

The same institution that improved its in-line fraud scoring later ranked worst among UK banks for reimbursing victims of authorized-push-payment scams in the regulator's bank-by-bank performance data — better detection and worse victim-outcome performance coexisting in one organization. And it was a rule, not a model, that moved the institutional behavior: the regulator's mandatory-reimbursement regime raised sector reimbursement from roughly two-thirds to about 89 percent, demonstrating that detection quality and the justice of the disposition are different levers held by different actors, and that the victim-outcome lever is a regulatory rule rather than a better classifier.

Sources: ukpaymentsystemsregulator2023

Appears on: /domains/cases/danske-fraud, /pan-lab

An internal team built an experimental recruiting engine — roughly 500 models scoring resumes one to five stars per role and location — trained on ten years of the company's own hiring decisions, a period whose hires were predominantly male. The models learned that history: they penalized the word 'women's' and downgraded graduates of women's colleges, reading gender proxies as negative signal. The team patched the identified terms but concluded that term-level fixes could not guarantee neutrality against unknown proxies, because the model had learned the pattern rather than the words, and the company scrapped the project around 2017; per the company, recruiters saw the tool's recommendations but it was never used as a sole ranking.

Sources: dastin2018

Appears on: /domains/cases/amazon-resume-engine, /pan-lab

Training a screener on an organization's past hiring decisions imports the past's selection function: research on hiring as exploration finds that models trained on prior hires raise hire rates but replicate historical selection, and that a screener which values exploration rather than only exploitation breaks that lock-in loop. Two governance lessons follow — the patch lever has a documented ceiling, since removing named proxies does not remove a learned correlation, and abandonment can itself be a governance outcome, taken here before any external harm was documented rather than after an adjudication.

Sources: li2020a

Appears on: /domains/cases/amazon-resume-engine, /pan-lab

An applicant-tracking platform whose AI screening and recommendation features operate inside thousands of employers' hiring pipelines at once is the subject of a live federal collective action testing whether the vendor is directly liable as the employers' agent. On the litigation record, the court sustained the agent theory at the dismissal stage in 2024 and preliminarily certified a nationwide age-discrimination collective in 2025, covering applicants forty and over since September 2020, on a record in which the lead plaintiff reported more than one hundred rejections across employers using the platform. The litigation is ongoing and nothing here is an adjudicated finding of discrimination; these are allegations and procedural rulings, not a verdict.

Sources: mobleyvworkday2024

Appears on: /domains/cases/workday-screening, /pan-lab

A 2026 discovery ruling in the same matter held the vendor's internal bias-testing data privileged because counsel had curated it — meaning the testing record exists and is legally unreachable, a configuration in which audit opacity is not the absence of testing but testing shielded from external verification. The case surfaces two further structural facts: a single vendor's screening model multiplied across many employer boundaries, so one learned defect can propagate as widely as the platform, and accountability diffusion between deployer and vendor, each holding part of the governance the other points to, against a survey backdrop showing assessment vendors' validation and bias-mitigation claims are often unverifiable from outside.

Sources: raghavan2020a, u2023b

Appears on: /domains/cases/workday-screening, /pan-lab

A graduate-hiring pipeline chained a games-based assessment with automated video-interview scoring, and the deployer reports roughly a 90 percent reduction in time-to-hire (from about four months to about four weeks), around 50,000 candidate interview hours saved, about one million pounds in annual savings, and a 16 percent improvement in diversity. Every one of those figures is company- or vendor-reported and none is independently audited, so they are the deployer's own dashboard rather than an external measurement — which is exactly what the family's service regime looks like from inside.

Sources: bestpracticeai

Appears on: /domains/cases/unilever-hiring, /pan-lab

Both vendors' audit machinery is on the public record in an honest but partial form. The games vendor underwent a cooperative academic audit with source-code access, in which its four-fifths-rule de-biasing pipeline was found faithfully implemented — with the independence caveat that vendor staff were co-authors — and the video vendor retired its facial-analysis input under scrutiny after internal research found visual features added only about 0.25 percent predictive power, publicizing a narrow-scope external audit. The family's structural blind spot applies in full: rejected candidates never re-enter the outcome data, so the claimed quality and diversity effects are measured on hires only.

Sources: wilson2021a, maurer2021a

Appears on: /domains/cases/unilever-hiring, /pan-lab

A machine-learning underwriting and pricing platform using education and other alternative data operated for five years under a regulator's no-action letter with a reporting obligation, and the regulator published the access results: 27 percent more applicants approved than a traditional model at 16 percent lower average APRs, with near-prime applicants (FICO 620 to 660) approved at roughly twice the rate, and gains across the tested demographic segments. This is the lending family's only regulator-verified service term. Underwriting is fully automated with no per-application human review, so the organizational levers are all upstream — model choice, the testing regime, the search for alternatives, and the reporting channel to the regulator.

Sources: consumerfinancialprotectionb2019

Appears on: /domains/cases/upstart-underwriting, /pan-lab

The same deployment carries the family's most detailed public fair-lending testing record: four reports from an independent monitorship agreed with civil-rights organizations found no close protected-class proxies quantitatively, but identified approval disparities for Black applicants, flagged a likely viable less-discriminatory alternative model, and ended in a documented methodological impasse over how hard the law requires an organization to search for such an alternative. Independently of the disparity question, adverse-action notices must give specific, accurate principal reasons for a denial regardless of the model's complexity — a governed explanation duty a complex model does not discharge by being accurate.

Sources: relmancolfaxpllc2021, consumerfinancialprotectionb2022b

Appears on: /domains/cases/upstart-underwriting, /pan-lab

A bank's automated credit-decisioning for a widely used consumer card was investigated by a state regulator after viral allegations of gender bias in credit-line assignment. The regulator analyzed roughly 400,000 in-state applicants and found no unlawful discrimination on a prohibited basis — the model was cleared on the numbers. But the same investigation documented failures of explanation, customer service, and perceived transparency: applicants could not learn why they received the terms they did, front-line staff could not explain the decisions, and the resulting opacity destroyed consumer trust even though the underwriting itself was found lawful. This is the domain's cleared-but-faulted case: a statistically clean model paired with a failed duty to explain.

Sources: newyorkstatedepartmentoffina2021

Appears on: /domains/cases/goldman-apple-card, /pan-lab

The lesson the cleared-but-faulted outcome carries is that a lawful, statistically clean model does not discharge the separate duty to explain a decision. Regulators have made explicit that adverse-action notices must give specific, accurate principal reasons regardless of how complex the model is, and that a model being a black box is not a defense — checking the nearest sample-form box does not comply. The explanation and customer-service channel is therefore a distinct, separately-resourced surface that can fail on its own: an organization can pass its fair-lending testing and still fail the people it decides on by being unable to tell them why.

Sources: consumerfinancialprotectionb2022b

Appears on: /domains/cases/goldman-apple-card, /pan-lab

A state attorney general reached a $2.5 million settlement with a student-loan lender over its AI underwriting. The documented conduct is the domain's cleanest failure-then-mandated-governance arc: the model used a cohort-default-rate feature — a school's aggregate default rate priced into an individual applicant's terms — that disparately impacted Black and Hispanic applicants, and an immigration-status rule that automatically denied certain non-citizen applicants, while the organization ran no disparate-impact testing and gave inadequate adverse-action notices. The remedy did not fine-and-close: it mandated the missing program — model governance, disparate-impact testing, documentation, and reporting controls — so the enforcement action wrote the governance the deployment had never built.

Sources: officeofthemassachusettsatto2025

Appears on: /domains/cases/earnest-ai-underwriting, /pan-lab

The mechanism the case turns on is the facially-neutral aggregate feature: a cohort default rate is a property of a school, not of the applicant, and no input names a protected class — yet pricing a group's aggregate history into an individual's terms can carry protected-class impact, which is exactly what disparate-impact testing exists to catch. Here that testing was not done, so the impact went unmeasured until an enforcement action found it. The remedy installed the program the deployment lacked, which is the governable reading: an aggregate feature can look neutral input-by-input and still produce a disparity only outcome testing would reveal, and the absence of that testing is itself the failure.

Sources: officeofthemassachusettsatto2025, consumerfinancialprotectionb2022b

Appears on: /domains/cases/earnest-ai-underwriting, /pan-lab

The strongest field evidence for an agent-assist copilot in customer service comes from a staggered randomized rollout of a generative-AI assistant to roughly 5,000 customer-support agents at a large software firm. Measured against a control group, the copilot raised issues resolved per hour by about 15 percent on average, and it also improved customer sentiment and agent retention. The gain, however, was sharply uneven: novice and low-skill agents improved by roughly 30 to 34 percent, agents with two months of experience performed like agents with six months and no AI, and the most experienced agents gained close to nothing, with some evidence of slight quality degradation. This is the contact-centre domain's cleanest measured benefit, and it is a distribution rather than a single number.

Sources: brynjolfsson2025a, brynjolfsson2023

Appears on: /domains/cases/fortune500-agent-copilot, /pan-lab

The lesson the randomized evidence carries is skill compression: an agent-assist copilot mostly raises the floor. Because almost the entire measured gain accrues to less-experienced agents and the most experienced gain close to nothing, an average productivity number overstates the effect for the agents who least need it and hides that the tool does little for the experienced while possibly costing a small amount of quality there. The governable reading is that the benefit must be measured as a distribution across agent skill, not reported as a scalar — a copilot that helps novices a great deal and experts not at all is a real and specific benefit, and describing it with one average misstates who it helps.

Sources: brynjolfsson2025a

Appears on: /domains/cases/fortune500-agent-copilot, /pan-lab

An airline's customer-facing website chatbot told a customer they could claim a bereavement fare retroactively — a policy that did not exist. The customer relied on the chatbot's statement, bought a ticket, and was then refused the fare by the airline's human staff. A civil-resolution tribunal found the airline liable for negligent misrepresentation and awarded damages, and in doing so rejected the airline's argument that the chatbot was a separate legal entity responsible for its own actions. The tribunal held that the organization is responsible for all the information on its website, whether it comes from a static page or a chatbot, and that a customer has no way to know which source to trust. This is the contact-centre domain's cleanest accountability ruling: the bot is a tool the company answers for, not an entity that answers for itself.

Sources: moffattv2024, sookman2024

Appears on: /domains/cases/air-canada-chatbot, /pan-lab

The duty the ruling establishes is that an organization must take reasonable care that its chatbot's representations are accurate, because the chatbot is a tool it deploys rather than a separate entity that answers for itself. A hallucinated policy or a wrong rule stated by the bot is therefore the organization's own misrepresentation, and a posture that treats the AI as speaking only for itself does not transfer that responsibility away. The governable reading is that a customer-facing chatbot is a channel the organization is accountable for exactly as it is accountable for a page on its own website — so the accuracy control on what the bot states, and the ownership of what it says, are the organization's to build, not the bot's to carry.

Sources: sookman2024, moffattv2024

Appears on: /domains/cases/air-canada-chatbot, /pan-lab

An organization published striking first-month numbers for its customer-facing AI assistant: it handled about two-thirds of customer-service chats (some 2.3 million conversations), was described as doing the equivalent work of about 700 full-time agents, cut average resolution time from about 11 minutes to under 2, was said to match human customer satisfaction, and was projected to improve profit by tens of millions. Every one of those figures was self-reported and not independently audited. Roughly a year later the same organization reversed course on quality grounds — its chief executive said cost had become too predominant an evaluation factor and the result was lower quality — and committed to always keeping a human available to customers who want one. This is the contact-centre domain's cleanest benefit-then-cost arc: the deflection numbers and the walk-back come from the same deployment.

Sources: klarnabankab2024, ivanova2025

Appears on: /domains/cases/klarna-ai-assistant, /pan-lab

The lesson the benefit-then-cost arc carries is that deflection is not resolution. A published deflection number reports how many contacts the AI handled, not whether it handled them well, and a figure that is impressive on cost can hide a quality cost that only shows up later — which is what the organization's own reversal described. The survey backdrop sharpens it: most customers say they would rather not meet AI in service and fear it makes reaching a human harder, and industry analysts expect a large share of organizations to abandon plans to reduce their customer-service workforce with AI. The governable reading is to measure resolution and repeat contact against deflection rather than counting deflection as a win by itself, and to protect the path to a human as the safety valve a deflection-maximizing design tends to erode.

Sources: ivanova2025, gartner2025b

Appears on: /domains/cases/klarna-ai-assistant, /pan-lab

A video platform ran an unintended natural experiment on automated content moderation. When the pandemic sent its human reviewers home, the platform said it would rely more on automated removal and deliberately chose over-enforcement rather than let harmful content stay up. The result, from the platform's own transparency reporting, was that automated removals more than doubled in a single quarter (to about 11.4 million videos), appeals roughly doubled, and the reinstatement rate on appeal jumped from about 25 percent to about 50 percent. The platform also withheld strikes where no human had reviewed the removal, treating the automated decision as provisional. The doubling of the reinstatement rate is the finding: it is direct evidence that the automation was making roughly twice the rate of catchable errors, and that the human review and appeals path was the loop catching them.

Sources: youtubegoogle2020

Appears on: /domains/cases/youtube-covid-enforcement, /pan-lab

The lesson the natural experiment carries is that the human review and appeals path is the error-correction loop for automated enforcement, not an optional add-on. Automated moderation makes errors at scale, and a doubling of the reinstatement rate when human review thinned is a measurement of those errors — they were always being made at that rate, and were visible only because the appeals queue surfaced them. Two things follow. Over-enforcement versus under-enforcement is a chosen trade-off: with review capacity cut, the organization decided which error to make, and that was a governance decision. And proactive removal acts before anyone sees the content, so an over-broad takedown is invisible unless appealed — and some removals are irreversible, as when automated systems destroyed documentation of war crimes with archival access declined, leaving no correction loop at all.

Sources: youtubegoogle2020, humanrightswatch2020

Appears on: /domains/cases/youtube-covid-enforcement, /pan-lab

A platform enforces its content standards with automated classifiers at a scale no human team could match, backed by a layered correction structure: an internal appeals process, and above it an external oversight board that issues binding decisions on the individual cases it takes and non-binding policy recommendations to the platform. In one year the board overturned the platform's original decision in around 90 percent of the cases it decided, and the platform reported implementing, in progress on, or already aligned with the large majority of the board's cumulative recommendations. This is the moderation domain's most built-out, institutionalized correction structure — layered appeals rising to an independent-adjacent external body that publishes its reasons.

Sources: metaplatforms, oversightboard2024

Appears on: /domains/cases/meta-content-enforcement, /pan-lab

The reach of the correction structure is the governable limit. The roughly 90 percent overturn rate is measured on selected cases — the board chooses emblematic disputes to set precedent, so the figure is evidence that escalated decisions were often wrong, not a random error rate, and the overwhelming majority of automated enforcement decisions never reach the board at all. The board is funded through a platform-established trust, which makes it independent-adjacent rather than fully independent, and its policy recommendations are non-binding. The honest reading is that this correction structure is real and genuinely better than most, and its reach is bounded to the tiny fraction of cases that escalate — so the governing question is whether the correction reaches the scale of the enforcement it is meant to check.

Sources: oversightboard2024, oversightboard2025

Appears on: /domains/cases/meta-content-enforcement, /pan-lab

A media outlet published AI-drafted finance explainers under a human-sounding staff byline without disclosing to readers that the articles were machine-written. When the practice came to light, the outlet's own audit found it had to issue corrections on a majority of the AI-written articles — on the order of 41 of 77. A byline implies a human review that the reader trusts, and a correction rate that high is a direct measurement that the review the byline implied was not actually performed before publication. A later and sharper case saw another outlet publish articles under entirely fabricated author personas presented as real people, so the failure ran from undisclosed AI drafting to invented human bylines.

Sources: bonifacic2023, harrisondupre2023

Appears on: /domains/cases/cnet-ai-drafting, /pan-lab

Editorial AI moves the failure from a takedown to a publication, but the governable structure is the same as in moderation: the byline is the accountability object, and it stands for a review that either happened or did not. Two things are owed to the reader — disclosure that AI was involved, and an editorial check that actually took place — and this deployment gave neither, publishing under a staff byline that implied both. When a large share of AI-drafted articles needs correction, the review was not performed, and the byline misrepresented who did the work. The governable reading is that a human byline on machine-drafted content is a claim about review and disclosure, and a high correction rate is the evidence that the claim was false.

Sources: bonifacic2023

Appears on: /domains/cases/cnet-ai-drafting, /pan-lab

A large automaker deployed in-line AI inspection at production scale: camera and acoustic systems that detect defects during assembly and feed real-time flags to the line worker via a smart device, on a line running on the order of a thousand-plus vehicles a day at a takt of under a minute per station. The system has been established as a company standard and is being extended to suppliers. The governing design is that the AI flags and a human on the line responds — the inspection is wired into a resourced response loop, including the ability to stop the line, so the benefit runs through the human response the flag triggers rather than through the model alone. The documented facts here are the system's function, the worker-interaction model, the scale, and the standardization; the deployment's benefit is reported through corporate and trade channels, and defect-rate deltas from a primary source are not public.

Sources: bmwgrouppressclub2025, metrologyandqualitynews2026, leanenterpriseinstitute

Appears on: /domains/cases/bmw-aiqx-inspection, /pan-lab

The lesson the deployment carries is that an in-line inspection AI is only as good as the human-response loop it triggers, and that loop is the governable object. When the AI flags a defect, a resourced response — a worker with the time to check the flag and the authority to stop the line — is what turns a detection into a caught defect; without it, the flag is just a decision no one acts on. This is why the failure modes in this domain are matters of the loop's calibration rather than the model's raw accuracy: too many false alarms and operators stop responding, too much trust and they stop checking. The honest boundary is that the benefit is reported through corporate and trade channels, and no named manufacturer has publicly attributed a shipped-defect escape to its AI inspection, so the response loop is drawn as the resourced strength and its calibration as the thing to govern, not as a claim about defects that did or did not ship.

Sources: bmwgrouppressclub2025, leanenterpriseinstitute

Appears on: /domains/cases/bmw-aiqx-inspection, /pan-lab

A peer-reviewed heavy-industry predictive-maintenance case study achieved a large, measured reduction in false alarms — on the order of 90 percent — through a closed operator-feedback loop: the maintenance crews investigated the alerts, labeled which were real, and the model retrained on those labels, so the false-alarm rate fell sharply over successive rounds. This is the industrial-QA domain's best-measured quantitative benefit, and it comes from an anonymized study site rather than a named-manufacturer press release, which is the pattern across this domain — the peer-reviewed magnitudes are at anonymized or smaller sites, while the named deployments report their benefit through corporate and trade channels.

Sources: hermansa2021a

Appears on: /domains/cases/heavy-industry-pdm, /pan-lab

The lesson the case carries is that the same closed feedback loop that produced the benefit is the thing that can break it, because the loop depends on the crews continuing to engage with the alerts — investigating them, labeling them, responding — and that engagement fails in two opposite directions. Alert fatigue: if too many false alarms arrive before the loop has tuned them down, crews stop trusting the alerts and stop responding, so the feedback the model needs to improve never arrives and the loop stalls. Automation bias: if crews defer to the alerts and stop applying their own judgment, the labels the model retrains on become an echo of its own calls rather than an independent check. Either way the loop degrades, so the measured benefit is contingent on the loop staying calibrated — enough trust that crews respond, enough independence that their labels still carry real judgment.

Sources: romeo2025a, wittbold2026

Appears on: /domains/cases/heavy-industry-pdm, /pan-lab

Machine-learning automated visual inspection of filled injectable drug products flags particulate and cosmetic defects that manual inspection or fixed-rule cameras would otherwise judge. In this safety-critical, regulated manufacturing setting the error trade-off is asymmetric and deliberate: a false accept — a missed defect in an injectable that reaches a patient — is a patient-safety failure, while a false reject — scrapping a good vial — is a cost, so the system is tuned to over-reject rather than risk a miss. Because the inspection sits inside a validated pharmaceutical quality process, the AI cannot simply be switched on; it must be qualified within that process, and a regulator is actively developing the framework for how AI in drug manufacturing should be validated and monitored.

Sources: veillon2023a, usfda2023

Appears on: /domains/cases/pharma-avi-inspection, /pan-lab

Two governable surfaces follow from putting AI inside a regulated inspection. First, qualification: an AI in a validated quality process is not simply deployed but must be qualified and monitored for drift, and because the regulator's AI-specific framework is still developing, the qualification of the model's behavior over time is an emerging, not-yet-settled check rather than a solved one. Second, the human backstop: the manual inspector is what catches the false accepts the over-reject tuning is meant to avoid, so if inspectors come to defer to the AI and stop scrutinizing, that backstop erodes exactly where it matters most — the missed defect the asymmetric tuning was designed to prevent. The governable reading is that the over-reject tuning lowers the visible risk without removing it, and the qualification and the human backstop are what keep the residual risk covered.

Sources: veillon2023a, usfda2023

Appears on: /domains/cases/pharma-avi-inspection, /pan-lab

A state's statewide dropout early-warning system used ensemble machine learning to label every grade 6 to 9 student's risk of not graduating on time and delivered the label to school staff through dashboards for about a decade. An independent, decade-scale audit found the system was wrong roughly 74 percent of the time when it predicted a student would not graduate, produced higher false-alarm rates for Black and Hispanic students, and that the deployer's own internal equity research had gone unpublished — while a survey of districts found administrators reporting no training on how to interpret a 'high risk' label. The state stopped publishing the dashboards in 2023 and said it was evaluating the system's future. The deployment is the education domain's clearest case of a risk label whose error and group disparity entered how students were seen rather than the help they received.

Sources: feathers2023, knowles2015a, wisconsindepartmentofpublici2023

Appears on: /domains/cases/wisconsin-dews, /pan-lab

The lesson the case carries is that a risk label is only as good as the intervention it triggers and the training of the human who reads it. A label that is wrong most of the time, delivered to staff with no guidance on interpreting it, imports the model's error and its group disparity into how students are perceived rather than into a resourced response — the flag becomes a lens on the student rather than a trigger for help. Set against this, a large district's transparent, low-tech on-track indicator, built on interpretable research and paired with real intervention, accompanied a rise in graduation to a record level. The contrast locates the benefit in the intervention the indicator makes legible enough for staff to act on well, not in the sophistication of the prediction — an interpretable indicator that drives help can outperform an opaque model that only labels.

Sources: allensworth2007, feathers2023

Appears on: /domains/cases/wisconsin-dews, /pan-lab

A public university required students to pan their webcam around their home before an online exam, using remote-proctoring software that flags suspected cheating from the video. A federal court held that the pre-exam room scan was an unreasonable search under the Fourth Amendment — a first-of-its-kind ruling that a routine proctoring practice violated a student's constitutional rights in their own home. Separately, peer-reviewed measurement of automated proctoring found the software produced more face-detection failures, more red flags, and higher priority scores for darker-skinned and Black students, with no corresponding difference in actual cheating. The deployment is the education domain's clearest case of surveillance-based integrity AI whose costs — a rights violation and a demographic burden of suspicion — are each independently established.

Sources: ogletreev2022a, yoderhimes2022a

Appears on: /domains/cases/cleveland-state-proctoring, /pan-lab

The lesson the case carries is that surveillance-based integrity AI is not a free default: it carries a rights cost that can be independently adjudicated and a demographic burden that can be measured, and both are owed a reckoning before the surveillance is imposed, not after a court or an audit finds the harm. A room scan of a student's home was held to be an unreasonable search, so the surveillance has a rights dimension a court can rule on regardless of the integrity goal. And because the software flags darker-skinned and Black students more often with no more actual cheating, and a flag is an accusation the student must answer, a disparate flag rate is a disparate burden of suspicion. The governable surfaces are the proportionality of the surveillance to the integrity problem it is trying to solve, and the measured flag rate by group.

Sources: yoderhimes2022a, ogletreev2022a

Appears on: /domains/cases/cleveland-state-proctoring, /pan-lab

A parcel carrier's route-optimization system is a documented operations-research success: it re-optimizes delivery routes across the fleet and was reported to save on the order of 100 million miles and about 10 million gallons of fuel a year, a genuine and peer-reviewed efficiency gain. The same system that computes the efficient route also dictates it to the driver and monitors adherence through vehicle telematics, so the efficiency is enforced through workplace surveillance — the optimization and the monitoring are one system, and the driver's discretion over how to run the route is what it replaces. The benefit is real and measured in miles and fuel; the cost is the driver autonomy the enforcement removes and the surveillance the enforcement requires.

Sources: holland2017, levy2023

Appears on: /domains/cases/ups-orion-routing, /pan-lab

The lesson the case carries is that an optimization which manages the worker executing it couples the efficiency gain to a cost the efficiency metric does not see: the worker's autonomy, and the surveillance required to enforce the plan. The system measures miles and fuel, not whether the pace it sets is feasible for a person or whether the monitoring it requires is proportionate — so the governable surfaces are whether the optimization internalizes the human executing it, meaning a route that is feasible and humane rather than merely optimal on paper, and whether the surveillance that enforces it is governed rather than treated as a free byproduct of routing. An optimization is a success on its own terms and can still externalize a cost onto the worker that never appears in the miles-and-fuel number it reports.

Sources: levy2023, holland2017

Appears on: /domains/cases/ups-orion-routing, /pan-lab

A warehouse operation's algorithmic management pairs a genuine, peer-reviewed human-robot picking benefit — robots and workers collaborating to raise throughput, documented in the operations-research literature — with a documented injury-productivity trade-off. When the algorithm sets the pace of the physical work, a federal safety regulator cited the operation for exposing workers to ergonomic hazards, and a legislative inquiry tied the speed the system demands to warehouses it described as uniquely dangerous. The productivity gain and the worker-injury risk are therefore coupled: the same pace that raises units per hour is the pace regulators and the inquiry connected to injury. The benefit is real and the injury cost is separately documented, one in the OR literature and one in safety-inspection findings and a legislative report.

Sources: allgor2023, ussenatecommitteeonhealth2024, usdepartmentoflabor2023a

Appears on: /domains/cases/amazon-fulfillment-management, /pan-lab

The lesson the case carries is that when an algorithm sets the pace of physical work, the productivity metric it optimizes — units per hour — cannot see the cost the pace imposes on the body executing it. The injury shows up in safety-inspection data and a legislative inquiry, not on the throughput dashboard, so a productivity number can rise while the cost accumulates unrecorded on the metric that reports success. The governable question is whether the pace-setting internalizes the worker's safety, treating a sustainable rate as part of what 'optimal' means, or externalizes it as an injury the metric never records. Ethnographic research describes this algorithmic management as a 'game' whose rules the worker cannot change, which is what makes the pace a management decision the organization owns rather than a fact of the work.

Sources: cheon2025, ussenatecommitteeonhealth2024

Appears on: /domains/cases/amazon-fulfillment-management, /pan-lab

A federal asylum agency uses dialect-recognition AI to estimate an applicant's country or region of origin from a short speech sample, as one input into the credibility assessment of their claimed origin. The tool's reliability is limited: government-reported recognition is around 80 percent for Arabic — roughly a 20 percent error rate — and computational linguists judge separating some closely related language varieties close to hopeless. The agency's own caseworkers describe the tool as only a rough compass, too imprecise to resolve the hard cases, and its outputs as clues rather than determinations. Used honestly as one clue among several it is defensible; the documented risk is that an imprecise output acquires more authority than its accuracy supports, in a determination where the state's tool is set against the applicant's own account of who they are.

Sources: lulamae2022a, scheel2024a

Appears on: /domains/cases/bamf-dias-dialect, /pan-lab

Two governable surfaces follow from putting a low-reliability signal into a high-stakes credibility determination. First, whether the tool's documented imprecision actually bounds the weight it carries: a rough compass treated as one is honest, but the same output can harden into a credibility finding it cannot support once a phrase like the software indicates a particular origin enters the record and confronts the applicant. Second, whether the applicant can see and contest the signal: in asylum determinations the person with the most at stake and the most knowledge of their own origin is often unable to see or challenge the AI's estimate, so the correction that would catch an error is severed on exactly the side that holds the truth. The governable reading is that reliability must bound authority, and the affected person must be able to contest a signal used against them.

Sources: scheel2024a, lulamae2022a

Appears on: /domains/cases/bamf-dias-dialect, /pan-lab

A government's immigration-enforcement triage algorithm identifies and recommends people for enforcement actions — returns, bail conditions, casework — drawing on sensitive data including detention, health, vulnerability, and location-monitoring records. Uncovered through roughly a year of freedom-of-information litigation, its training materials show an asymmetric override design: officials must record a justification for rejecting a recommendation but not for accepting one. That design builds a rubber-stamping incentive into the workflow — accepting the algorithm is frictionless, overriding it requires work — so the human in the loop is nominal rather than a real check. It is the corpus's clearest documented instance of automation bias engineered into an agency workflow, in one of the highest-stakes enforcement settings a state operates.

Sources: privacyinternational2024b, privacyinternational2024a

Appears on: /domains/cases/home-office-ipic, /pan-lab

The lesson the case carries is that nominal human oversight is not real oversight. An asymmetric override — where accepting the algorithm's recommendation is frictionless and rejecting it requires a recorded justification — engineers automation bias into the process by making deference the path of least resistance, so the claim that a human makes the final decision can be true and empty at once. Two governable surfaces follow. Whether the review is genuinely symmetric: the official as free and as prompted to reject as to accept, so an error is as likely to be caught as waved through. And whether the affected person is told the AI is used and can contest it: applicants are frequently not told, which severs the correction on the side that could challenge the recommendation, so the one check that survives the asymmetric override — the person it is about — is cut out too.

Sources: privacyinternational2024c, privacyinternational2024a

Appears on: /domains/cases/home-office-ipic, /pan-lab

A large automaker developed an in-house deep-learning system to detect hairline cracks in pressed sheet-metal parts, trained on several terabytes of images drawn from seven presses at its home plant plus several sister plants, in development since mid-2016 and tested for series deployment. The documented change is a generational replacement: the system takes over an inspection duty previously performed by manual visual checks plus fixed-rule camera systems, rather than augmenting a human inspector's judgment on each part. The record — a reprint of the manufacturer's own press material with its CIO quoted — documents the development lineage, the data scale, and what the system replaced; it publishes no quantitative defect-rate figures, so the deployment's benefit magnitude is a corporate claim, not an audited measurement.

Sources: justauto2018

Appears on: /domains/cases/audi-press-shop-inspection, /pan-lab

The governance shape of this deployment is inheritance rather than assistance: by replacing the manual visual check and the fixed-rule camera generation, the learned system inherits the whole inspection duty for the defect class it covers, so there is no per-part human judgment running alongside it to catch what it misses. Its training data is pooled across presses and plants, which means one model's blind spots are correlated across every line it inspects. The failure regime is mechanism-level — drift as dies wear and parts change, complacency over an inspection nobody re-performs — because no named manufacturer, including this one, has publicly attributed a shipped-defect escape to its AI inspection.

Sources: justauto2018

Appears on: /domains/cases/audi-press-shop-inspection, /pan-lab

A rail service provider operates sensor-based predictive maintenance on a high-speed fleet under a priced availability contract: roughly 300 sensors per train read at five-minute intervals (on the order of a million readings per train-year), overlaid with human-written failure reports, maintained against a promise that refunds the full fare if a journey is delayed more than fifteen minutes. The documented results are only one noticeably delayed journey in 2,300 (by five minutes) and discovered failure signatures such as an engine-temperature pattern preceding failure by three days. The fleet outcome is documented in a vendor-side trade case study; the analytics' own precision is not published, so the model-level figures remain unstated while the operational outcome is on the record.

Sources: rcrwirelessnews2016

Appears on: /domains/cases/siemens-renfe-velaro-pdm, /pan-lab

The governance shape of this deployment is uptime-as-contract: the party that operates and tunes the analytics is the service provider who pays for misses under the refund promise, so the incentive to prevent a delay is priced into the same organization that holds the model levers - an alignment the domain's other deployments lack. Two documented dependencies temper it: the failure signatures were discovered by joining sensor streams to human-written failure reports, so the discovery loop runs on documentation crews write for their own purposes; and a continuous sensor stream is the analytics' only view of the machine, so a failing sensor and a failing train arrive looking the same until someone goes and looks.

Sources: rcrwirelessnews2016

Appears on: /domains/cases/siemens-renfe-velaro-pdm, /pan-lab

A large urban school district operationalized a transparent ninth-grade indicator - course credits earned plus no more than one core-course failure - from consortium research showing it predicts high-school graduation with about 85 percent accuracy, and wired it to school-level attention rather than to an opaque score. District graduation rates subsequently rose to record highs. The indicator is a rule anyone can read: a teacher can explain to a student exactly why they are off-track and exactly what would change it, so the contest-and-correction loop that opaque early-warning deployments sever is open by construction.

Sources: allensworth2007

Appears on: /domains/cases/cps-freshman-ontrack, /pan-lab

The documented limits are as instructive as the result. The indicator's accuracy and the district's graduation rise are associational at district scale - no randomized trial assigns schools to use it - and the benefit mechanism runs through the intervention, not the flag: an indicator wired to attention still depends on the attention being resourced, and the research base's central finding is that what predicted graduation was a condition schools could act on (freshman-year course performance), not a fixed trait of the student. The rule's power is that it points at something changeable, and the district's practice is what changed it.

Sources: allensworth2007

Appears on: /domains/cases/cps-freshman-ontrack, /pan-lab

A storied sports outlet published product-review articles under entirely fabricated author personas - invented names, AI-generated headshots, fictional biographies - produced by a third-party content contractor, with the AI involvement disclosed to no reader. An investigation surfaced the personas by reading the public site; the articles were then deleted rather than corrected, the outlet attributed the content to the contractor, and the parent company's chief executive was subsequently fired. The corroborated record documents the fabrication, the deletion, the contractor attribution, and the executive consequence.

Sources: harrisondupre2023, npr2023

Appears on: /domains/cases/sports-illustrated-advon, /pan-lab

The governance failure ran across an organizational seam: the drafting, the bylines, and the personas were produced by a contractor, and the outlet's editorial function demonstrably did not operate across that boundary - the fabrication was discovered by outside investigation, not by any internal check, and the accountability afterward ran through contract and employment rather than through any editorial process. A byline makes two claims to the reader - that a person produced this, and that the outlet's review stands behind it; this deployment fabricated the first and vacated the second, and the deletion afterward removed the evidence rather than correcting the record.

Sources: harrisondupre2023, npr2023

Appears on: /domains/cases/sports-illustrated-advon, /pan-lab

A parcel firm's customer-facing support chatbot, after a system update, was prompted by a customer into swearing and into composing a poem calling its own operator the worst delivery firm in the world. The firm attributed the behavior to the update and disabled the AI element immediately. The documented governance facts are exactly two: the update preceded the behavior, and the off switch worked - the firm learned of the incident from the customer's viral post rather than from any release gate, but the disablement was immediate once it knew.

Sources: itvnews2024

Appears on: /domains/cases/dpd-uk-chatbot, /pan-lab

The failure shape is a change-management regression, not a wrong policy or a deflection metric: guardrails that had held in production stopped holding after a change, publicly, within hours, in a channel that talks to anyone. What the deployment lacked was a release gate between the update and the public - the constraint layer's behavior after the change was tested by a customer with a prompt, not by the firm with a suite - and the discovery path ran through screenshots of one conversation going viral.

Sources: itvnews2024

Appears on: /domains/cases/dpd-uk-chatbot, /pan-lab