Skip to content
ParamergeParamerge
Stephen Lieberman
Through Paramerge
Contact

PAN Lab example

Illinois Rapid Safety Feedback

The alarm that cried wolf: a child-safety scorer

A vendor model scores maltreatment reports and points investigators at the highest-risk children. Modeled on Illinois's Rapid Safety Feedback. It failed both ways at once: thousands of kids flagged at extreme risk who weren't, and real high-risk cases it never caught. Trust a miscalibrated alarm and you get false calm and false panic together.

Stylized model of a documented deploymentChild welfare & family services

Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.

What this models

This example runs on the RSF-class predictive child-safety scorer network: 5 components and 11 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.

Evidence base: 4 assumed · 2 published baseline. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.

  • assumed

    The released tracking data is about an artifact, so the artifact is drawn: the list of children the model placed at ninety-percent-or-higher probability of death or serious injury, more than 4,100 names deep. A list that size is not a triage aid, it is a queue no investigator pool could clear - and the same released data showed children who died with low scores, so the flood and the misses are two properties of one list.

  • assumed

    This models the predictive child-safety-scoring pattern documented in the Illinois Rapid Safety Feedback case file — not a reconstruction of the actual tool.

  • baseline

    The model-to-model self-loop encodes the documented double failure as correlated error: one miscalibrated scorer over- and under-warned across the entire report stream simultaneously.

  • baseline

    The documented pattern combined many false alarms with missed true high-risk cases; the baseline treats the score as anchoring attention without reliably catching the harm it was meant to catch.

  • assumed

    The incoming-reports stream is a dynamics-bearing input node: the mandated maltreatment reports the tool scored carry a real inflow into the model, so the stream enters the failure regime rather than sitting inertly on a pathway.

  • assumed

    Demographics and differential harm to children are not modeled here; those are documented externally in the case file.

What this example does not show

  • Differential harm to children and families is not modeled here; the Lab models institutional propagation only, and that harm is documented in the case file and measured outside any diagram like this one.

Sources and evidence

What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.

  • Illinois's Rapid Safety Feedback flagged thousands of children at 90-percent-or-higher risk of serious harm — beyond any caseload's capacity to act — while children who died in known-to-system cases had not been flagged; the agency ended its use in 2017.

    empirical
    • Investigative Chicago Tribune, Can an algorithm tell when kids are in danger? (2017) https://www.chicagotribune.com/2017/12/06/can-an-algorithm-tell-when-kids-are-in-danger/
    • Investigative The Imprint, Illinois Drops Rapid Safety Feedback, A Predictive Analytics Tool (2017) https://imprintnews.org/politics/stateline-illinois-drops-rapid-safety-feedback-predictive-analytics-tool/28913
    • Trade press Government Technology, Illinois Ends Child Abuse Prediction Program (2017) https://www.govtech.com/health/illinois-ends-child-abuse-prediction-program.html

  • Internal DCFS tracking data released under Illinois public-records law showed the Rapid Safety Feedback tool flagged more than 4,100 children at a 90-percent-or-higher probability of death or serious injury within two years, including 369 children under age 9 assigned a 100-percent probability, while children who died in cases already known to the system — among them 17-month-old Semaj Crosby, found dead after at least ten DCFS investigations — were not flagged as top-risk; the roughly $366,000 program was ended in 2017.

    empirical
    • Investigative Chicago Tribune, Can an algorithm tell when kids are in danger? (2017) https://www.chicagotribune.com/2017/12/06/can-an-algorithm-tell-when-kids-are-in-danger/
    • Trade press Government Technology, Illinois Ends Child Abuse Prediction Program (2017) https://www.govtech.com/health/illinois-ends-child-abuse-prediction-program.html

Where this connects

Institutional pressures in this domain

  • Workload surge — Demand outruns staffing; per-case attention shrinks and review becomes triage.
  • Deadline pressure — Statutory or managerial timeliness rules reward fast approval of machine output over slow disagreement.
  • Staff turnover — Experienced skepticism leaves; new staff calibrate their trust on the tool itself.
  • Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).
  • Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.

All of them in context on the Child welfare & family services domain page.

Levers available here and the patterns behind them

Documented case histories