Skip to content
ParamergeParamerge
Stephen Lieberman
Through Paramerge
Contact

PAN Lab example

Earnest AI underwriting

A neutral-looking feature and the testing no one ran

An automated model underwrites student loans. Modeled on a deployment a state attorney general settled with for $2.5 million. The model priced a school's cohort default rate into an individual's terms - a feature that names no protected class and still disparately impacted Black and Hispanic applicants - and automatically denied certain non-citizen applicants, while the organization ran no disparate-impact testing and its denial notices were inadequate. The failure here is the governance that was missing, and the remedy was to install it.

Stylized model of a documented deploymentLending & credit collections AI

Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.

What this models

This example runs on the Automated-underwriting-class with the governance an order had to write network: 6 components and 13 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.

Evidence base: 3 assumed · 2 published baseline. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.

  • assumed

    The settlement names two specific model features, and both are drawn as components rather than left as prose on a pathway. The school cohort-default-rate figure is an inbound feed, not part of the applicant's own file: it describes a group of borrowers and arrives from outside, which is exactly why input-level neutrality review passes it and only outcome testing catches it. It runs at full strength because the disparate-impact finding runs through it. The immigration-status rule is an enforcement path, because the settlement describes it denying outright - a categorical rule applies to everyone it names, so nothing is left for a person to weigh, and it runs at full strength for the same reason. Drawing them makes the case's mechanism legible: one component turns a group statistic into an individual's price, and the other reaches an outcome with no person in the path.

  • baseline

    This models the failure-then-mandated-governance pattern documented in the case file - not a reconstruction of the actual model. A state attorney general reached a $2.5 million settlement over the AI underwriting; the documented conduct is two model features (a cohort-default-rate feature disparately impacting Black and Hispanic applicants; an immigration-status rule automatically denying certain non-citizen applicants) together with the absence of disparate-impact testing and inadequate adverse-action notices. The remedy mandated the missing program - governance, testing, documentation, reporting.

  • assumed

    The cohort-default-rate feature is drawn on the record-to-model training pathway because that is where the mechanism lives: a school's aggregate default rate priced into an individual's terms is facially neutral and names no protected class, yet carries protected-class impact only outcome testing would reveal. The absent testing is drawn as the empty independent model check - the failure is the absence, since input-level neutrality is not outcome-level fairness.

  • baseline

    The mandated governance program is drawn as the latent oversight check, empty at baseline: the settlement did not invent new controls, it installed the standard ones the deployment had skipped - model governance, disparate-impact testing, documentation, and reporting. The honest reading is that the levers that would have prevented the harm are ordinary, nameable functions available from the start, so this is a resourcing choice made too late rather than an unavoidable harm.

  • assumed

    No credit outcome and no applicant is modeled here. This Lab reads institutional propagation only, and applicants are boundary-only. The documented features, the absent testing, the inadequate notices, and the mandated remedy live in the case file, and are never computed from anything in this diagram.

What this example does not show

  • No credit outcome and no applicant is modeled. The Lab reads institutional propagation only; applicants are boundary-only, and the documented features, the absent disparate-impact testing, the inadequate adverse-action notices, and the mandated remedy live in the case file, never computed on this diagram.
  • The settlement documents specific model features (a cohort-default-rate feature, an immigration-status auto-denial) and specific governance absences (no disparate-impact testing, inadequate adverse-action notices), remedied by a mandated governance program; the diagram draws those as the aggregate-feature training edge plus two latent checks (the absent test, the mandated program), not a computed harm.

Sources and evidence

What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.

  • A state attorney general reached a $2.5 million settlement with a student-loan lender over its AI underwriting. The documented conduct is the domain's cleanest failure-then-mandated-governance arc: the model used a cohort-default-rate feature — a school's aggregate default rate priced into an individual applicant's terms — that disparately impacted Black and Hispanic applicants, and an immigration-status rule that automatically denied certain non-citizen applicants, while the organization ran no disparate-impact testing and gave inadequate adverse-action notices. The remedy did not fine-and-close: it mandated the missing program — model governance, disparate-impact testing, documentation, and reporting controls — so the enforcement action wrote the governance the deployment had never built.

    empirical
    • Government Office of the Massachusetts Attorney General (2025, July 10). AG Campbell Announces $2.5 Million Settlement With Student Loan Lender For Unlawful Practices Through AI Use (Assurance of Discontinuance, Earnest Operations LLC). https://www.mass.gov/news/ag-campbell-announces-25-million-settlement-with-student-loan-lender-for-unlawful-practices-through-ai-use-other-consumer-protection-violations

  • The mechanism the case turns on is the facially-neutral aggregate feature: a cohort default rate is a property of a school, not of the applicant, and no input names a protected class — yet pricing a group's aggregate history into an individual's terms can carry protected-class impact, which is exactly what disparate-impact testing exists to catch. Here that testing was not done, so the impact went unmeasured until an enforcement action found it. The remedy installed the program the deployment lacked, which is the governable reading: an aggregate feature can look neutral input-by-input and still produce a disparity only outcome testing would reveal, and the absence of that testing is itself the failure.

    empirical
    • Government Office of the Massachusetts Attorney General (2025, July 10). AG Campbell Announces $2.5 Million Settlement With Student Loan Lender For Unlawful Practices Through AI Use (Assurance of Discontinuance, Earnest Operations LLC). https://www.mass.gov/news/ag-campbell-announces-25-million-settlement-with-student-loan-lender-for-unlawful-practices-through-ai-use-other-consumer-protection-violations
    • Regulatory Consumer Financial Protection Bureau (2022, 2023). Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms; and Circular 2023-03 on Regulation B sample forms. https://www.consumerfinance.gov/compliance/circulars/circular-2023-03-adverse-action-notification-requirements-and-the-proper-use-of-the-cfpbs-sample-forms-provided-in-regulation-b/

Where this connects

Institutional pressures in this domain

  • Vendor opacity — The deploying institution cannot inspect the model, data, or update pipeline it is accountable for.
  • Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.
  • Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).
  • Austerity & recovery incentives — Cost-cutting and overpayment-recovery targets tilt the system toward denial and enforcement errors.
  • Reviewer bottleneck — One fixed-capacity checking stage sits between AI output and consequence; everything queues behind it.

All of them in context on the Lending & credit collections AI domain page.

Levers available here and the patterns behind them

Documented case histories