Skip to content
ParamergeParamerge
Stephen Lieberman
Through Paramerge
Contact

PAN Lab example

Google's child-safety detection and account enforcement

The flag was right about the image and wrong about the person

A father photographs his toddler son's swollen groin because an advice nurse asks him to, ahead of an emergency telehealth consultation during pandemic-era remote care. The doctor uses the photographs to diagnose the infection and prescribes antibiotics, which clear it up. The images auto-upload from his phone. Two days later his entire account is disabled for harmful content described as a severe violation that might be illegal, and within a week of the photographs police have served search warrants on the company and on his internet service provider for everything the account holds. Modeled on the documented record of a large platform operator's child-safety detection and enforcement pipeline. A near-identical case runs in parallel in another city, on a decade-old paid account, with the photographs taken at a pediatrician's request. POLICE INVESTIGATED BOTH MEN AND CLEARED BOTH: the first investigator, holding warrant returns on the entire account, concluded that the incident did not meet the elements of a crime and that no crime occurred, and the second father was cleared quickly on his correspondence with the pediatrician. Both appealed with that finding in hand. Neither got his account back. Hold four things steady before you touch anything. First, nothing here malfunctioned. The classifier surfaced never-before-seen content resembling previously confirmed material, which is exactly what it was built to do; a specialist reviewer drawn from law enforcement, child advocacy and social work agreed the image met the federal definition; and the item genuinely was a photograph of a child's genitals. The one variable that would have changed the answer, a clinician's request, lived outside every input this deployment had. That is why the confirmation pathway on this board is drawn wide rather than thin. Second, the sanction is not the item. Nothing was merely taken down: what went was mail going back more than a decade, documents, the whole photographic record of a child's first years, contacts, and telephone service from the operator's own carrier product, which meant a new number from another carrier. A later case adds work-schedule messages, bank statements and every third-party service reachable only through that sign-in. The operator's own help text runs from suspension to permanent disablement and consideration for deletion, and describes no partial remedy anywhere. Third, no published number covers this. The operator's regulated filings report measured mis-flags for the comparison against known material, 18 content items in one year and 10 in the next, every one caught during detection with nothing removed and no account access lost, and a reported rate of zero in the third. It states that classifiers carry no risk of false positives by reason of that technology alone, because their output is a queue for human confirmation. Under that definition these cases are, by construction, not counted as detection errors anywhere. Say that as a measurement-scope fact and not as an accusation about the published figures. Fourth, and this is the shape of the board: the party that got it right cannot say so. The operator's own filing calls its reports to the national clearinghouse one-way reporting, and an independent field study of the same channel found platforms rarely receive either outcome information or report-quality feedback and that the clearinghouse's structured law-enforcement outcome field is rarely completed. That study's remedy, a published set of material reported as abusive but verified not to be, which would let platforms stop reports and automated account terminations on known legal content, had not shipped. So a police finding of no crime exists as paper in a cleared man's hand and has no port on this system. What an appeal does is narrower than it looks, and the operator says so itself, three years running: a reinstatement was not due to an error in detection or a content-level false positive, but to contextual information indicating that the content was correctly identified without an intent to harm. Across three European reporting years, 297 appealing accounts produced 10 reinstatements, 216 produced 19, and 254 complaints produced 7 restorations, with zero judicial complaints in any year. Those figures cover the mail and messaging services in Europe and exclude the photo service and the video service in which the documented closures happened, which is why the annual filing is drawn as its own record on this board. One channel worked and it has no standing at all: a parent recovered her account four months after a newspaper's inquiry, another one day after a reporter asked following more than a month of denied appeals, and the operator had no answer for how to escalate a denial of an appeal other than emailing that reporter. And the pressure runs the other way at the same time. The federal statute prices a failure to report at up to one million dollars for the largest providers while disclaiming any duty to search or scan, and in March 2026 a United States senator opened an investigation into the operator for failing to remove such material fast enough, demanding removal-response times, per-product report counts, staffing and budgets, and any decision limiting deployment of detection technology. That is a demand and an allegation and no finding. It is here because it prices the asymmetry you are governing: under-detection draws a congressional document demand, and a wrongly closed account draws a two-appeal form. Before you pick a target level: this board cannot be won under Service and Safety Targets or All Governance Targets, and you are not short of money here. None of the 21,486 arrangements inside your twelve clears every gate, and neither does any arrangement at any strength once the budget is ignored entirely. What holds you out is the pathways rather than the benefit. Four of them stay open whatever you buy. Two are the automatic scanning of everything that arrives, one on each detection channel. The third is the supplied list of known material one of those channels compares against. The fourth is the single closure that reaches every service at once. Four instruments already take you to that floor, and adding the other nine moves it by nothing. Lift the pathway requirement alone and more than two hundred and fifty arrangements under either target level clear every remaining gate for nine of your twelve, leaving three unspent. Lift any single service margin instead and nothing wins at all. Explore and Service Targets Only can be won, and cheaply: two instruments, costing five of your twelve.

Stylized model of a documented deploymentContent moderation & editorial AI

Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.

What this models

This example runs on the Platform-class detection and account closure network: 14 components and 27 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.

Evidence base: 11 published baseline · 2 measured. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.

  • baseline

    The detection path in both documented cases was the classifier for never-before-seen material, not hash matching. The photographs were newly created, so no hash of them could have existed; the reporting states the images were flagged by the artificial intelligence and that a human moderator then reviewed them to confirm they met the federal definition. Two trade outlets attribute the flag to a third party's hash-matching product and this network does not repeat that attribution.

  • baseline

    Police investigated both men and cleared both. The San Francisco investigator, holding warrant returns on the entire account, concluded that the incident did not meet the elements of a crime and that no crime occurred; the Houston father was cleared quickly after showing a detective his correspondence with the pediatrician. Every pathway on this network that carries a flag, a confirmation, a report or a closure is read with that clearance attached.

  • baseline

    Nothing on this network is drawn as a malfunction, because the record documents none. The classifier surfaced content resembling confirmed material, which is what it was built to do; a trained specialist agreed the image met the federal definition; and the item genuinely was a photograph of a child's genitals. The confirmation pathway is drawn wide for that reason. The variable that would have changed the answer, a clinician's request, was outside every input this deployment had.

  • baseline

    No error rate exists for the classifier channel, anywhere, and its absence is a finding of the case rather than a gap in this model. The operator's published error counts cover hash matching alone: 18 content items incorrectly flagged in 2023, 10 in 2024, and a reported rate of zero in 2025, each caught during detection with nothing removed and no account access lost. The operator states that classifiers carry no risk of false positives by reason of that technology alone, because their output is a queue for human confirmation. Under that definition the documented cases are by construction not counted as detection errors anywhere. This is a measurement-scope fact and not an accusation that any published number is false.

  • measured

    The appeal and reinstatement figures on this board are European and cover the mail and messaging services under a European regulation. They exclude the photo service and the video service, which are the services in which the three documented closures happened. They are the best public appeal-outcome data that exists for this deployment and they are not the numbers for the pipeline that closed those accounts, which is why the annual filing is drawn as its own record and its compile pathway is drawn narrow.

  • measured

    What a successful appeal overturns is the inference of intent and not the content verdict, in the operator's own repeated regulated wording: a reinstatement was not due to an error in detection or a content-level false positive, but to contextual information identified during the appeal indicating that the content was correctly identified but did not appear to be possessed or shared with intent to harm, abuse or exploit children. The appeal desk's narrow read of the account is carried in its own description and on its write back into the account rather than drawn as a pathway of its own, and that write is drawn narrow for the same reason.

  • baseline

    The zero on the pathway from the investigation back into the referral record records a channel that was attempted and carried nothing, rather than one nobody tried. Both men presented the police clearance and both were refused. The operator's own filing describes its reports to the clearinghouse as one-way reporting, and a 2024 field study found platforms rarely receive either outcome information or report-quality feedback and that the clearinghouse's structured law-enforcement outcome field is rarely completed.

  • baseline

    The press channel is drawn with a measured effect rather than as an auditor of the whole process, because what the record shows it doing is getting two accounts restored, and that effect is drawn as a check that gets the appeal desk to reverse a denial rather than as a write of its own. It is not drawn wider than that: its reach is one case at a time, its coverage is whatever a newsroom takes up, and the two men whose cases the reporting was about had not recovered their accounts when the reporter who broke the story was asked directly in December 2025. That outcome answer is second-hand through the person who asked her and no first-party statement of it exists.

  • baseline

    No queue depth, backlog, per-reviewer workload or review-time figure has ever been published for this deployment by anyone. The demand value rests on published report and suspension volumes across several years and counting bases, which are never chained into a series, and the capacity value rests on the published split between provider-detected and publicly reported material rather than on any measure of review staffing.

  • baseline

    The two decisions in the record went the operator's way on different questions and neither involves either documented father. A federal district court dismissed a self-represented plaintiff's contract, fraud and due-process challenge to an account termination at the pleading stage in July 2024. A state supreme court held unanimously in February 2026 that the operator acted as a private actor rather than as an instrument or agent of the government when it scanned files and an employee viewed those flagged; that case involves a convicted defendant and is cited on this board only for the legal architecture. Neither father sued, and that absence is stated as an absence rather than as evidence that the conduct was lawful or that remedies exist.

  • baseline

    The pressure on this deployment runs in both directions in the same period and the network is read with both on it. The federal statute requires a report as soon as reasonably possible after actual knowledge, with failure-to-report penalties running to one million dollars for a repeat violation by the largest providers, while disclaiming any duty to monitor or to affirmatively search, screen or scan. In March 2026 a United States senator opened an investigation into the operator for failing to remove such material fast enough, demanding removal-response times, per-product report counts, staffing and budgets, and any decision limiting deployment of detection technology. That is a demand and an allegation and no finding, and it is carried because it prices the asymmetry: under-detection draws a congressional document demand, while a wrongly closed account draws a two-appeal form.

  • baseline

    Account holders are not modelled. No suspension, appeal, reinstatement or outcome for any person is computed from anything drawn here, and no individual user, parent or reviewer is named or characterised. The two fathers are described by role and place because the record's own naming posture requires it, and the counts of documented cases are recorded external observations rather than a rate: the operator's volume figures come from different periods and counting bases and none of them is a denominator for two documented cases.

  • baseline

    Several documented flows are drawn inside a neighbouring pathway rather than as pathways of their own, and each is stated where it is carried. A classifier hit reaches the statutory report and the police only through the specialist confirmation and the report it writes; a flagged item reaches the appeal desk only through the suspension; and a reporter's inquiry restores an account through the appeal desk's own reversal. None of these foldings changes a width, and none of them is a claim that the folded flow is weaker than the record shows.

What this example does not show

  • STATUS AND POSTURE, verbatim from the evidence dossier and load-bearing. Operating and unchanged in its essentials. The enforcement action (whole-account closure) and the referral path (one-way CyberTipline report) are still in place; the documented remedy is an appeals-process change (more specific suspension reasons, an opportunity to supply context) announced in late 2022. Neither documented father regained his account. No litigation was brought by either man; the two US decisions that do exist (D.D.C. 2024; Wis. 2026) both went Google's way on different questions.
  • HANDLE WITH CARE, and this governs every sentence on the board. This is a case about two men wrongly suspected of a serious crime against their own children. Police investigated both and cleared both, and that clearance appears wherever the flagging, the confirmation, the report or the closure appears. Nothing here is rendered as a case where the system may have been right, because the record does not support that reading. The two fathers are described by role and place only, never by the first names the reporting carries.
  • THE ERROR RATES ARE NOT COMPARABLE TO THESE CASES, and that is a measurement-scope fact rather than an accusation. The operator's published error metrics cover hash matching alone: 18 mis-flags in 2023, 10 in 2024, a reported rate of zero in 2025, every one caught before any action. It states that classifiers carry no risk of false positives by reason of that technology alone because their output is only a queue for human confirmation. Under that definition the medical-photo cases are by construction not counted as detection errors anywhere. No published number is alleged to be false.
  • THE APPEAL STATISTICS ARE EUROPEAN AND COVER OTHER SERVICES. The 297-to-10, 216-to-19 and 254-to-7 figures come from the operator's Irish entity's filings under Regulation (EU) 2021/1232 and, from 2025, Regulation (EU) 2024/2916, and they cover the mail and messaging services and, in the earliest year, a historic chat product. They exclude the photo service and the video service, which are the two services in which the documented closures happened. They are the best public appeal-outcome data that exists and they are not the numbers for the pipeline that closed those accounts. The 2025 denominators are complaints rather than appealing accounts and its suspension count is smaller than its complaint count, so those three years are carried as a trend and never as a series.
  • THE DETECTION PATH IS STATED PRECISELY. The photographs at issue were newly created, so a comparison against known material could not have matched them; the flag came from the classifier for never-before-seen content and a specialist reviewer then confirmed it against the federal definition. Two trade outlets attribute the flag to a third party's hash-matching product, and one of them carries that attribution in its own headline slug. This board does not repeat it. The operator's spokesperson line about a combination of hash matching and artificial intelligence describes the whole system, not these flags.
  • VOLUME FIGURES COME FROM DIFFERENT PERIODS AND SCOPES AND ARE NEVER CHAINED. Over 600,000 reports and 270,000 accounts disabled is 2021; over one million reports is the first half of 2022; more than two million is 2022; 1,470,958 tips is 2023 as counted by the clearinghouse; approximately 270,000 accounts suspended annually is an operator round number rather than a measured year. The two documented cases are never presented as a rate against any of them, and no denominator exists: a trade analysis noted at the time that most people affected by this shape will never speak publicly, which cuts both ways on any attempted rate.
  • THE OUTCOME ANSWER IS SECOND-HAND AND IS LABELLED THAT WAY. That neither man recovered his account rests on a writer relaying the reporter's direct answer to him in December 2025, with the note that one of them retrieved some account data that had been turned over to police. No operator statement, court record or first-party account confirms it. It is the strongest evidence located and it is attributed to that chain rather than asserted flatly.
  • NO LITIGATION BY THE DOCUMENTED PARTIES, stated as an absence and never as vindication. Neither father sued and no class action or regulatory enforcement over these facts was located. The federal district court decision in July 2024 concerns a different, self-represented plaintiff and was a pleading-stage dismissal. The state supreme court decision of February 2026 is a criminal suppression appeal involving a convicted defendant, is quarantined from these cases entirely, and is cited only for the private-actor and no-duty-to-scan architecture. The absence of suits is not evidence that the conduct was lawful or that remedies exist.
  • THE POLICY CHANGE IS REAL AND PARTIAL, and dating it is fiddly. The operator's trust and safety leadership said on 28 October 2022 that the company was actively working on more specific suspension reasons and better appeals; a shipped path was reported on 30 December 2022; the December 2023 reporting describes the more-specific-reason change as already in effect. It changed what a person is told and what they may submit. It did not change the sanction, the two-appeal cap, the one-way referral, or the fact that a police clearance has no port, and cases in 2023 and 2026 show the same shape after it.
  • THE MARCH 2026 CONGRESSIONAL INVESTIGATION IS ALLEGATION, NOT FINDING. A senator's announcement and document demand carry no adjudicated conclusion and no operator response was located. It is used for exactly what it demonstrably is: evidence of the direction and intensity of pressure on the detection side in the same period, which is what makes the reinstatement asymmetry legible. It is not a finding that the operator under-removes, and it is not a vindication of the account closures.
  • OPERATOR STATEMENTS ARE LABELLED AS OPERATOR STATEMENTS. That the company will reinstate an account if an error has been made, that it understands violative content was not uploaded maliciously, and that classifiers carry no risk of false positives by themselves are the operator's characterisations, attributed and not adopted. The company has never conceded error in either documented case; this board says that it stood by the decisions, not that it admitted a mistake.
  • ACCOUNT HOLDERS ARE NOT MODELLED. No suspension, appeal, reinstatement or outcome for any person is computed from anything on this diagram. The people this deployment acted on appear as recorded external observations and as a count of two documented cases, never as a dynamic and never as a rate. The one February 2026 trade source describing a wave of similar bans is user-report aggregation with no operator comment, is carried for the single claim that the pattern persists, and is never used for counts.

Sources and evidence

What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.

  • In February 2021 a father in San Francisco photographed his toddler son's swollen, painful groin because an advice nurse asked for images ahead of an emergency telehealth consultation during pandemic-era remote care; the doctor used the photographs to diagnose the infection and prescribed antibiotics, which cleared it up. The images auto-uploaded from an Android phone to Google Photos, and two days later his entire Google Account was disabled for 'harmful content' that was 'a severe violation of the company's policies and might be illegal'. Google reported the material to the National Center for Missing & Exploited Children's CyberTipline, and San Francisco police served search warrants on Google and on his internet service provider within a week of the photographs, seeking 'everything in Mark's Google account: his internet searches, his location history, his messages and any document, photo and video'. He learned of it in December 2021, when an envelope arrived containing the warrants and a letter telling him he had been investigated. THE POLICE CLEARED HIM: the investigator, with access to everything Google held, concluded that 'the incident did not meet the elements of a crime and that no crime occurred'. A near-identical case ran in parallel in Houston, where a father photographed his toddler's genital infection at a pediatrician's request, the images auto-backed up and were sent to his wife over a Google messaging service, and his decade-old paid account was locked while he was in the middle of buying a house; he too was cleared, quickly, after showing a detective his correspondence with the pediatrician. Both men appealed with the exculpatory material in hand: 'A few days after Mark filed the appeal, Google responded that it would not reinstate the account, with no further explanation.' Google publicly stood by the decisions and has never conceded error in either case. Asked directly in December 2025, the reporter who broke the story said neither parent had recovered his account, though one had been able to retrieve some account data that was turned over to police; that answer is relayed second-hand by the writer who asked her, and no first-party statement of the outcome exists.

    empirical
    • Investigative Hill, K. (2022, August 21). A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. The New York Times (publisher blocked to automated fetch on 2026-08-28; carried through the corroborating sources recorded in this block) https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html
    • Investigative Bhuiyan, J. (2022, August 22). Google refuses to reinstate man's account after he took medical images of son's groin. The Guardian https://www.theguardian.com/technology/2022/aug/22/google-csam-account-blocked
    • Advocacy Mullin, J. (2022, August 22). Google's Scans of Private Photos Led to False Accusations of Child Abuse. Electronic Frontier Foundation https://www.eff.org/deeplinks/2022/08/googles-scans-private-photos-led-false-accusations-child-abuse
    • Trade press Gizmodo (2022, August 22). Google Flagged Parents' Photos of Sick Children as Sexual Abuse (carried for the sanction scope and the parallel case only; its hash-matching attribution of the flag is not adopted here) https://gizmodo.com/google-csam-photodna-1849440471
    • Trade press Heer, N. (2025, December 17). From 2022, on Google's Account Locking Practices. Pixel Envy (relaying Kashmir Hill's direct answer that neither father recovered his account) https://pxlnv.com/linklog/google-account-locking/

  • Google describes its child-safety detection as two technologies used in combination and augmented by human review. Hash matching compares uploads against verified sets of previously confirmed material, with hashes drawn from the Internet Watch Foundation, the National Center for Missing & Exploited Children, and content Google itself confirms, each independently verified before deployment. Separately, machine-learning classifiers trained on confirmed material 'flag new content that is very similar to patterns of previously confirmed CSAM'. Specialist reviewers with backgrounds in law enforcement, child advocacy, and social work confirm both hash matches and classifier hits before action. THE DOCUMENTED FLAGS CAME FROM THE CLASSIFIER, NOT FROM HASH MATCHING: the photographs in both 2021 cases were newly created, so no hash of them could have existed, and the New York Times reported that the images were flagged by the artificial intelligence and that 'a human content moderator for Google would have reviewed the photos after they were flagged by the artificial intelligence to confirm they met the federal definition of child sexual abuse material'. Two trade outlets attribute the flag to Microsoft PhotoDNA hash matching, one of them in its own headline slug, and that attribution is wrong for these photographs. Google's spokesperson line describing 'a combination of hash matching technology and artificial intelligence' describes the whole system, not these flags. The same classifier layer is distributed to other platforms through Google's Child Safety Toolkit: a Content Safety API that prioritises never-before-seen images and video for partner review and a video hash-matching tool, with Google stating that partners 'process billions of files' and naming NCMEC, Adobe, Yahoo, Nextdoor, Scribd, and Substack among them, and the 2023 reporting adding that the new-material classifier was made available to other companies including Meta and TikTok. Volume, from different periods and counting bases that are not chained: over 600,000 CyberTipline reports and more than 270,000 accounts disabled in 2021; over one million reports in the first half of 2022; more than two million across 2022; 1,470,958 reports in 2023 as counted by NCMEC, about 4 percent of all platform tips; and approximately 270,000 accounts suspended for this ground annually as an operator round number.

    empirical
    • Vendor Jasper, S. (2022, October 28). How we detect, remove and report child sexual abuse material. Google (The Keyword) https://blog.google/technology/safety-security/how-we-detect-remove-and-report-child-sexual-abuse-material/
    • Investigative Hill, K. (2022, August 21). A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. The New York Times (publisher blocked to automated fetch on 2026-08-28; carried through the corroborating sources recorded in this block) https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html
    • Vendor Google. Child Safety Toolkit: tools for partners (the Content Safety API and the video hash-matching tool distributed to other platforms) https://protectingchildren.google/tools-for-partners/
    • Investigative Hill, K. (2023, December 10). How your child's online mistake can ruin your digital life. The New York Times, read via Seattle Times syndication https://www.seattletimes.com/business/technology/how-your-childs-online-mistake-can-ruin-your-digital-life/
    • Academic Grossman, S., Pfefferkorn, R., Thiel, D., Shah, S., DiResta, R., Perrino, J., Cryst, E., & Stamos, A. (2024, April 22). The Strengths and Weaknesses of the Online Child Safety Ecosystem: Perspectives from Platforms, NCMEC, and Law Enforcement on the CyberTipline and How to Improve It. Stanford Internet Observatory / Cyber Policy Center https://stacks.stanford.edu/file/druid:pr592kc5483/cybertipline-paper-2024-04-22.pdf

  • No error rate exists for the detection channel that produced the documented cases, and the reason is a definition rather than an omission. Google's regulated filings under Regulation (EU) 2021/1232 publish measured error counts for hash matching alone: 18 content items incorrectly flagged in 2023 and 10 in 2024, in both years caught by human review during detection with nothing removed, nothing reported externally, and no account access lost; the 2025 filing reports 1,604 items automatically flagged as known material, 335 subject to human review, and a reported error rate of zero. For classifiers, Google states that because they only sort and prioritise content for human confirmation, 'there is no risk of false positives by reason of this technology alone'. Under that definition the medical-photo cases are by construction not counted as detection errors in any published figure. This is a measurement-scope fact and not an accusation that any published number is false. The scope of those filings is narrower still: they cover Google's messaging and mail services in the European Union and exclude Google Photos and YouTube, which are the services in which the documented account closures happened. Independent analysis converges on the shape rather than the rate. The peer-reviewed scanning literature states that 'false positives are inevitable: some innocuous content will be flagged as targeted', that error likelihood rises where training and deployment distributions differ, and gives the scale arithmetic explicitly. A Stanford trust-and-safety fellow told the New York Times in 2023 that adjudicating value judgements at this scale is 'just a very, very hard-to-solve problem' and that 'when you roll the dice that many times, you are going to roll snake eyes'. The Electronic Frontier Foundation recorded comparative evidence in 2022 that most flagged accounts are non-malicious: Facebook found 75 percent of accounts reported for alleged CSAM had sent 'non-malicious' images, and LinkedIn reported 75 accounts to EU authorities with manual review confirming CSAM in 31. A trade analysis at the time cautioned that most people affected by this shape will never speak publicly, which cuts both ways on any attempted rate.

    empirical
    • Vendor Google Ireland Limited (2024, 2025, 2026). Transparency Reports under Regulation (EU) 2021/1232 and Regulation (EU) 2024/2916, reporting periods 1 January to 31 December 2023, 2024 and 2025 https://storage.googleapis.com/transparencyreport/report-downloads/pdf-report-23_2025-1-1_2025-12-31_en_v1.pdf
    • Academic Abelson, H., Anderson, R., Bellovin, S. M., Benaloh, J., Blaze, M., Callas, J., Diffie, W., Landau, S., Neumann, P. G., Rivest, R. L., Schiller, J. I., Schneier, B., Teague, V., & Troncoso, C. (2024). Bugs in our pockets: the risks of client-side scanning. Journal of Cybersecurity, 10(1), tyad020 https://academic.oup.com/cybersecurity/article/10/1/tyad020/7590463
    • Investigative Hill, K. (2023, December 10). How your child's online mistake can ruin your digital life. The New York Times, read via Seattle Times syndication https://www.seattletimes.com/business/technology/how-your-childs-online-mistake-can-ruin-your-digital-life/
    • Advocacy Mullin, J. (2022, August 22). Google's Scans of Private Photos Led to False Accusations of Child Abuse. Electronic Frontier Foundation https://www.eff.org/deeplinks/2022/08/googles-scans-private-photos-led-false-accusations-child-abuse

  • The unit of enforcement in this deployment is the account rather than the item. The San Francisco father lost more than a decade of Gmail, Google Drive documents, Google Photos including the entire photographic record of his son's first years, contacts for friends and former colleagues, and his Google Fi phone service, which required obtaining service and a new number from another carrier. The Houston father, a paying customer, lost a decade-old account in the middle of buying a house, disrupting the transaction. The December 2023 case shows the downstream blast radius on a third account: work-schedule messages, bank statements, and third-party applications signed in with the Google Account; February 2026 trade reporting adds gig-work platforms, banking, and home-security services. Google's own current help text sets the shape of the remedy: disablement reasons include child sexual abuse and exploitation; 'For some policy violations, Google will review up to 2 appeals'; data download is unavailable for certain violations 'including but not limited to: Valid legal requests, Account hijacking, Egregious content violations'; and if an appeal is not approved 'your entire Google Account will remain unavailable... your account will be permanently disabled and considered for deletion'. Google's own transparency-centre description of its appeals estate lists product-specific appeal forms for advertising, video, applications, maps and search and a general account-restoration path, and lists no child-safety-specific redress instrument; the child-safety entry point is a reporting form rather than a redress form. No partial remedy appears anywhere in the record.

    empirical
    • Investigative Hill, K. (2022, August 21). A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. The New York Times (publisher blocked to automated fetch on 2026-08-28; carried through the corroborating sources recorded in this block) https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html
    • Trade press Gizmodo (2022, August 22). Google Flagged Parents' Photos of Sick Children as Sexual Abuse (carried for the sanction scope and the parallel case only; its hash-matching attribution of the flag is not adopted here) https://gizmodo.com/google-csam-photodna-1849440471
    • Investigative Hill, K. (2023, December 10). How your child's online mistake can ruin your digital life. The New York Times, read via Seattle Times syndication https://www.seattletimes.com/business/technology/how-your-childs-online-mistake-can-ruin-your-digital-life/
    • Vendor Google. Fix a disabled Google Account (Google Account Help; the two-appeal cap and the permanent-disablement endpoint) https://support.google.com/accounts/answer/40695?hl=en
    • Trade press PiunikaWeb (2026, February 3). Google Photos users hit by wave of false CSAM account bans (user-report aggregation, no operator comment; carried only for the claim that the pattern persists) https://piunikaweb.com/2026/02/03/google-photos-false-csam-flags-users-locked-out/

  • Google's own regulated filings state what an appeal overturns, in the same construction three years running: a reinstatement 'was not due to an error in detection or a content-level false positive, but rather a reinstatement based on contextual information identified during the appeal process, which indicated that the content was correctly identified but did not appear to be possessed or shared with intent to harm, abuse, or exploit children'. The appeal re-reads intent; the record does not describe it re-reading the image. The measured yields, under Regulation (EU) 2021/1232 and, from 2025, Regulation (EU) 2024/2916: in 2023, 635 accounts identified by automated technologies, 734 CyberTipline reports, 1,558 content items, 297 accounts appealed, and 10 reinstated; in 2024, 503 accounts, 508 reports, 1,824 content items, 216 appealed, and 19 reinstated; in 2025, on a new Commission standard form, 380 known-material reports covering 1,419 images and 92 videos, 483 content items removed, 114 accounts suspended, 254 complaints lodged with the internal mechanism, 7 accounts restored, and 6 instances where an initial content verdict was overturned on review with the file made available to the user for download. Judicial complaints in all three years: zero. SCOPE, stated every time these figures are used: they are European Union only, cover Google's messaging and mail services and, in 2023, a historic chat product, and exclude Google Photos and YouTube, which are the services in which the documented closures happened. The 2025 denominators are complaints rather than appealing accounts and its suspension count (114) is smaller than its complaint count (254), so the three years are a trend rather than a series. Google also describes the quality regime behind the verdicts these appeals contest: weekly quality audits of reviewer verdicts, precision and recall monitored and reported monthly against an agreed target (its own example is 95 percent) with root-cause analysis and corrective action below it, and reporting automated only for content matching a hash previously confirmed as CSAM by a manual reviewer, with automated verdicts sampled rather than each manually reviewed before reporting.

    empirical
    • Vendor Google Ireland Limited (2024, 2025, 2026). Transparency Reports under Regulation (EU) 2021/1232 and Regulation (EU) 2024/2916, reporting periods 1 January to 31 December 2023, 2024 and 2025 https://storage.googleapis.com/transparencyreport/report-downloads/pdf-report-23_2025-1-1_2025-12-31_en_v1.pdf

  • The referral path carries no return leg, and Google says so in its own regulated filing: 'While Google's reports to the NCMEC CyberTipline are one-way reporting, the information sharing and collaboration with NCMEC and NGOs provide the necessary feedback loop to continuously improve Google's detection technology.' Stanford's 2024 ecosystem study, based on dozens of interviews with platforms, NCMEC, and law enforcement, found independently that platforms 'rarely get either' outcome information or report-quality feedback from law enforcement, and that NCMEC built a structured law-enforcement outcome field into the report flow which law enforcement rarely fills in. A platform respondent described the resulting posture: without that feedback 'you are stuck in a system where turning over anything is better than trying to think through how to do this well.' The study's proposed remedy names this deployment's exact mechanism: NCMEC should 'publish a negative hash set of images that have been reported as CSAM but have been verified to not be violative... This would allow platforms to stop reports (and automated processes such as account termination) on known legal content.' NCMEC's April 2024 response appreciated the analysis, disputed nothing specifically, and said it would explore the recommendations; no such published negative hash set was located as of this run. The scale of the channel on the receiving side: NCMEC's CyberTipline received 21.3 million reports in 2025, of which 21,181,300 (99.2 percent) came from electronic service providers and 170,193 (0.8 percent) from the public, with five providers accounting for more than 75 percent of reports; NCMEC referred more than 18.8 million reports to law enforcement, designated more than 4.5 million as informational, issued over 172,000 removal notices with a 2.6-day average takedown, and distributes to task forces in all fifty US states and to law enforcement in 170 countries. In 2023, 245 platforms reported at all, 41 percent of them submitting 20 or fewer reports, and NCMEC escalated 63,892 tips as urgent or imminent-danger. So an investigator's conclusion that no crime occurred exists as paper in a cleared person's hand and has no documented input anywhere in the enforcement system.

    empirical
    • Vendor Google Ireland Limited (2024, 2025, 2026). Transparency Reports under Regulation (EU) 2021/1232 and Regulation (EU) 2024/2916, reporting periods 1 January to 31 December 2023, 2024 and 2025 https://storage.googleapis.com/transparencyreport/report-downloads/pdf-report-23_2025-1-1_2025-12-31_en_v1.pdf
    • Academic Grossman, S., Pfefferkorn, R., Thiel, D., Shah, S., DiResta, R., Perrino, J., Cryst, E., & Stamos, A. (2024, April 22). The Strengths and Weaknesses of the Online Child Safety Ecosystem: Perspectives from Platforms, NCMEC, and Law Enforcement on the CyberTipline and How to Improve It. Stanford Internet Observatory / Cyber Policy Center https://stacks.stanford.edu/file/druid:pr592kc5483/cybertipline-paper-2024-04-22.pdf
    • Vendor National Center for Missing & Exploited Children (2024). In Response to Stanford Internet Observatory Cyber Policy Center's Report (a non-disputing receipt of roughly 300 words; no finding acknowledged, none disputed) https://www.missingkids.org/blog/2024/in-response-to-stanford-internet-observatory-cyber-policy-centers-report
    • Vendor National Center for Missing & Exploited Children, CyberTipline data. https://www.missingkids.org/cybertiplinedata

  • The channel with a demonstrated success rate in this record has no formal standing in the process. On 28 October 2022 Google's VP of Trust and Safety Operations published the company's account of the pipeline and said Google was 'actively working on ways to increase transparency' about suspension reasons and to improve the appeals experience. On 30 December 2022 the New York Times reported the resulting change: users flagged for child-safety violations now receive a more specific reason and a path to supply context, and a mother in Colorado recovered her account after four months, following a Times inquiry. The same piece summarised the two 2022 fathers: 'The police determined that the fathers had committed no crime, but the company still deleted their accounts.' In December 2023 the Times documented a third shape: a mother in Australia lost her whole Google Account after her seven-year-old uploaded a video to YouTube; the upload was flagged within minutes, her repeated appeals were denied even on a paid support channel, and the account was restored one day after a Times reporter asked about it. Google's statement was that 'we understand that the violative content was not uploaded maliciously', and the company 'had no response for how to escalate a denial of an appeal beyond emailing a Times reporter.' The policy change is real and partial: it altered what a person is told and what they may submit, and it did not alter the sanction, the two-appeal cap, the one-way referral, or the absence of any port for an exculpatory finding produced outside the platform. February 2026 trade reporting describes a fresh wave of Google Photos false-positive account bans, with one appeal rejected in about ten minutes against a stated review window of up to two days and one account restored after 24 hours on a second appeal explicitly requesting human review; that source aggregates user posts with no operator comment and no independent verification, and is carried here only for the claim that the pattern persists.

    empirical
    • Vendor Jasper, S. (2022, October 28). How we detect, remove and report child sexual abuse material. Google (The Keyword) https://blog.google/technology/safety-security/how-we-detect-remove-and-report-child-sexual-abuse-material/
    • Trade press Heer, N. (2025, December 17). From 2022, on Google's Account Locking Practices. Pixel Envy (relaying Kashmir Hill's direct answer that neither father recovered his account) https://pxlnv.com/linklog/google-account-locking/
    • Investigative Hill, K. (2023, December 10). How your child's online mistake can ruin your digital life. The New York Times, read via Seattle Times syndication https://www.seattletimes.com/business/technology/how-your-childs-online-mistake-can-ruin-your-digital-life/
    • Trade press PiunikaWeb (2026, February 3). Google Photos users hit by wave of false CSAM account bans (user-report aggregation, no operator comment; carried only for the claim that the pattern persists) https://piunikaweb.com/2026/02/03/google-photos-false-csam-flags-users-locked-out/

  • The legal and political forces on this pipeline run hardest in the direction of detecting and reporting more. 18 U.S.C. section 2258A requires a provider to report an apparent violation 'as soon as reasonably possible after obtaining actual knowledge', sets out what a report may include (subscriber identity, upload and transmission timestamps and time zones, geographic data, the visual depictions, and the complete communication), and prices a failure to report from $600,000 to $1,000,000 depending on the offence and the provider's user base; the same section, at subsection (f)(3), disclaims any duty to 'monitor any user' or to 'affirmatively search, screen, or scan'. 47 U.S.C. section 230(c)(2)(A) immunises voluntary good-faith restriction of objectionable material. On 4 March 2026 a United States senator opened an investigation into Google for failing to remove child sexual abuse material and assist survivors, demanding by 18 March ten categories of documents including internal detection and removal policies, victim removal-request response times since January 2020, annual CyberTipline reports broken out by product, every case where content was not removed within 48 hours, Trust and Safety staffing levels and budgets, and any decision LIMITING deployment of CSAM detection technology. Those are a senator's allegations and demands and no adjudicated finding, and no Google response was located. They are recorded here for what they demonstrably are: a measure of the direction and intensity of pressure on the detection side in the same period in which the redress available to a wrongly closed account remained up to two appeals and a text box. Independent legal commentary makes the same point from the other end, observing that regulatory pressure to do 'more' against this material tightens filters and raises false positives.

    empirical
    • Government 18 U.S.C. section 2258A, Reporting requirements of providers (Legal Information Institute, Cornell Law School) https://www.law.cornell.edu/uscode/text/18/2258A
    • Government Office of U.S. Senator Josh Hawley (2026, March 4). Hawley Opens Investigation into Google for Failure to Remove Child Sex Abuse Material (a document demand, not a finding) https://www.hawley.senate.gov/hawley-opens-investigation-into-google-for-failure-to-remove-child-sex-abuse-material
    • Vendor Google. Fix a disabled Google Account (Google Account Help; the two-appeal cap and the permanent-disablement endpoint) https://support.google.com/accounts/answer/40695?hl=en

  • Neither documented father sued, and that absence is recorded as an absence rather than as evidence that the conduct was lawful or that remedies exist. No class action or regulatory enforcement over these facts was located. Two United States decisions on this deployment shape do exist and both went Google's way on different questions, neither involving either man. In Baker v. Google LLC, No. 1:23-cv-02013, 2024 WL 3551878 (D.D.C. 26 July 2024), Judge Kollar-Kotelly dismissed a self-represented plaintiff's challenge to a CSAM-based account termination at the pleading stage: the contract claim because 'Plaintiff does not allege any facts indicating that Defendant was contractually prohibited from removing her Google account', the fraud claim, and the constitutional claim because 'Defendant Google is a private business, not a state actor'. That is a pleading-stage dismissal of one complaint, not a general holding that such terminations are lawful in every circumstance. In State v. Rauch Sharak, 2026 WI 4 (Wis. 24 February 2026), No. 2024AP469-CR, the Wisconsin Supreme Court held unanimously that Google 'acted as a private actor — not as an instrument or agent of the government — when it scanned Rauch Sharak's files and an employee opened and viewed files flagged as CSAM', reasoning from section 2258A(f)(3)'s disclaimer that searches are not required and from section 230(c) being 'entirely passive', and collecting the federal courts of appeals in agreement. THAT CASE INVOLVES A CONVICTED DEFENDANT, has no connection to the medical-photo cases, and is cited only for the legal architecture. The practical consequence for the correction channel is the point: the doctrine that keeps the scan outside the Fourth Amendment is the same doctrine that makes a platform wary of taking direction, or evidence, from law enforcement. A March 2026 legal round-up places both decisions in context and confirms the current posture, that Section 230 continues to immunise suspension decisions while granting no incentive to scan.

    empirical
    • Government Baker v. Google LLC, No. 1:23-cv-02013, 2024 WL 3551878 (D.D.C. 26 July 2024) (memorandum opinion, Kollar-Kotelly, J.), via CourtListener https://www.courtlistener.com/opinion/10015695/baker-v-google-llc/
    • Government State v. Rauch Sharak, 2026 WI 4 (Wis. 24 February 2026), No. 2024AP469-CR (cited for the private-actor and no-duty-to-scan architecture only; the defendant and the facts of that prosecution have no connection to the medical-photo cases) https://www.wicourts.gov/sc/opinion/DisplayDocument.pdf?content=pdf&seqNo=1082831
    • Academic Grossman, S., Pfefferkorn, R., Thiel, D., Shah, S., DiResta, R., Perrino, J., Cryst, E., & Stamos, A. (2024, April 22). The Strengths and Weaknesses of the Online Child Safety Ecosystem: Perspectives from Platforms, NCMEC, and Law Enforcement on the CyberTipline and How to Improve It. Stanford Internet Observatory / Cyber Policy Center https://stacks.stanford.edu/file/druid:pr592kc5483/cybertipline-paper-2024-04-22.pdf

Where this connects

Institutional pressures in this domain

  • Reviewer bottleneck — One fixed-capacity checking stage sits between AI output and consequence; everything queues behind it.
  • Austerity & recovery incentives — Cost-cutting and overpayment-recovery targets tilt the system toward denial and enforcement errors.
  • Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.
  • Vendor opacity — The deploying institution cannot inspect the model, data, or update pipeline it is accountable for.
  • Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).

All of them in context on the Content moderation & editorial AI domain page.

Levers available here and the patterns behind them

Documented case histories