Content moderation & editorial AI

YouTube ran an unintended natural experiment on automated content moderation. When the pandemic sent its human reviewers home, the platform relied more on automated removal and deliberately chose over-enforcement. From its own transparency reporting, automated removals more than doubled in a single quarter (to about 11.4 million videos), appeals roughly doubled, and the reinstatement rate on appeal jumped from about 25 percent to about 50 percent — with strikes withheld where no human had reviewed. The doubling of the reinstatement rate is the finding: it is direct evidence that the automation was making roughly twice the rate of catchable errors, and that the human review and appeals path was the loop catching them. Two things follow: over- versus under-enforcement is a chosen trade-off, and proactive removal acts before anyone sees the content — so an over-broad takedown is invisible unless appealed, and some removals are irreversible.

Meta enforces its content standards with automated classifiers at a scale no human team could match, backed by a layered correction structure: an internal appeals process, and above it an external oversight board that issues binding decisions on the cases it takes and non-binding policy recommendations. In one year the board overturned the platform's original decision in around 90 percent of the cases it decided, and the platform reported implementing or aligning with the large majority of the board's cumulative recommendations. It is the moderation domain's most institutionalized correction structure. Its limit is reach: the 90 percent is measured on selected, emblematic cases the board chooses, the board is funded through a platform-established trust (independent-adjacent), its recommendations are non-binding, and the overwhelming majority of automated decisions never reach it at all.

A storied sports outlet published product reviews under entirely fabricated author personas - invented names, AI-generated headshots, fictional bios - produced by a third-party contractor, with the AI involvement disclosed to no reader. An investigation surfaced the personas by reading the public site; the articles were deleted rather than corrected, the outlet blamed the contractor, and the parent company's CEO was subsequently fired. The governance failure ran across an organizational seam: the outlet's editorial function demonstrably did not operate across the contractor boundary, and accountability afterward ran through contract and employment rather than any editorial process.

A media outlet published AI-drafted finance explainers under a human-sounding staff byline without disclosing to readers that the articles were machine-written. When the practice came to light, the outlet's own audit found it had to issue corrections on a majority of the AI-written articles — on the order of 41 of 77. A byline implies a human review the reader trusts, and a correction rate that high is a direct measurement that the review the byline implied was not performed before publication. A later, sharper case saw another outlet publish under entirely fabricated author personas presented as real people. The editorial-AI failure is a byline that made two claims to the reader — disclosure and review — and this deployment honored neither.

Two nonprofits run the same radical design from opposite ends: a person who holds intimate material of themselves hashes it on their own device — the material never leaves their hands — and voluntary platforms match the hash against uploads, with 2 million images protected on the adult index by the operator's own count. The privacy design and the missing audit surface are one decision: nothing exists anywhere against which an index entry could ever be verified, independent researchers reconstructed recognizable faces from the very hash classes one operator's FAQ still says cannot be reverse engineered, and their disclosure has gone unanswered for three years.

The EU's Digital Services Act forced X to publish the only language-indexed error signal in commercial content moderation — a table showing automated hate-speech enforcement whose overturn rate on appeal spans 0.0 to 72.2 percent across twenty EU languages, produced by a reviewer workforce that is 88.5 percent English with thirteen official languages checked through machine translation, and an error sensor that recorded literally nothing in the languages where nobody appealed — before the disclosure regressed from language to country in the very next report.

The one large-scale content-moderation system in this atlas whose enforcement action takes nothing down. Volunteer contributors write a note on a post; other contributors rate it; an open-source bridging algorithm publishes the note only when raters whose past ratings point in opposite directions both call it helpful, and the whole note-and-rating corpus is released daily so anyone can grade the result. What appears is usually right — 97.5 percent of a physician-and-virologist-reviewed sample of displayed notes was entirely accurate, and an attached note cut reposts by 46 to 61 percent in two independent causal designs. What appears is also late and rare: 87.7 percent of notes never clear the consensus gate, a purposive sample of 2024 US election posts found 209 of 283 with accurate notes that were never shown, and the median note lands after most of the resharing is done — which is why the same studies put the system-wide effect at roughly 12 to 15 percent. No litigation anywhere concerns it, and the only regulator that has ever named it opened a proceeding in December 2023 and has published no finding.

Thirty-nine platforms share one index of terrorist and violent extremist content: approximately 2.4 million hashes covering approximately 408,000 unique and distinct items at the end of 2025, on the consortium's own count, with three quarters of the behaviourally labelled entries in the taxonomy's least determinate category, 'glorification of terrorist acts'. One company's judgement about one item on its own service becomes an input to every other member's upload comparison. Only the company that contributed a hash may take it out; the Global Internet Forum to Counter Terrorism, which owns the taxonomy, is recorded by its own commissioned reviewers as lacking the ability to modify even a label. And because the database stores hashes rather than content, there is nothing for an outside auditor to inspect — a point the peer-reviewed literature and the consortium's own reports make in almost the same words. The one erroneous entry ever described in public was two hashes of a music video that was not violent, graphic or explicit, caught because a second company happened to look.

A machine-learning classifier built to surface never-before-seen child sexual abuse material flagged two fathers' photographs of their toddlers' genital infections, taken because a nurse and a pediatrician asked for them. A trained specialist reviewer confirmed each flag against the federal definition, Google filed a CyberTipline report, and Google closed each man's entire account: a decade of Gmail, Drive documents, the whole photographic record of a child's early years, contacts, and in one case phone service from Google Fi. Police investigated both men and cleared both; one report concluded that 'the incident did not meet the elements of a crime and that no crime occurred'. Both appealed with the clearance in hand and neither was reinstated. Google's own regulated filings explain why the appeal could not absorb it: reinstatements come 'not due to an error in detection or a content-level false positive' but from context about intent, and its reports to the clearinghouse are, in the company's words, 'one-way reporting'. Nothing in the pipeline malfunctioned. The variable that would have changed the answer lived outside every input the system had.

The atlas's inverse moderation case. Every other deployment in this domain fails by acting on content it should have left alone; this one fails by routing some accounts' identified content out of enforcement into a queue that was known to be under-resourced, so the protection and the error land on different populations by construction. Listed accounts' flagged posts stayed fully up through as many as five layers of human reconsideration — a mean of more than five days, about twelve for United States content, about seventeen for Afghanistan and Syria, and 222 days at the extreme — while everyone else's likely-mistaken removals went through unreviewed when the same overstretched reviewers ran out of hours. Meta's external review body found the programme had never checked whether the exception path was actually more accurate than the ordinary one, cleared the backlog by asking, and was then refused on exactly the five recommendations that would have let anyone outside see who was protected or count what the delay cost.

The atlas's statutory-funnel case, and the only deployment in it whose central constraint is constitutional rather than technical. Federal law makes reporting mandatory, detection voluntary, and every field that would make a report usable optional at the sender's sole discretion; it then requires the clearinghouse to make every report it receives available to law enforcement, with no discretion to filter. So a private non-profit triages 21 million reports a year on information the senders may withhold — and, because a court held it part of the government for search purposes, its analysts open only the files a platform employee already opened. In one appellate record a platform reviewer opened 31 of 156 flagged files and the clearinghouse employee opened exactly those 31; the report then went to the wrong county and sat for half a year. The channel that would tell anyone which reports were worth the hours exists, is well designed and is voluntary: federal agencies returned 7,085 pieces of feedback on 3.4 million reports.

The domain's inversion, and the reason it earns a file. Everywhere else in content moderation the human reviewer is the remedy — the correction channel that catches what the classifier — the software that sorts items into policy categories — gets wrong. Here the review workforce is where the harm lands. Approximately 200 reviewers covered a sub-continent across roughly eleven African languages, working to a reported fifty-second average handling time and an eighty-four per cent accuracy score enforced against the same person from opposite directions, with the one hour of weekly wellness break approved by the managers who owned the throughput. Throughput was metered continuously; psychological load was measured once, four years later, by a hospital psychiatrist, for a court. The party that set the targets, wrote the policy, supplied the tool and composed the queue is not the party that employed, insured or medically supported the people who met them, and it defends the claims by saying so. Four years after the first filing, a Kenyan appellate court has confirmed the claims may be heard against the foreign client and has set aside every interim protection the workers had won — on the same day — and no merits ruling has been delivered.

A platform under two open European Commission investigations into whether it manages its systemic risks well enough reduced its EU content-moderation workforce from more than six thousand people at the end of 2023 to 4,596 at the end of June 2025, while its European audience grew about a quarter. It told a House of Commons committee that of approximately 430 London roles at risk, roughly a third were the people who labelled the training data for the automated moderators replacing them — because, it said, progress in the development of those models had significantly decreased the need for that labelling — and another significant proportion were ancillary roles including the teams that train moderators on the Community Guidelines. The Commission had already put on the record that the Digital Services Act 'does not prescribe any specific rules about the resources to be dedicated to content moderation'. Neither proceeding names moderation staffing as a ground. The parties who actually extracted a public account of what was happening were a German union that struck for four days, a London union-recognition ballot the redundancy notices preceded by eight days, and a select committee that asked for the safety risk assessment, was not given one, and published that fact.

A machine-learning service scores essentially every Wikipedia edit for damage as it is saved and publishes the result to anyone who asks, without a credential — and has no way of its own to revert, block, warn or report anything. Acting on the score is somebody else's job, under somebody else's governance: a volunteer-run bot on English Wikipedia holding user groups the community granted and can revoke, an operator-built agent that twelve other Wikipedias have each switched on for themselves, and volunteer patrollers working score-filtered queues at a threshold their wiki chose from a table of published consequences. The measured results cut both ways and both are public. Showing the flag narrowed the gap between how registered and unregistered editors are treated; the classifier behind it flags unregistered editors at a disparate impact ratio of 20.02 against a base-rate ratio of 7.93. There is no regulator, no court, no consent order and no reporting duty anywhere in this record — every instrument here exists because the operator and the communities chose it, and the one that made threshold-setting participatory was removed in an infrastructure migration as an old, unused feature.

A fingerprint comparison runs over every video uploaded to YouTube and checks it against reference files supplied by rights-holders admitted through an access gate. On a match, that rights-holder's standing instruction fires automatically — block the video, take its advertising revenue, or track its figures — with no case-by-case human decision on the claiming side. YouTube processed 2,502,941,368 such claims in calendar 2025 on its own count, 99.48 per cent of every copyright action taken on the platform that year, and over 90 per cent of claims take the money rather than the video. About half a per cent of claims are ever disputed, and a dispute is answered by the party that made the claim: it has thirty days, and its silence releases the claim. Every rung the uploader climbs raises the chance that party converts the matter into a legal removal request, which carries a copyright strike, and three strikes in ninety days ends the account. Of 45,724 failed appeals in one half-year, 13,841 produced a removal and 31,883 ended because the uploader cancelled the appeal or deleted the video. The whole funnel is published by the deployer, voluntarily, and it contains no measure of the claims that were wrong and never contested.