deniz.in

Markets

Weather

Loading weather

· via TechCrunch

OpenAI's free CSAM classifier cuts flagged images by 48% while catching the same abuse

OpenAI released a free CSAM image classifier, developed with NCMEC, that flagged 48% fewer images at the same catch rate, alongside ChatGPT warnings, 24-hour suspensions and law enforcement tooling.

OpenAI's free CSAM classifier cuts flagged images by 48% while catching the same abuse

What OpenAI shipped

On November 16, OpenAI announced a set of child-safety tools centred on a new image classifier built to detect child sexual abuse material (CSAM), including images produced by generative AI. According to TechCrunch, select companies can use the classifier through OpenAI's API at no cost, and the company is also distributing access through the Tech Coalition and ROOST, industry bodies that coordinate trust-and-safety work.

The classifier was developed with the National Center for Missing and Exploited Children (NCMEC). Instead of comparing files against a database of known material, it inspects visual indicators of abuse in images it has never seen before and outputs a score that signals to human reviewers how a given image should be handled.

Where the 48% figure comes from

The number drawing the most attention comes from OpenAI's early testing with partners, as reported by TechCrunch: the classifier reduced the volume of flagged images by 48% while still surfacing the same amount of abusive material, and the images it did flag were twice as likely to be confirmed as abusive. Both figures are OpenAI's own, and a dev.to analysis notes that no outside party has validated them so far.

The technical significance, as dev.to explains, is about review capacity rather than detection reach. Hash-matching systems such as PhotoDNA only catch material already recorded in a database, and AI-generated CSAM is new by definition, so it evades that approach entirely. Every flagged image still has to be examined by a person, the slowest and most harrowing stage of the pipeline, so nearly halving flag volume at constant recall directly reduces that burden. The doubled precision figure is what suggests the improvement is genuine rather than a loosened threshold.

Enforcement inside ChatGPT

ChatGPT is also gaining warning screens for users who attempt prompts involving sexualized content of children. TechCrunch reports that these users see an alert, are pointed toward specialist support intended to help them stop harmful behaviour, and have their account suspended for 24 hours as a warning; repeated attempts can end in permanent removal. OpenAI says it shaped these messages with outside organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who separately evaluated a trial of the deterrent messages.

Writing on dev.to, the author adds detail from OpenAI's help centre: when the system intervenes, the conversation halts, disappears from chat history and cannot be reopened, and a temporary suspension signs the user out of the service entirely, with appeals not automatically restoring access.

Guidelines and law enforcement support

Two further pieces round out the release. OpenAI has published guidelines for developers on preventing, detecting and reporting CSAM; in early testing, the company says these helped several developers adopt child-safety tooling and register to report abusive material to NCMEC. For law enforcement, OpenAI is providing AI-powered features such as dedicated plugins, investigative tools and complex data analysis, with access controlled by program administrators, alongside investigator training and a field guide.

Caveats worth tracking

The dev.to writeup raises two concerns. First, access is gated: the free API is limited to chosen partners, meaning the small platforms and open-source projects arguably most in need of a novel-image classifier cannot obtain it yet, even though Google has offered its Content Safety API free to qualifying partners for years. Second, enforcement leans on the same classifier's judgement, and the author points to forum posts from subscribers describing wrongful CSAM-related deactivations, including one account that waited weeks for an appeal response.

The backdrop is competitive and legal pressure. TechCrunch notes lawsuits against OpenAI over harm to users, a Common Sense Media assessment that rated its technology an unacceptable risk for teens, and the use of rival chatbots such as Grok to create nonconsensual sexualized imagery of children. A Center for Countering Digital Hate study cited by dev.to found Grok generated three million sexualized images in 11 days, roughly 23,000 of them involving children. Over the past year OpenAI has published a safety blueprint, tightened content policies, retuned its models and released a ChatGPT variant aimed at teenagers.

Why it matters

Generative AI broke the assumption underlying most CSAM detection: that abusive images circulate repeatedly and can be caught by hash matching. Novel, synthetic material requires classifiers that can judge unseen images, and OpenAI is now supplying one for free, developed with NCMEC. If the 48% and precision figures survive outside scrutiny, the practical effect is fewer human reviewers exposed to traumatic content at the same level of protection. But the launch also highlights what remains unresolved: vendor-reported benchmarks, restricted access for the platforms furthest down the long tail, and an enforcement path where a classifier error can cost someone their account and chat history.

  • #openai
  • #csam
  • #content-moderation
  • #ai-safety
  • #chatgpt

Related posts