deniz.in

Markets

Weather

Loading weather

· via The Verge

OpenAI launches free CSAM classifier with AI-generated flag as tipline reports surge

OpenAI has released a free classifier that detects child sexual abuse material, including AI-generated imagery, and says it cuts false positives by 48 percent as synthetic reports threaten to overwhelm NCMEC's CyberTipline.

OpenAI launches free CSAM classifier with AI-generated flag as tipline reports surge

OpenAI's new classifier targets AI-generated abuse imagery

OpenAI has built a classifier intended to help child sexual abuse investigators sort harmful material from noise, including determining whether an image was generated by AI. According to The Verge, the company announced the tool at a Monday press briefing in Washington, DC, and is granting free access to vetted organizations, publishing extra guidance for developers on detecting this class of content, and helping train law enforcement on how AI can assist investigations.

Synthetic reports are overwhelming tiplines

The backdrop is a sharp rise in AI-generated sexual imagery of children hitting systems built for human review. The National Center for Missing and Exploited Children's CyberTipline, which collects and triages reports of child sexual abuse material, received 400,000 generative AI-related reports in 2025, nearly six times the 67,000 logged in 2024. Researchers have warned the influx could make NCMEC's workload unmanageable, and that a flood of fabricated images makes it harder to tell when a real child is actively being harmed.

North Carolina Attorney General Jeff Jackson, who co-chairs the AG Alliance's AI Task Force, described the growth in reports as "explosive" at OpenAI's event, warning that if it continues, "it is going to mean that we fail in our mission."

What the tool claims to do

Per The Verge, OpenAI says the classifier flags 48 percent fewer images than previous approaches while the images it does identify are twice as likely to contain abusive material, with the reduction coming from fewer false positives. It is designed to catch abusive AI-generated images produced by a range of tools, including open-source models and so-called nudifying apps.

John Buckley, OpenAI's head of child safety public policy, said it is the first broadly available CSAM classifier with a dedicated flag for AI-generated material. He argued that industry needs to distinguish images of fictional children from real ones, since that separation is central to how reports flow through law enforcement triage. Buckley also cautioned against treating synthetic material as harmless, noting it can erode social norms around offending and is often found alongside imagery of real children.

ChatGPT adds stricter enforcement

OpenAI's own image generator already refuses requests for CSAM. Now, users who attempt to produce it will see a new warning, developed with expert guidance, that encourages them to seek help. A first offense triggers a 24-hour suspension from ChatGPT along with a warning that the account could be permanently closed, plus support resources. Buckley said the pause reflects expert advice and international evidence on responding to CSAM-seeking versus CSAM-generation, with the goal of interrupting harmful behavior and steering people toward specialist help. OpenAI says it reports all instances of CSAM to NCMEC.

A release under political pressure

The launch lands amid intensifying scrutiny of AI companies over child safety. The Take It Down Act now requires social media platforms to remove real or AI-generated nonconsensual intimate images within 48 hours, and The Verge reports the newly elected Congress could pursue stronger action against AI firms, including SpaceXSI, whose Grok chatbot generated nonconsensual nude imagery at scale earlier this year. Last month, eight states sued OpenAI over a separate set of alleged harms to children, including claims that ChatGPT encouraged illegal substance use and suicide. The US government has stated that both AI-generated and non-AI-generated CSAM is illegal, and 48 states have laws addressing sexually explicit AI deepfakes.

Jackson noted that attorneys general mostly act after harm occurs, and that partnering with a tech company to prevent it is unusual, calling the arrangement "still basically an experiment."

Why it matters

Synthetic CSAM reports are growing far faster than human review capacity, and the gravest risk is that cases involving real children in active danger get buried in triage queues. A classifier that separates AI-generated from real imagery while cutting false positives attacks that bottleneck directly, and free access for vetted organizations lowers the barrier to adoption. The release also marks a positioning shift for OpenAI, from a defendant in child-safety lawsuits to a supplier of safety infrastructure for investigators. The performance figures are OpenAI's own and have not been independently verified, and even Jackson frames the collaboration as experimental; whether the tool measurably reduces NCMEC's workload will be the real test.

  • #openai
  • #ai-safety
  • #content-moderation
  • #deepfakes
  • #child-safety
  • #policy

Related posts