UK Technology Firms and Child Protection Agencies to Examine AI's Capability to Create Exploitation Content

Technology companies and child protection organizations will receive permission to assess whether artificial intelligence tools can generate child abuse material under recently introduced UK legislation.

Significant Increase in AI-Generated Harmful Content

The announcement coincided with revelations from a protection watchdog showing that reports of AI-generated child sexual abuse material have increased dramatically in the last twelve months, rising from 199 in 2024 to 426 in 2025.

Updated Legal Framework

Under the changes, the government will permit designated AI companies and child protection groups to inspect AI systems – the foundational systems for chatbots and image generators – and verify they have adequate safeguards to stop them from producing depictions of child exploitation.

"Fundamentally about stopping abuse before it happens," declared the minister for AI and online safety, adding: "Experts, under rigorous protocols, can now detect the danger in AI systems early."

Tackling Legal Obstacles

The amendments have been implemented because it is against the law to create and own CSAM, meaning that AI developers and other parties cannot generate such content as part of a testing process. Until now, officials had to wait until AI-generated CSAM was uploaded online before addressing it.

This legislation is designed to averting that problem by enabling to halt the creation of those images at their origin.

Legal Framework

The amendments are being introduced by the government as modifications to the crime and policing bill, which is also implementing a ban on owning, producing or distributing AI systems designed to generate exploitative content.

Real-World Impact

This week, the official visited the London base of a children's helpline and listened to a mock-up call to advisors involving a report of AI-based abuse. The call portrayed a adolescent seeking help after facing extortion using a explicit AI-generated image of themselves, constructed using AI.

"When I learn about children experiencing blackmail online, it is a source of extreme anger in me and justified concern amongst families," he said.

Alarming Statistics

A leading online safety foundation stated that cases of AI-generated abuse content – such as webpages that may contain multiple files – had significantly increased so far this year.

Instances of the most severe content – the most serious form of abuse – rose from 2,621 visual files to 3,086.

• Girls were overwhelmingly victimized, accounting for 94% of prohibited AI images in 2025
• Portrayals of newborns to two-year-olds increased from five in 2024 to 92 in 2025

Industry Reaction

The legislative amendment could "constitute a vital step to guarantee AI tools are secure before they are released," stated the head of the online safety foundation.

"Artificial intelligence systems have enabled so survivors can be victimised all over again with just a simple actions, providing offenders the ability to make possibly limitless amounts of sophisticated, lifelike child sexual abuse material," she added. "Material which additionally commodifies survivors' trauma, and renders young people, particularly female children, less safe both online and offline."

Counseling Session Information

Childline also released information of support sessions where AI has been referenced. AI-related harms mentioned in the sessions comprise:

• Using AI to evaluate body size, body and appearance
• AI assistants discouraging young people from talking to trusted adults about harm
• Being bullied online with AI-generated content
• Online extortion using AI-manipulated images

During April and September this year, the helpline delivered 367 counselling sessions where AI, chatbots and related topics were discussed, four times as many as in the equivalent timeframe last year.

Half of the references of AI in the 2025 sessions were related to mental health and wellbeing, including utilizing AI assistants for assistance and AI therapeutic apps.