Investigating potential models of child sexual abuse could yield results
Why are AI companies that develop and distribute AI-generated child sexual abuse material (CSAM) tools valued in millions and billions of dollars?
An image generator called Stability Diffusion version 1.5, developed by AI firm Runway with funding from Stability AI, is specifically involved in CSAM production. And popular sites like HuggingFace and Civitai may have trained that model and others on real images of child sexual abuse. In some cases, organizations may be violating laws by hosting artificial CSAM objects on their servers. Why do major corporations and investors like Amazon, Google, Nvidia, Intel, Salesforce, and Andreessen Horowitz pay these companies hundreds of millions of dollars? Their support subsidizes content for pedophiles.
As AI safety experts, we are asking these questions to call out these companies and put pressure on them to take the right steps we describe below. Today we are pleased to report a major breakthrough: in response to our questions, Stable Diffusion version 1.5 has been removed from the hugging stage. But there is still a lot to do and legislation may be required to make meaningful progress.
The Scope of the CSAM Problem
Child protection advocates began sounding the alarm last year: Researchers from Stanford’s Internet Lab and tech nonprofit Thorn published a disturbing report in June 2023. They found that widely available and “open source” AI image generation tools already exist. Malicious actors misuse them to sexually exploit children. In some cases, bad actors create their own custom versions (known as micro-editing) of these models with actual child sexual abuse material to create specific images of specific victims.
Last October, a report from the U.K. nonprofit Internet Watch Foundation (which collects reports of child sexual abuse) detailed how easily malicious actors are now creating transparent AI-generated child sexual abuse content. The researchers included a “snapshot” study of a dark web CSAM forum, examining more than 11,000 AI-generated images posted over a one-month period; About 3,000 of them were sentenced to punishments severe enough to qualify as criminals. The report also calls for stronger regulatory oversight of the AI models it creates.
AI models may be used to create this content because they have seen examples before. Last December, Stanford researchers found that the most important data set used to train image-generation models contained hundreds of CSAM fragments. The most popular downloadable open source AI image generators, including the popular Standard Diffusion version 1.5 model, were trained using this data. When Runway developed that version of Stable Diffusion, Stability AI put computing power to generate the dataset and train the model, and Stability AI released subsequent versions.
Runway did not respond to a request for comment. A Stability AI spokesperson stressed that the company did not release or maintain Stability Spread version 1.5, and that the company has “implemented robust safeguards” against CSAM, including using filtered data sets for training.
Last December, researchers at social media analytics firm Graphika discovered the proliferation of dozens of “clothing removal” services, many of which were based on open-source AI image generators, including Standard Spread. These services allow users to upload photos of clothed individuals and experts create consensus intimate images (NCIIs) of both minors and adults, sometimes referred to as deep porn. Such websites can be easily found through Google searches and users can pay for the services online using credit cards. Many of these services only work with women and girls, and these types of tools have been used to target female celebrities like Taylor Swift and politicians like U.S. Representative Alexandria Ocasio-Cortez.
AI-generated CSAM has real consequences. The child protection ecosystem is already overtaxed, with millions of suspicious CSAM files reported to hotlines each year. Anything that is added to that content – especially mildly abusive – makes it much harder to find children who are active in harmful ways. To make matters worse, some bad actors are using existing CSAM to create synthetic images of these survivors – again violating their rights. Others use readily available “nude” apps to create sexual content from innocuous images of real children, then use the newly created content in sexual abuse schemes.
One Victory Against AI-Generated CSAM
It is well known in the AI community that Stable Diffusion 1.5 was trained on child sexual abuse, as were other models trained on the LAION-5B data set based on the Stanford investigation last December. These models are actively misused by malicious actors to create AI-generated CSAM. And even when they are used to create more harmless content, their use naturally revives images of child abuse in their training data. So we asked popular AI hosting sites Hugging Face and Civita why they hosted Stable Diffusion 1.5 and derived models.
Specifically, Jeff Allen, a data scientist at the Integrity Institute, found that Stable Diffusion 1.5 from Hugging Face has been downloaded more than 6 million times in the past month, making it the most popular AI image generator on the platform.
When asked why Hugging Face continues to host this model, company spokeswoman Brigitte Toussinant did not answer directly, but said that the company does not tolerate CSAM on its platform and has incorporated various security tools to encourage the community to use a secure stable distribution model that identifies and suppresses inappropriate images.
Then, yesterday, we checked the hacking phase and found that the stable spread 1.5 is no longer available. Toussinant told us that Hugging Face did not remove it and suggested that we contact Runway – we did it again, but we still have not received a response.
The fact that this model can no longer be downloaded from Hugging Face is undoubtedly a success. Unfortunately, it is still available as Civic, as are hundreds of derivative models. When we contacted CVD, a spokesperson told us they were not aware of what training data Stable Diffusion 1.5 used and would only remove it if there was evidence of abuse.
Sites should be nervous about their liability. Pavel Durov, the CEO of messaging app Telegram, was arrested last week as part of an investigation into CSAM and other crimes.
What’s Being Done About AI-Generated CSAM
The persistent noise of misleading reports and news about AI-generated CSAM and NCAI has not abated. As some companies try to improve the safety of their products with the help of technology alliances, what progress have we seen on the broader issue?
In April, Thorn and All Tech Is Human announced an initiative to bring together leading tech companies, creative AI developers, model hosting platforms and others to define and ensure safety by design, with a focus on preventing child sexual exploitation in the product development process. Ten companies (including Amazon, Civitai, Google, Meta, Microsoft, OpenAI and StabilityAI) have committed to these principles, and some have co-authored a related article with detailed recommended mitigations. The policies call on companies to develop, deploy and maintain AI models that proactively address child safety risks; develop systems to ensure that any misused material produced can be reliably detected; and limit the distribution of the underlying models and services used to exploit this abuse.
These kinds of voluntary commitments are a start. Rebecca Portnoff, Thorn’s head of data science, says the initiative demands accountability by having companies report on their progress on mitigation measures. It collaborates with standards-setting organizations like the IEEE and NIST to coordinate its efforts with new and existing standards, which opens the door for third-party audits that “go beyond the honor system,” Portnoff says. Portnoff also notes that Thorn works with policymakers to craft laws that are technically feasible and impactful. Indeed, many experts say it’s time to move beyond volunteer duties.
We believe there’s currently a reckless race to the bottom in the AI industry. Companies fight so furiously to stay technologically ahead that many of them overlook the ethical and even legal implications of their products. While some governments—including the European Union—are making progress in regulating AI, they have not gone far enough. For example, if laws make it illegal to provide AI systems capable of creating CSAM, tech companies might take notice.
The reality is that while some companies comply with voluntary obligations, many do not. Of those that do, many move too slowly, either because they are not ready or because they are struggling to maintain their competitive advantage. Meanwhile, bad actors can dominate those services and wreak havoc. That decision is unacceptable.
What Tech Companies Should Do About AI-Generated CSAM
- Detect and remove and report CSAM from their training data sets before training their AI models.
- Incorporate robust watermarks and content sourcing systems into your generative AI models so that generated images can be linked to the models that generated them, as required under a California bill that would create digital content sourcing standards for companies doing business in the state. The bill will be signed by Governor Gavin Newsom next month.
- Remove any generative AI models trained in CSAM or known to be capable of generating CSAM from your platform. Refuse to rehost these models until they have been fully rebuilt with CSAM removed.
- Find well-modified models intentionally laden with CSAM and permanently remove them from your platform.
- Remove “naked” apps from app stores, block search results for these tools and services, and work with payment providers to stop payments to their creators.
There is no reason why generative AI needs to stop and help perpetuate the terrible abuses that are happening to children. But we will need all the tools—voluntary commitments, regulation, and public pressure—to change course and stop the race to the bottom.
