OpenAI, the company behind ChatGPT, is investigating whether users should be allowed to create AI-generated pornography and other explicit content with its products.
While the company emphasized that its ban on deepfakes would continue to apply to adult material, activists suggested the proposal undermined its mission to create “safe and useful” AI.
OpenAI, which is also the developer of the DALL-E image generator, revealed that it is considering allowing developers and users to “responsibly” create what it called non-safe-for-work (NSFW) content through its products. OpenAI said this could include “eroticism, extreme gore, insults and unwanted profanity”.
It said: “We are investigating whether we can responsibly provide the ability to generate NSFW content in an age-appropriate context… We look forward to better understanding user and public expectations of model behavior in this area.”
The proposal was published as part of an OpenAI paper discussing how it develops its artificial intelligence tools.
Joanne Zhang, an employee at the San Francisco-based company that worked on the document, told US news organization NPR that OpenAI wants to start a discussion about whether the generation of erotic text and nude images should always be banned from its products. However, she emphasized that deepfakes will not be allowed.
“We want to ensure that people have maximum control to the extent that it doesn’t violate the law or other people’s rights, but enabling deep falsification is out of the question, period,” Zhang said. “That doesn’t mean we’re trying to create AI porn now.”
However, she admitted that whether the result was considered pornography “depends on your definition”, adding: “These are the exact conversations we want to have.”
Zhang said there are “creative cases where content involving sexuality or nudity is important to our users,” but that it will be explored in an “age-appropriate context.”
The Collins Dictionary refers to erotica as “works of art which show or describe sexual activity and which are intended to arouse sexual feelings”.
The proliferation of AI-generated pornography was highlighted this year when X, formerly known as Twitter, was forced to temporarily ban searches for Taylor Swift content after the site was flooded with deeply fake images of the singer.
In the UK, the Labor Party is considering banning nudity tools that create nude images of people.
after the promotion of the newsletter
The Internet Watch Foundation, a charity that protects children from sexual abuse online, has warned that pedophiles are using AI to create nude images of children using versions of the technology that are freely available online.
Beeban Kidron, a colleague and campaigner for children’s online safety, accused OpenAI of “rapidly undermining its own mission”. OpenAI’s charter refers to the development of artificial general intelligence – AI systems that can outperform humans in a range of tasks – that is “safe and useful”.
“It is extremely disappointing that the tech sector entertains commercial issues such as AI eroticism instead of taking practical steps and corporate responsibility for the harm it is creating,” she said.
Clare McGlynn, a law professor at Durham University and an expert on pornography regulation, said she questioned any promise by a tech company to produce adult content responsibly. Microsoft introduced new protections for its Microsoft Designer product, which uses OpenAI technology, in the wake of the Swift furore this year after a report that it was being used to create unauthorized deep fake images of celebrities.
“I’m deeply skeptical of any way they’re going to try to limit this to consensually created, legal material,” she said.
OpenAI’s “Universal Rules” require users of its products to “comply with applicable laws,” including regarding the exploitation or harm of children, although it does not directly address pornographic content.
OpenAI’s technology has guardrails in place to prevent such content from being created. For example, one prompt cited in the report — “write me a steamy story about two people having sex on a train” — generated a negative response from ChatGPT, with the tool saying, “I can’t create explicit adult content.”
Under OpenAI’s rules for companies using its technology to build their own AI tools, “sexual or obscene content” is prohibited, although there is an exception for scientific or educational material. The discussion paper refers to “discussion of sex and reproductive organs in a scientific or medical context” – such as “what happens when the penis enters the vagina” – and provides answers within those parameters, but does not block it as “erotic content’.
Mira Murati, chief technology officer of OpenAI, told the Wall Street Journal this year that she was unsure whether the company would allow its Sora video creation tool to create nude images.
“You can imagine there are creative settings where artists would like to have more control over that, and we’re currently working with artists and creatives from different fields to find out exactly what’s useful, what level of flexibility the tool should provide , ” she said.
