Comedy and humor are infinitely nuanced and subjective, but Google DeepMind researchers found agreement among professional comedians: “AI is very bad at it.”
This was one of many comments collected during a survey conducted with twenty professional comedians and performers during workshops at the Edinburgh Festival Fringe last August 2023 and online. The findings show that large language models (LLMs) accessed by chatbots present significant challenges and raise ethical concerns about using AI to generate entertaining material.
The research involved a three-hour workshop where comedians participated in a comedy writing session with popular LLMs such as ChatGPT and Bard. It also assessed the quality of output through a human-computer interaction questionnaire based on the decade-old Creativity Support Index (CSI), which measures how well a tool supports creativity.
Participants also discussed the motivations, processes and ethical issues of using AI in comedy in a focus group.
The researchers asked comedians to use AI to write comedy routines and then had them rate the results and share their thoughts. The results were… not good.
One participant described the AI-generated material as “the most tasteless, boring thing – I stopped reading it. It was so bad.” Another called the result “vomit that I know I’ll have to repeat and improve.”
“And I don’t want to live in a world where it gets better,” said another.
The study found that LLMs were able to generate outlines and snippets of longer routines, but they clearly lacked the human elements that make something funny. When asked to generate the structure of the draft, the models “spit out a scene that provided a lot of structure,” but when it came down to the details, “LLMs failed as a tool to support creativity.”
Among the reasons, the authors note, is the “global mapping of LLMs’ cultural values,” as the tools used in the study generate material based on all accumulated materials spanning every possible discipline. It also introduced a form of bias that the comedians pointed out.
“Participants noted that existing moderation strategies used in safety filtering and instructional LLMs reinforced hegemonic viewpoints by erasing minority groups and their perspectives and qualified this as a form of censorship,” the study said.
Popular LLMs are limited, the researchers said, citing the so-called “HHH criteria” calling for a fair, harmless and useful output — encapsulating what “most of what users want from an aligned AI.”
The material was described by one participant as “cruise ship comedy stuff from the 1950s, but a little less racist.”
“The more attractive something is, the less great it can be,” said another participant. “If you make a one-size-fits-all thing, it’s probably going to be nobody’s favorite thing.”
The researchers emphasized the importance of considering the subtle difference between harmful speech and offensive language used in resistance and satire. Meanwhile, comedians have also complained that AI has failed because it doesn’t understand nuances like sarcasm, black humor or irony.
“A lot of my stuff can have dark parts in it, and then he wouldn’t write me any dark stuff because he kind of thought I was going to kill myself,” reports a participant. “So he just stopped giving me anything.”
The fact that chatbots are based on written material didn’t help, the study found.
“Given that current widely available LLMs are accessed primarily through a text-based chat interface, they believe that the utility of these tools is limited to only a subset of the domains necessary to produce a complete comedy product,” the researchers note.
“Any script can be a good script, but a great actor could probably make it very enjoyable,” said a participant.
The study revealed that the limitations of AI in comedy writing extend beyond simple content generation. Comedians emphasized that perspective and point of view are uniquely human traits, with one comedian noting that people “add a lot more nuance, emotion and subtlety” because of their lived experience and connection to the material.
Many describe the centrality of personal experience to good comedy, enabling them to use memories, acquaintances and beliefs to construct authentic and engaging narratives. In addition, the comedians emphasized the importance of understanding the cultural context and audience.
“The kind of comedy I could do in India would be very different to the kind of comedy I could do in the UK because my social context would change,” said one participant.
Thomas Winters, one of the researchers cited in the study, explains why this is a difficult thing for AI to address.
“The premise of reframing humor reveals its difficulty for a machine to digest,” he said. “This essential reliance on insight into human thought—memory recall, linguistic abilities for semantic integration, and world knowledge inference—often leads researchers to conclude that humor is a complete AI problem.”
Speaking about the threat AI poses to human jobs, OpenAI’s CTO Mira Muratti recently said that “some creative jobs might disappear, but maybe they shouldn’t have existed in the first place.” Given the current capabilities of technology, however, it seems that comedians can breathe a sigh of relief.
Edited by Ryan Ozawa.
