Study finds readers rate AI-generated stories higher than human-written fiction, struggle to distinguish between them

A Villanova University study published in Judgment and Decision Making reveals that readers struggle to distinguish between human-written fiction and ChatGPT-generated stories, actually rating the AI versions 6% higher in quality and 8% higher in engagement. Led by Dr. Deena Weisberg, the research tested over 2,500 adults aged 18 to 81. It found a distinct bias toward human authorship: stories labeled as human-written scored highest, even when they were actually generated by AI. While literary expertise did not help readers identify AI text, higher AI literacy did.

AI story quality ratings

• ▪In the Villanova University study, AI-generated stories were rated 6% higher in quality and 8% higher in engagement than human-written stories.
• ▪A study by Villanova University researchers found that readers rated short stories generated by ChatGPT higher in quality and engagement than stories written by human authors.
• ▪Senior study author Dr. Deena Weisberg suggested that readers preferred AI-generated stories because they tend to be clearer, more direct, and easier to process than complex human writing.

Reader identification failure

• ▪Participants in the Villanova University study failed to reliably distinguish between human-written and AI-generated stories, guessing correctly at rates near or below random chance.
• ▪In the study's third experiment, participants correctly identified the author of the stories 51.9% of the time, which is statistically no different from chance.
• ▪The study found that self-reported expertise in literature did not improve a participant's ability to identify AI-generated stories, whereas higher AI literacy did.
• ▪In the study's second experiment, participants correctly identified the author of the stories only 39.9% of the time, which is statistically worse than a random coin flip.

Human authorship label bias

• ▪The highest overall ratings in the study were given to AI-generated stories that participants were falsely told had been written by a human author.
• ▪Dr. Deena Weisberg stated that the preference for stories labeled as human-written reveals a bias based on the assumption that creative writing requires uniquely human qualities.
• ▪The Villanova University study revealed a bias toward human authorship, as stories labeled as human-written received higher ratings regardless of whether a human or AI actually wrote them.

Study methodology details

• ▪The second and third experiments of the study evaluated a combined total of 905 adult participants who were asked to identify the author of two stories.
• ▪In the first experiment of the study, 1,682 readers each read a single story and rated its quality and how absorbed they felt while reading it.
• ▪The Villanova University study, published in Judgment and Decision Making, was built around three human-written short stories and three corresponding ChatGPT-generated versions.

Participant age demographics

• ▪The Villanova University study evaluated adult participants spanning a wide age demographic, ranging from 18 to 81 years old.
• ▪The age range of 18 to 81 years was maintained across the study's multiple experiments to test the general public's ability to distinguish human and AI writing.

8 sources