AI text detection tools are really easy to fool

Daphne Ippolito, a Google senior researcher specializing in natural language generation, who didn’t work on the project, raises another concern.

“If automatic detection systems are to be employed in educational settings, understanding their false positive rates is critical, as falsely accusing a student of cheating can have dire consequences for their academic career,” he says. “The false negative rate is also important, because if too many AI-generated texts pass for human writing, the detection system is not helpful.”

Compilatio, which makes one of the tools the researchers tested, says it’s important to remember that its system only flags suspicious passages, which it classifies as potential plagiarism or potentially AI-generated content.

« It is up to the schools and teachers who mark the analyzed documents to validate or impute the knowledge actually acquired by the author of the document, for example by implementing additional means of inquiry: oral interrogations, additional questions in a controlled classroom environment, etc. » a spokesperson for Compilatio said.

“In this way, the Compilatio tools are part of a real didactic approach that encourages the learning of good research, writing and citation practices. The Compilatio software is a proofing aid, not a proofreader,” the spokesperson added. Turnitin and GPT Zero did not immediately respond to a request for comment.

We’ve known for a long time that tools intended to detect text written by AI don’t always work as they should. Earlier this year, OpenAI unveiled a tool designed to detect text produced by ChatGPT, admitting that it had marked just 26% of AI-written text as « probably written by AI. » OpenAI directed MIT Technology Review to a section on its own website for educators’ considerations, which warns that tools designed to detect AI-generated content are « far from foolproof. »

Still, those failures haven’t stopped companies from launching products that promise to do the job, says Tom Goldstein, an assistant professor at the University of Maryland who was not involved in the research.

“Many of them aren’t terribly accurate, but they aren’t all a complete mess either,” he adds, noting that Turnitin has managed to achieve some detection accuracy with a fairly low false-positive rate. And while studies that shed light on the shortcomings of so-called AI text detection systems are very important, it would have been helpful to expand the study’s remit to AI tools beyond ChatGPT, says Sasha Luccioni, a researcher at AI startup Hugging Face.

For Kovanović, the whole idea of ​​trying to locate text written by the AI ​​is flawed.

« Don’t try to detect AI, let the use of AI not be the problem, » he says.