'C'était ça ou mourir': Is Publishing Phenomenon A French AI Literary Scandal?

When it comes to the success of the novel C'était ça ou mourir (It was that or dying), you can pick your cliché to describe it. Runaway bestseller. Taking critics by storm. Etc.

The author is Thélyson Orélien, who was born in Haiti but grew up in Quebec. It tells the story of a man who escapes the destruction of his neighborhood in Port-au-Prince and the subsequent odyssey that takes him through a series of harrowing encounters in several countries as he seeks refuge.

The book has already been translated into 20 languages. In France, it has won prestigious literary prizes such as the Prix Fnac, the Prix Méduse, and the Prix Première Plume. It has also been considered a strong favorite for even bigger accolades, such as the Prix Goncourt. Critics have been falling over each other to heap praise on it.

But yesterday, things took a potentially darker turn when the X account Balance ton Claude claimed that the AI writing detection tool Pangram had determined that the book was almost 100% written by AI:

For those not familiar, Pangram is an AI text detection tool that uses...AI to detect AI. And yes, everyone points out the irony, so don't bother. It has gotten a lot of attention recently because Substack integrated it directly into its platform to help readers weed out "slop."

In a statement on Wednesday from his publisher, the author strenuously denied the AI accusation:

“I started writing [the novel] in the fall of 2017 and I finished the first draft in 2019."

He also defended the stylistic elements of his writing that may have triggered the AI flags: "I come from a Haitian and Caribbean literary tradition where poetry, images, repetitions and rhythm carry a living word...We can discuss my style, but making his processes evidence of artificial intelligence has nothing to do with literary criticism."

Of course, this begs the question: Is one just using slop to detect slop?

Online, many anti-AI literary sleuths think so and have begun pouring the classics of French literature into Pangram to prove they triggered the AI warnings. Others have countered that running previous writing from the author in Pangram does not trigger the warnings, and therefore the issue is not his writing style. Still others have noted that some online communities have suggested the book was AI-generated for several months.

The problem is that, like so many tech and AI tools, Pangram is a black box, but it also tries to act like it isn't. For instance, it publishes papers that disclose a fair bit of detail about how it trains its models.

To be clear, this is not a tool that just looks at the surface-level stuff that you hear about, like the notorious 'em-dash' and the rule of 3. Because now that these have been tagged as AI writing, everyone knows not to use them, and so people either edit them out or they have built AI "humanizer" tools because we are fundamentally a sick society.

And yet, when you use Pangram, it spits back an overall score and summary but doesn't explain in detail why it determined something was written by AI, or partially written by AI.

According to Pangram, Pangram is pretty damn good: "On our benchmarks, Pangram 4 correctly identifies 99.66% of AI-generated documents while flagging human writing as AI only 0.0041% of the time, or roughly one false positive for every 24,000 documents."

The Pangram Rabbit Hole

Full disclosure: I use tools like Claude and ChatGPT for some writing and research projects, though not for writing whole articles. Still, I wanted to better understand how robust Pangram really is. So, using its published datasets and documentation, I created a Pangram skill for Claude using the Pangram API.

To be clear, even with this tool, it can not fully reveal the exact reasons Pangram flagged something as AI. Still, it uses the disclosed information about the model to make an informed analysis of what likely triggered Pangram AI alarms.