Show HN: Training a model to identify AI web content from structure alone

summary

AI-generated commercial web content possesses distinct structural signatures that remain consistent even when the text is reworded. By analyzing how information is organized, ordered, and voiced rather than focusing on specific words, it is possible to identify AI authorship with high accuracy. This structural approach overcomes the brittleness of word-level detectors which are easily fooled by paraphrasing. The resulting method can not only detect AI content but also attribute it to specific frontier models.

key points

1. Structural signatures—such as the order of evidence and the way information is presented—serve as a more reliable indicator of AI authorship than word-choice patterns. These patterns are so ingrained in how LLMs organize content that they persist even after the AI rewrites its own output.
2. The SlopShape instrument uses 187 structural features to achieve a 98% macro-F1 score in distinguishing AI posts from human-written ones. This demonstrates that AI content typically follows a 'tidy' and self-announcing shape that differs significantly from the rare, varied configurations found in human writing.
3. Beyond simple detection, structural analysis allows for the attribution of text to specific AI models. The system successfully attributed content to the correct source model nearly 80% of the time, far exceeding the random chance rate.

who should read it

Researchers in natural language processing and digital forensics who need robust methods for identifying synthetic text. It is also useful for web publishers and SEO experts concerned with the proliferation of AI-generated 'slop' on the internet.

discussion skeptical