AI-generated blog posts detectable from structure alone, even after rewording

A preprint posted to arXiv and surfaced on Hacker News's front page argues that AI-generated commercial web writing can be identified not from word choice but from structure — the order in which information appears, the evidence it leans on, and the voice it adopts. The paper, "SlopShape: Identifying AI-Generated Commercial Web Content" by Jochen Madler, reports that an analysis of 187 structural features picked out AI-written blog posts with 98.0 macro-F1 on companies excluded from training, and that the score barely moved, at 98.1, when every AI post was reworded by the model that produced it.

The brittleness problem

Word-level detectors, the paper's motivation, flag unedited machine text almost flawlessly but are documented to fail once a text is paraphrased. A word-level probability score also says little else: it neither describes what makes a text machine-like nor indicates which model wrote it. The author asks whether the tell sits one level deeper, in structural signatures that survive changes to surface wording.

From fiction to commercial content

According to the abstract, the study replicates StoryScope (Russell et al., 2026), which demonstrated such structural patterns in AI-generated fiction, and carries the approach over to commercial writing. The corpus pairs 2,250 human blog posts — written before ChatGPT existed, from 268 company domains — with 11,250 AI-generated counterparts produced by five frontier models, five per human post. A 214-feature instrument was applied by an LLM and checked against a human gold-annotation session, with agreement of 0.928 between human annotators and 0.946 between humans and the model. Detection relies on structural features alone, evaluated on held-out companies so the results cannot simply memorize particular domains.

What the analysis found

Three findings stand out. First, detection is strong and paraphrase-proof: 98.0 macro-F1 on unseen companies, essentially unchanged after rewording every AI post with its own model. Second, the signal is attributable — 79.3% of AI posts were assigned to the correct source model, against a 16.7% chance rate across five models. Third, the two authorship classes behave differently in kind, not just degree: AI posts converge on a tidy, formulaic organization that lays out its own structure explicitly, while human posts occupy unusual, rarely repeated combinations of structural traits. The paper reports that all effects match StoryScope's direction and exceed it in magnitude, and that the pipeline, instrument, prompts, code and aggregate artifacts are being released.

Caveats to keep in mind

This is a preprint — the second version is dated 17 September 2026 — and has not been peer-reviewed; the details here come from the abstract shared via Hacker News. The corpus design, using human writing that predates ChatGPT against current model output, guarantees human origin but may widen the gap relative to today's human-written web text, some of which is AI-assisted or edited. Evaluation on held-out companies within the same corpus does not settle how the method generalizes to other genres, lightly edited hybrid text, or newer models. And while feature annotation was LLM-driven, the human validation session suggests the measurements are reproducible.

Why it matters

Paraphrase resistance targets the main weakness of deployed AI-text detectors, which can be defeated by a light rewrite. Structural detection that survives rewording — and that can name the likely source model — gives search engines, moderators and archivists a sturdier signal than word statistics. It also reframes the problem for content producers: if the giveaway is how modern models organize an argument rather than which words they pick, blending in would require structural change, not just lexical variety. As machine-written commercial content spreads across the web, a measurable, attributable structural fingerprint becomes a practical tool for provenance and a useful constraint on how convincingly generic AI writing can imitate human work.