Essay argues chat-based LLMs replicate the psychology of a psychic's cold-reading con

An essay arguing that chat-based large language models pull the same psychological levers as a psychic's con has drawn wide attention after reaching the front page of Hacker News. The piece, published on the Software Crisis letters page under the title "The LLMentalist Effect," claims that the impression of intelligence these systems create lives in the user's mind, not in the model.

According to the essay, its author has spent the past year or so researching how language and diffusion models are used in software businesses. What puzzled them during that work was how many people are convinced that chat-based LLMs are intelligent. In the author's account, an LLM is a mathematical model of language tokens: you give it text and it returns a mathematically plausible response. Nothing in that setup mirrors how people or animals reason, and the author points out that AI researchers and vendors have themselves repeatedly stressed that these models do not think.

Two possible explanations

The essay frames the conviction users report as having only two possible explanations. Either the tech industry has accidentally invented the early stages of a completely new kind of mind, built on unknown principles with no parallel in the biological world, or the intelligence is an illusion sitting in the mind of the user. The author, who wrote a book on generative AI risk titled "The Intelligence Illusion," places themselves firmly in the second camp, and says that working through the idea has left them believing there is even less intelligence and reasoning in these models than they previously assumed.

The psychic's playbook

The central claim is that the illusion runs on the same mechanism as cold reading. The author credits a blog post by Terence Eden on how common Forer statements are in chatbot replies with triggering the connection. Both the psychic and the chatbot rely on validation statements: phrasings built on the Forer effect that feel extremely specific to an individual while being statistically generic. The psychic uses them to appear to read minds; the chatbot uses them to appear to be an intelligence engaging specifically with you and your work, an impression the essay describes as a statistical trick.

The essay then walks through how a psychic's performance typically works. The audience selects itself, since attendees are already disposed to believe and sceptics are a small, easily managed minority. The scene is set: popular media have taught audiences the rules of readings, such as that they begin murky and grow clearer, that errors are expected because the "spirits" are vague, and that non-believers can weaken the connection. Meanwhile, staff research attendees through social media or conversation and note their demographics. The psychic then narrows the room with a statement that sounds specific but is statistically likely for that demographic, watches for a reaction, and follows up with a loop of confident, demographically probable guesses. The mark leaves convinced the psychic was the real thing.

The author argues that the tone of enthusiasm around AI echoes that of such a mark, quoting reactions like "This is real. It's a bit worrying, but it's real." as carrying the same blend of awe, disbelief and dread heard from victims of mentalist performers.

What it means for business

The essay's conclusion for the industry is blunt: many proposed use cases now look like "borderline fraudulent pseudoscience" to the author. The implication for businesses is that a product whose perceived value rests on users feeling personally understood is leaning on the same subjective validation loop that makes a cold reading convincing. Judged that way, demos and evaluations that measure how capable a model feels may be measuring the audience rather than the artifact, and deployments would need to be tested against concrete task outcomes instead.

Why it matters

The essay offers a concrete mechanism for a question the industry keeps deferring: why users report intelligence from systems whose own builders say no thinking occurs. If the author is right, the effect can be named, tested and guarded against, much as exposing cold reading defuses it. That makes the argument directly relevant to procurement, product design and evaluation practices around chat-based AI. The claim is contested; the essay itself acknowledges the rival view that the industry has stumbled onto a genuinely new kind of mind, and rejects it. But its spread on Hacker News suggests the comparison has resonated with a technical audience, and it shifts the burden onto vendors to show that perceived understanding corresponds to actual capability.