Glasshouse v0.1 Launches an Open Benchmark for Long-Term Memory in AI Systems

What launched

A developer has released Glasshouse v0.1, an open benchmark for measuring long-term memory in AI systems. Announced in a post on dev.to, the project builds on earlier benchmark write-ups the author had published on the platform, and the code, data and instructions are hosted on GitHub at wontopos/glasshouse.

According to the author, the motivation came from recurring complaints in developer communities: vendor-published memory numbers rarely match what anyone else measures, and swapping the model used to grade the results moves the scores more than the actual gap between the systems being compared. Glasshouse is pitched as a shared yardstick to replace that murk.

What the benchmark contains

The v0.1 release packs 2,847 questions into a conversation that runs to 1.97 million tokens, covering 10 languages and including 50 photographs.

The conversation ships in four sizes, from 1,882 turns up to 103,572 turns, and the section holding the answers is identical in all of them. What changes is how much history accumulates around those answers. That structure is the point: where a system's score falls apart indicates whether it degrades as the history grows, rather than whether the questions got harder.

Scores are reported separately for each axis, with no single headline number. The author's reasoning is that a system can be strong on one axis and useless on another, and an aggregate figure would blur exactly the distinction that matters.

Grading more than recall

Several axes test behavior beyond retrieving stored facts, as described in the dev.to post:

• Stale facts: when a value changes during the conversation and the system cannot find the new one, answering "I don't know" scores better than confidently repeating the old value.
• Contradictions: when two stored facts disagree and nothing settles the conflict, reporting the disagreement is the correct answer; silently picking one is marked wrong.
• False-memory probes: when the answer was never stated at all, the benchmark checks whether the system says so.

The author also notes that some numbers have moved since an earlier post, and that the earlier post had said they would.

An empty submissions folder, on purpose

The project's submissions directory is currently empty, and the author has not submitted a result either. Individuals are asked to run the benchmark and open a pull request or an issue when something looks wrong. There is no participation threshold for individuals, deliberately, and an objection that names a specific error is answered in public.

The design was shaped by outside input during development. A file in the repository lists suggestions collected on Reddit while v0.1 was being built and what each one became: the stale-fact axis, the contradiction axis and the false-memory probes all began as comments. One suggestion that was not used is listed anyway, on the argument that a record showing only what was accepted cannot be checked. Companies can submit results or add themselves through a process documented in the repository.

Why it matters

Long-term memory is becoming a load-bearing component of AI agents and assistants, which increasingly work across extended histories. Comparisons in this area have been unreliable, with the dev.to post describing vendor figures that fail to replicate and grader models whose influence swamps the differences being measured. By holding the answer-bearing portion of the conversation fixed and varying only the surrounding history, Glasshouse isolates degradation under accumulating context, the exact failure mode long-running systems face.

The choice to score each axis separately, and to reward honest uncertainty over confident error, also nudges evaluation toward behavior rather than raw recall. And the empty leaderboard is part of the pitch: by inviting scrutiny first, publishing a public log of both adopted and rejected suggestions, and answering named errors in the open, the project is trying to build trust in the benchmark itself before anyone posts a score. Whether submissions follow will determine if it becomes a common reference point; either way, the open methodology lets anyone test against it.