Pyrite benchmark: expert chemists rank protein-ligand poses near chance

A benchmark built around pose pairs

Researchers have introduced Pyrite, a benchmark for ranking protein-ligand binding poses. According to a dev.to post describing the work, which cites a paper on arXiv, Pyrite consists of 1,500 pose pairs drawn from a diverse set of protein-ligand complexes.

The format is deliberately simple: for each item, two candidate poses for the same complex are presented, and the task is to decide which one is better. By reducing the question to a pairwise comparison, the benchmark isolates the ranking problem — judging pose quality — from the separate problem of generating poses in the first place.

What pose ranking involves

In structure-based drug discovery, a ligand's binding pose is the three-dimensional arrangement it adopts inside a protein's binding site. That geometry underlies how researchers reason about whether a candidate molecule is likely to bind a drug target.

The practical difficulty is that modern tools rarely produce a single answer. Docking software and newer cofolding models, which jointly predict a protein's structure together with a bound ligand, typically output multiple plausible candidate poses. Selecting among those candidates then falls to a human reviewer or an automated scoring step. Pyrite is designed to measure exactly how well that selection can be done.

Experts land near a coin flip

The headline finding reported in the dev.to post is that expert chemists performed near chance on the benchmark. These were not unaided guesses: the experts had access to 3D visualization tools and physics-based scoring functions, the standard toolkit for this kind of judgment. Even so, when shown two poses for the same complex, they could not reliably identify the better one at a rate meaningfully above random.

The implication is that the difficulty is intrinsic to the task rather than a gap in expertise or tooling. Fine-grained distinctions between candidate poses appear to resist the conventional workflow of visual inspection supplemented by physics-based scores.

PyriteRep and automated ranking agents

Alongside the benchmark, the work introduces PyriteRep, a structured molecular representation intended for automated ranking agents. Rather than depending on depictions or scores designed for human interpretation, an agent would consume this structured description of a complex and learn to rank poses directly.

The analysis in the post suggests a possible architecture for future drug discovery systems: combine cofolding models that generate candidate poses with a learned ranking component that filters them. That would move candidate selection beyond the human-interpretable chemical scores that currently dominate the field.

Why it matters

Correct pose ranking sits near the center of computational drug discovery. As cofolding models make it cheap to generate large numbers of candidate poses, the bottleneck shifts to choosing among them — and Pyrite suggests the current human-plus-scores approach offers little signal for that choice.

The near-chance expert baseline also recalibrates how AI results on this task should be read. Any automated ranking method that clears the chance level by a meaningful margin is outperforming specialists using standard tools, which makes the benchmark a useful measuring stick for progress in molecular AI.

There are caveats worth holding onto. The findings are summarized from an arXiv paper, and details such as how the pose pairs were constructed and how many experts participated determine how far the near-chance result generalizes. As with any new benchmark, adoption by other research groups will be the real test of whether it becomes a standard reference point.

Even so, the signal is a valuable one: it gives the field a concrete, quantified example of where expert judgment ends, and where learned ranking would have to begin.