Independent replication tests Direct Nash Optimization on Llama 3.1 and Mistral models

What happened

An independent research effort has re-implemented DNO-Prct, a practical variant of Direct Nash Optimization (DNO), and applied it to two widely used open-weight instruction-tuned models: Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1. According to a summary published on dev.to, which cites arXiv as the underlying publication, the goal was to check whether the method's core idea — steering a model toward an equilibrium of preferences rather than maximizing a fixed reward score — holds up outside the original authors' code, data and annotators.

How the method works

As described in the write-up, DNO-Prct runs as an iterative loop with four stages. First, the policy generates responses on-policy, meaning samples come from the model at its current training state rather than from a static dataset. Second, an LLM acts as a judge and rates each generated response in comparison with a teacher response to the same prompt. Third, the ratings are used to form preference pairs, with stronger outputs marked as chosen and weaker ones as rejected. Fourth, the model receives a DPO-style update — the contrastive likelihood objective popularized by Direct Preference Optimization — and the loop begins again with fresh samples from the newly updated policy.

Two of these design choices carry most of the weight. On-policy sampling keeps the training signal matched to the distribution the model actually produces, avoiding the drift that comes from training on stale or off-distribution data. And using an LLM judge in place of a separately trained reward model removes a component that classical RLHF pipelines must build, calibrate and defend against gaming.

From reward models to equilibria

In the standard reinforcement learning from human feedback setup, preference data is first distilled into a reward model, and the policy is then optimized to maximize that scalar signal. DNO, introduced in a 2024 arXiv paper, proposed a different target: optimize the policy directly toward the equilibrium of a preference game, with no explicit reward model in the loop. The dev.to analysis suggests the new replication is evidence for a broader shift in alignment practice, away from frozen reward functions and toward iterative, on-policy preference optimization.

What the replication adds

Independent replication is rare in machine learning research, and alignment methods in particular often depend on subtle implementation details — judge prompts, hyperparameters, data curation — that can quietly carry results from one lab to another. By rebuilding the pipeline from scratch and running it on two model families from different organizations, Meta's Llama and Mistral AI's Mistral, the study offers a practical signal that the approach is not tied to a single model's quirks or a single team's annotation process.

The dev.to summary does not report specific benchmark numbers, so the strength of the replication's outcome is best assessed from the underlying arXiv paper rather than the summary alone. Open questions also remain around the usual weak points of judge-driven training: bias in the LLM rater, sensitivity to the quality of the teacher responses, and the compute cost of repeated on-policy sampling.

Why it matters

Reward-model-based RLHF is expensive to build and vulnerable to over-optimization, where a model learns to satisfy the reward signal rather than genuine preferences. If equilibrium-based, judge-driven loops replicate reliably, they offer smaller teams a cheaper path to aligning open-weight models without training a reward model at all. Just as importantly, the study is a data point for replication culture in alignment research: a method validated only by its inventors is a hypothesis, and independent evidence on commodity hardware and public models is what turns it into a technique practitioners can reasonably adopt.