On-policy distillation trails GRPO on accuracy per training FLOP in arXiv comparison

What the study compared

According to a dev.to write-up summarising arXiv research, a team of researchers ran a direct comparison between two widely used post-training techniques for language models: on-policy distillation (OPD) and GRPO, a reinforcement learning algorithm common in recent reasoning-model training.

In the vanilla reverse-KL form of on-policy distillation, the student model generates its own samples while a larger teacher model supplies token-level probability targets, giving the student a learning signal at every position. GRPO instead trains against a reward given to whole completions, scoring each sample relative to others produced for the same prompt.

The study's design was deliberately broad. Researchers varied task difficulty, the size of the student model, the size of the teacher, the task domain, and a set of optimisation choices. The central metric was not peak performance but efficiency: held-out accuracy measured against the training FLOPs consumed, so the two methods are compared on how much quality each unit of compute buys.

The result

With an external teacher that had already been post-trained, on-policy distillation generally came out behind GRPO at matched settings, the dev.to summary reports. Given the same training budget, the RL method tended to reach higher held-out accuracy than distillation.

The write-up's analysis points to the cause: teacher inference. The appeal of OPD is density, meaning feedback at every token instead of one reward per completion. Producing that feedback, however, requires running the teacher across every batch of student-generated text, and those forward passes count toward the training bill. In the configurations tested, the denser gradient signal failed to pay back the compute spent generating it.

Where the finding stops

The dev.to summary is explicit about scope. The result covers vanilla reverse-KL OPD, the specific teachers, tasks and optimisation choices that were examined. Whether alternative distillation objectives, differently trained teachers, or domains that lack a dependable reward signal would change the picture remains an open question.

The experiments also relied on an external post-trained teacher. Setups where a teacher is cheaper to run, or where its cost is amortised across other uses, fall outside what the reported result establishes.

Why it matters

Distillation is often treated as the economical route to stronger small models: transfer what a large model knows instead of paying for a full reinforcement learning pipeline. As the dev.to summary frames it, the finding questions that assumption. On the accuracy-per-FLOP axis, dense teacher supervision was not the bargain it is usually assumed to be, and that should influence how labs allocate training compute for smaller models.

For engineering teams, the practical takeaway is straightforward. Budget the teacher's inference cost as part of distillation's true price, and evaluate post-training options on accuracy per training FLOP rather than on final benchmark scores alone. GRPO-style RL may be the better default where a usable reward signal exists; where none does, a teacher may still be the only supervision available, whatever the compute cost.