· via dev.to (home feed)
DeepSeek V4 keeps GRPO for specialist training, swaps mixed RL for on-policy distillation
DeepSeek's V4 report keeps GRPO for training domain specialists after supervised fine-tuning, but replaces the earlier mixed-RL stage with on-policy distillation to build the final unified model.

DeepSeek's V4 technical report, released in April 2026, still uses Group Relative Policy Optimization (GRPO) — but the pipeline around the algorithm has been redrawn. According to a dev.to write-up summarising the report, V4 trains domain specialists with supervised fine-tuning (SFT) and domain-specific GRPO, then folds their capabilities into a single unified model through on-policy distillation.
The report is direct about the scale of the change, stating that the mixed reinforcement-learning stage of the earlier pipeline was "entirely replaced by On-Policy Distillation." That does not mean GRPO has been retired. As the dev.to analysis stresses, the algorithm remains the reinforcement-learning method inside specialist training; what has moved is its place in the overall recipe, and what happens after it.
How GRPO works, and what stays the same
GRPO generates a group of candidate responses to the same prompt and assigns each one a reward. An individual response's advantage is computed relative to the other members of its group, and the policy is updated with a clipped objective plus regularisation against a reference policy. Compared with PPO, its main simplification is structural: no separate value critic is required, because the group's statistics serve as the baseline.
None of this changes in V4, per the write-up. The group-relative principle remains the engine of the specialist RL stage, and the report does not introduce a new GRPO variant.
The V4 recipe in three steps
The dev.to summary reduces the published pipeline to three stages:
- Fine-tune separate specialist models for individual domains.
- Apply GRPO to each specialist, using domain-specific prompts and reward signals.
- Distil the specialists' behaviour into a single unified model on-policy.
The architecture, in other words, is modular: RL is scoped to the domains where its reward signals are clearest, and consolidation is handled by distillation rather than another round of reinforcement learning.
What was different in V3.2
The previous release, DeepSeek-V3.2, took a more monolithic approach: a single mixed RL stage spanning reasoning, agent and human-alignment tasks, supported by rule-based rewards and generative reward models. V4 splits that stage apart. Specialists are adapted per domain with SFT and GRPO, and the final unified model is distilled from those specialists instead of being trained through the mixed stage.
The write-up draws a useful distinction here: not every change to a training pipeline is a change to the algorithm inside it. Describing V4 as GRPO placed in a staged, modular pipeline is closer to the report than claiming a new optimiser has arrived.
Why it matters
For anyone post-training large models, V4 is evidence that a leading lab has restructured its use of RL rather than abandoned it. Splitting training into per-domain specialists lets teams attach tailored prompts and reward functions to each capability, which is often easier to debug than one mixed stage juggling reasoning, agent behaviour and alignment simultaneously. Handing final consolidation to on-policy distillation also sidesteps the difficulty of designing a single reward signal that fairly arbitrates between very different domains.
The underlying report — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, on arXiv — also serves as a terminology checkpoint. GRPO headlines most discussions of DeepSeek's training, but V4's actual story is the surrounding pipeline: SFT, domain-specific GRPO and distillation staged together. Readers comparing training recipes should keep the two separate, since a change at the pipeline level says nothing about the optimiser underneath.
- #deepseek
- #grpo
- #reinforcement-learning
- #llm-training
- #distillation