GRPO (Group Relative Policy Optimization)

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for language models that drops PPO's learned value network. For each prompt it samples a group of responses and uses each response's reward relative to the group mean, scaled by the group standard deviation, as its advantage.

The group-relative advantage

For a prompt $x$, sample $G$ responses $y_1,\dots ,y_G$ with rewards $r_1,\dots ,r_G$. Each response gets

and, with outcome rewards, every token in $y_i$ shares that advantage. The group mean plays the role of a baseline, so no critic needs to be trained.

The objective

GRPO then maximises a clipped surrogate averaged over the group, with a KL penalty added directly to the loss instead of to the reward:

where ${\rho }_{i,t}={\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})/{\pi }_{{\theta }_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t})$ is the per-token importance ratio, $ϵ$ is the clip range and $\beta$ the KL weight.

Worked example

A group of four answers to a math problem gets verifier rewards $(1,0,0,0)$. The mean is 0.25 and the population standard deviation is about 0.433, so the correct answer's advantage is $0.75/0.433\approx 1.73$ and each wrong answer's is $-0.25/0.433\approx -0.58$. If all four were correct, every advantage would be zero and the prompt would teach nothing. Implementations differ on sample versus population standard deviation (here 0.5 versus 0.433), which changes the scale.

Trade-offs

• Saves the memory and tuning of a value model, which matters at large scale.
• Needs several samples per prompt, and fails to learn from prompts that are always or never solved.
• The per-group standard deviation and per-length averaging introduce biases (for example towards length on hard prompts) that later variants remove.

Why it matters

Introduced by Shao et al. (2024), GRPO became a default for RLVR on reasoning tasks. For evaluation and safety, group-normalised advantages amplify any small reward difference, so a slightly exploitable verifier can be exploited quickly; watch for reward hacking on held-out checks.

Related terms

• PPO (Proximal Policy Optimization)
• RLVR (reinforcement learning with verifiable rewards)
• Advantage function
• Baseline (policy gradient)
• KL penalty
• Verifier

Sources

1. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al.. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arxiv.org