Entropy bonus: Definition and Examples

An entropy bonus is a term added to a reinforcement learning objective that rewards the policy for keeping its action distribution spread out. It discourages the policy from becoming deterministic too early, which keeps exploration going and makes optimisation more stable.

Definition

The entropy of a policy in state $s$ is $H (π (\cdot ∣ s)) = - \sum_{a} π (a ∣ s) lo g π (a ∣ s)$ . With a bonus coefficient $β > 0$ , the objective becomes

J_{β} (θ) = E_{π_{θ}} [t \sum γ^{t} (r_{t + 1} + β H (π_{θ} (\cdot ∣ s_{t})))] .

In many implementations, such as A3C and PPO, the bonus is simply an extra loss term $- β H$ averaged over visited states, with $β$ around 0.01. Maximum-entropy RL, as in soft actor-critic, puts the entropy inside the return itself and tunes the coefficient (called the temperature) automatically to hit a target entropy.

Worked example

With two actions, a uniform policy has entropy $ln 2 \approx 0.693$ nats; a policy with probabilities 0.9 and 0.1 has about 0.325 nats. With $β = 0.1$ , collapsing from uniform to 0.9/0.1 costs $0.1 \times (0.693 - 0.325) \approx 0.037$ of objective per step. The policy will make that move only if it gains more than 0.037 in expected reward per step. Small gains from a lucky early estimate are not enough, so the policy keeps sampling the other action until the evidence is stronger.

What it does

Exploration: keeps probability on actions whose value is still uncertain. It is undirected, though: it spreads probability evenly, not toward the most informative actions.
Optimisation: smooths the objective and prevents the softmax from saturating, where gradients vanish.
Hedging against model error: maximum-entropy policies hedge across near-optimal actions instead of betting everything on one.

Relation to the KL penalty

An entropy bonus is a KL penalty toward the uniform distribution, up to a constant. The KL penalty used in RLHF instead pulls toward a reference language model, because for text the uniform distribution over tokens is meaningless. For language models an explicit entropy bonus is uncommon, but entropy is still monitored: a rapid collapse in token entropy during RL often precedes a loss of output diversity.

Why it matters

Premature convergence is one of the most common failure modes of policy gradient training. The entropy bonus is the cheapest defence, and watching the policy's entropy over training is one of the most informative diagnostics you can log.

Sources

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning, 2016. arxiv.org
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, 2018. arxiv.org