A sparse reward is a reward signal that is zero almost everywhere and nonzero only on rare events, such as reaching a goal, winning a game or passing a unit test. The agent receives no feedback about partial progress, so learning depends on stumbling onto success often enough to learn from it.
Why sparse rewards are hard
Learning needs at least some trajectories with nonzero reward. If random behaviour almost never produces one, every policy gradient estimate is zero and every value estimate stays flat. Even after the first success, the agent faces a hard credit assignment problem: which of the many earlier actions caused it?
Worked example
Consider a chain of 10 states where the agent must choose "right" 10 times in a row to reach the only reward, and any "left" ends the episode. A uniformly random policy succeeds with probability , so it needs about 1,000 episodes on average before seeing its first nonzero reward. With a chain of 30 states, the expected wait exceeds a billion episodes. The difficulty grows exponentially with the number of correct decisions required, which is why dense intermediate signals or better exploration are needed.
Common remedies
- Reward shaping: add dense progress signals, ideally potential-based so the optimal policy is unchanged.
- Better exploration: curiosity or novelty bonuses, count-based bonuses and optimism push the agent toward unvisited states (see exploration versus exploitation).
- Hindsight relabelling: in goal-conditioned tasks, treat the state actually reached as if it had been the goal, turning failures into useful training data.
- Demonstrations and curricula: start from imitation or from easier versions of the task.
- Strong priors: a pretrained policy that already succeeds sometimes.
Sparse rewards in RL for language models
Training language models with verifiable rewards is a sparse-reward problem: a long response gets a single 0 or 1 from a checker. It works because the pretrained model already solves a useful fraction of problems, so successes are not rare. Prompts the model never solves give no gradient at all, and prompts it always solves give none either once a group baseline is subtracted, so curating problems of intermediate difficulty matters. A process reward model is one way to densify the signal.
Why it matters
Sparse rewards are often the honest specification of a task: success or failure. Every technique that makes them easier to learn from, especially hand-built dense rewards, adds a new place where the agent can optimise something other than what you meant.
Related terms
- Reward shaping
- Credit assignment
- Exploration-exploitation trade-off
- RLVR (reinforcement learning with verifiable rewards)
- Process reward model (PRM)
Sources
- Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba. Hindsight Experience Replay, 2017. arxiv.org
- Andrew Y. Ng, Daishi Harada, Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping, 1999. people.eecs.berkeley.edu