Credit assignment

Credit assignment is the problem of deciding which of an agent's earlier decisions deserve credit or blame for a reward that arrives later. Marvin Minsky named it in 1961, and it remains the central difficulty of learning from delayed feedback.

Two kinds of credit assignment

• Temporal: which actions in a sequence caused the outcome? A chess game is won or lost on move 60, but the decisive mistake may have been on move 12.
• Structural: which parts of the system, such as which weights in a network or which agent in a team, were responsible? Backpropagation is the standard solution inside a network.

How RL algorithms assign credit

Monte Carlo methods credit every action with the full return that followed it. Temporal-difference methods propagate credit backwards one step at a time through the TD error. Eligibility traces blend the two: when a TD error ${\delta }_t$ occurs, each earlier step $k$ receives a share proportional to its trace,

where $\gamma$ is the discount factor and $\lambda \in [0,1]$ sets how far back credit reaches. Learned advantages and GAE apply the same idea inside policy gradient methods.

Worked example

With $\gamma \lambda =0.9$, a step taken 10 steps before a surprising reward receives $0.9^{10}\approx 0.35$ of the credit given to the most recent step, and one taken 50 steps earlier receives about 0.005. If the true cause was 50 steps back, the learner will mostly reinforce whatever happened just before the reward. That is the failure mode credit assignment research tries to fix.

Credit assignment in language model training

When a 500-token response gets one scalar reward, REINFORCE-style methods give every token the same advantage: the token that introduced a reasoning error is pushed the same way as the token that fixed it. Learned value functions try to assign per-token credit, and process reward models provide step-level feedback directly. Both help, but both introduce new learned components that can be wrong or exploited.

Why it matters

Poor credit assignment makes learning slow, because signal is spread thinly, and can make it wrong, because irrelevant behaviour that happens to precede reward gets reinforced. For safety, this is one route by which superstitious or unintended behaviours become entrenched: they were present when reward arrived, so they were credited.

Related terms

• Sparse reward
• TD error
• Advantage function
• Process reward model (PRM)
• Temporal-difference learning
• Reward shaping

Sources

1. Marvin Minsky. Steps toward Artificial Intelligence, 1961. doi.org
2. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net