Return: Definition and Examples

In reinforcement learning, the return $G_{t}$ is the sum of rewards an agent receives from time step $t$ onward, usually with later rewards discounted. It is the quantity an agent is trying to maximise in expectation, and the thing every value function predicts.

Definition

For an episode that ends at time $T$ , the discounted return is

G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots = k = 0 \sum T - t - 1 γ^{k} R_{t + k + 1},

where $R_{t + k + 1}$ is the reward received $k + 1$ steps after $t$ and $γ$ is the discount factor. In continuing tasks with no terminal state, $T = \infty$ and $γ < 1$ keeps the sum finite. The key identity is the recursion

G_{t} = R_{t + 1} + γ G_{t + 1},

which is the seed of the Bellman equation.

Reward vs return

A reward is a single scalar signal at one step. The return is the long-run total that follows. An agent that maximised reward would be greedy; an agent that maximises return will accept a bad step now for a better future. Confusing the two is a common source of bugs, especially when rewards are logged per step but evaluated per episode.

Worked example

An episode produces rewards $(0, 0, 1, 0, 5)$ with $γ = 0.9$ . Computing backwards with the recursion: $G_{4} = 5$ , $G_{3} = 0 + 0.9 \times 5 = 4.5$ , $G_{2} = 1 + 0.9 \times 4.5 = 5.05$ , $G_{1} = 0 + 0.9 \times 5.05 = 4.545$ , and $G_{0} = 4.0905$ . The backward pass costs one multiply-add per step, which is how training code computes "rewards-to-go" for every time step at once.

How returns are used

Monte Carlo methods use sampled returns directly as unbiased targets.
REINFORCE weights $\nabla lo g π (A_{t} ∣ S_{t})$ by $G_{t}$ , the reward-to-go, rather than the full-episode return, which lowers variance because rewards before $t$ cannot depend on $A_{t}$ .
TD methods replace the tail $G_{t + 1}$ with an estimate, and n-step returns interpolate between the two.

Why it matters

The return is the objective, so it defines what "good behaviour" means for an agent. In RL for language models, a response is often one episode with a single terminal reward and $γ = 1$ , so the return of every token equals the final score. That makes credit assignment across tokens hard, and it is one reason process reward models exist.

Sources

Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net