Discount factor: Definition and Examples

The discount factor, written $γ$ , is a number between 0 and 1 that multiplies each future reward by $γ^{k}$ when it arrives $k$ steps later. A small $γ$ makes the agent short-sighted; a $γ$ close to 1 makes it weigh distant consequences almost as much as immediate ones.

Definition

In a Markov decision process, the discounted return from time $t$ is

G_{t} = k = 0 \sum \infty γ^{k} R_{t + k + 1}, 0 \leq γ < 1,

where $R_{t + k + 1}$ is the reward received $k + 1$ steps after time $t$ . Because $γ^{k}$ shrinks geometrically, the sum is finite whenever rewards are bounded: if every reward is at most $R_{m a x}$ , then $∣ G_{t} ∣ \leq R_{m a x} / (1 - γ)$ . The same $γ$ appears in the Bellman equation, where it scales the value of the next state.

Effective horizon

A useful rule of thumb is that $γ$ corresponds to an effective planning horizon of roughly $1/ (1 - γ)$ steps. With $γ = 0.9$ the horizon is about 10 steps; with $γ = 0.99$ it is about 100. A reward 100 steps away is worth $0.9 9^{100} \approx 0.37$ of its face value at $γ = 0.99$ , but only $0. 9^{100} \approx 0.00003$ at $γ = 0.9$ .

Worked example

An agent can take a reward of 1 now, or walk five steps to a reward of 10. With $γ = 0.9$ the distant option is worth $0. 9^{5} \times 10 \approx 5.9$ , so the agent walks. With $γ = 0.6$ it is worth $0. 6^{5} \times 10 \approx 0.78$ , which is less than 1, so the agent grabs the immediate reward. Nothing about the environment changed; only the agent's patience did.

Three ways to read gamma

Preference: the designer genuinely prefers sooner reward, as in finance.
Termination probability: discounting by $γ$ is equivalent to an undiscounted problem in which the episode ends with probability $1 - γ$ at every step.
Variance control: in practice $γ$ is often a hyperparameter that trades bias for variance. Lower values give lower-variance return estimates and faster learning, at the cost of optimising a slightly different objective. GAE adds a second parameter, $λ$ , for the same purpose.

Why it matters

The discount factor is part of the objective, not just a training knob, so changing it changes which policy is optimal. Too low and the agent ignores delayed consequences, which matters for safety when harms show up late; too high and learning becomes slow and noisy under sparse rewards. In RL for language models, a whole response is often treated as one episode with $γ = 1$ , because the reward arrives only at the end and the horizon is short and fixed.

Sources

Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net