Value function: Definition and Examples

A state-value function $V^{π} (s)$ is the expected discounted return an agent collects if it starts in state $s$ and follows policy $π$ from then on. It answers "how good is it to be here?" for a particular way of behaving.

Definition

V^{π} (s) = E_{π} [G_{t} ∣ S_{t} = s] = E_{π} [k = 0 \sum \infty γ^{k} R_{t + k + 1} S_{t} = s] .

The expectation covers both the randomness in the policy's choices and in the environment's transitions. The optimal value function is $V^{*} (s) = max_{π} V^{π} (s)$ . Because the return is recursive, $V^{π}$ satisfies the Bellman expectation equation, $V^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}} P (s^{'} ∣ s, a) [R (s, a) + γ V^{π} (s^{'})]$ , which is what makes it learnable.

Worked example

From state $s$ , a policy goes left or right with probability 0.5 each. Left earns reward 1 and ends the episode. Right earns 0 and moves to state $s^{'}$ , where $V^{π} (s^{'}) = 8$ . With $γ = 0.9$ ,

V^{π} (s) = 0.5 \times 1 + 0.5 \times (0 + 0.9 \times 8) = 0.5 + 3.6 = 4.1.

The number is specific to this policy. A policy that always goes right would have value 7.2 in the same state.

V vs Q

$V$ scores states; the Q-function scores state-action pairs. Knowing $V$ alone does not tell you which action to take unless you also have a model of the environment, which is why model-free control methods usually learn $Q$ , and why actor-critic methods use $V$ as a baseline rather than as a decision rule.

How it is estimated

Monte Carlo methods average observed returns from each state. Temporal-difference learning updates $V (s)$ toward a one-step bootstrapped target. With function approximation, a neural network $V_{ϕ} (s)$ is regressed on either target.

Why it matters

Value functions turn a long-horizon problem into local, one-step comparisons, which is the core trick of reinforcement learning. They are also the critic in actor-critic methods and the source of the advantage used to reduce policy gradient variance. In RL for language models, a learned value head plays this role in PPO, while methods such as GRPO drop it and estimate baselines from groups of samples instead.

Sources

Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net