Q-function: Definition and Examples

The Q-function, or action-value function, $Q^{π} (s, a)$ is the expected discounted return from taking action $a$ in state $s$ and following policy $π$ afterwards. Where the state-value function scores situations, the Q-function scores choices.

Definition

Q^{π} (s, a) = E_{π} [G_{t} ∣ S_{t} = s, A_{t} = a] = R (s, a) + γ s^{'} \sum P (s^{'} ∣ s, a) V^{π} (s^{'}) .

The first action is fixed; only later actions follow $π$ . The two value functions are linked by $V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)$ : the value of a state is the policy-weighted average of its action values. The optimal Q-function $Q^{*}$ satisfies the Bellman optimality equation

Q^{*} (s, a) = R (s, a) + γ s^{'} \sum P (s^{'} ∣ s, a) a^{'} max Q^{*} (s^{'}, a^{'}) .

Why learn Q rather than V

If you know $Q^{*}$ , the optimal policy is simply $π^{*} (s) = ar g max_{a} Q^{*} (s, a)$ . No model of the transitions is needed to make the choice. With only $V$ , choosing an action requires looking one step ahead through $P$ , which model-free agents do not have. This is why Q-learning and its deep variants learn $Q$ directly.

Worked example

Take a state $s$ where going left earns 1 and ends the episode, and going right earns 0 and leads to a state worth 8, with $γ = 0.9$ . Then $Q (s, left) = 1$ and $Q (s, right) = 0.9 \times 8 = 7.2$ . A policy that picks each with probability 0.5 has $V (s) = 4.1$ . The greedy policy picks right, and the advantage of right is $7.2 - 4.1 = 3.1$ : how much better it is than the policy's average behaviour.

In practice

Tabular Q-functions store one number per state-action pair. Deep Q-networks output a vector of Q-values for all discrete actions in one forward pass. For continuous actions, actor-critic methods learn $Q_{ϕ} (s, a)$ as a critic and train a separate actor to maximise it. Learned Q-values tend to be overestimated, because taking a max over noisy estimates is biased upward; double Q-learning addresses this by using one estimator to choose the action and another to evaluate it.

Why it matters

The Q-function is the bridge between evaluation and control: it tells you not just how well a policy does, but which single change would improve it. That one-step improvement, $π^{'} (s) = ar g max_{a} Q^{π} (s, a)$ , is the policy improvement theorem behind policy iteration, and the same quantity, minus a baseline, drives policy gradient updates.

Sources

Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net