Policy

A policy is the decision rule of a reinforcement learning agent: it maps each state to an action, or to a probability distribution over actions. Learning in RL means improving the policy so that it collects more expected return.

Definition

A stochastic policy gives the probability of each action in each state, $\pi (a\mid s)=\mathrm{Pr}(A_t=a\mid S_t=s)$. A deterministic policy picks one action, $a=\mu (s)$. The objective is the expected return from the start-state distribution:

Here the expectation is over the trajectories produced by following $\pi$ in the environment and $\gamma$ is the discount factor. In every finite MDP there is at least one deterministic policy that is optimal in all states simultaneously.

How policies are represented

• Implicitly, through values: act greedily with respect to a Q-function, $\pi (s)=\arg {\max }_{a}Q(s,a)$, possibly with epsilon-greedy exploration.
• Explicitly, as a parameterised distribution ${\pi }_{\theta }$, such as a softmax over discrete actions or a Gaussian over continuous ones, trained by policy gradient methods.
• As a language model: an LLM is a policy over tokens, ${\pi }_{\theta }(y_t\mid x,y_{<t})$, which is why RLHF and related methods can treat it with standard RL machinery.

Worked example

A softmax policy has scores $h=(2,1,0)$ for three actions. The probabilities are $e^2,e^1,e^0$ normalised by their sum $\approx 11.11$, giving $(0.665,0.245,0.090)$. Raising the score of the third action by 1 more than doubles its probability, to about 0.21, while the first action still dominates. Stochastic policies like this explore naturally and are differentiable, which is what gradient methods need.

Why it matters

The policy is the artefact you deploy. Value functions, reward models and critics are training aids; the policy is what acts. Two distinctions recur throughout RL: the policy that collects data versus the one being learned (on-policy vs off-policy), and the policy that maximises the written reward versus the behaviour you intended. The second gap is where reward hacking lives.

Related terms

• Action
• Value function
• Q-function
• Policy gradient
• On-policy vs off-policy
• Epsilon-greedy

Sources

1. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net