What is reinforcement learning? A guide from gridworlds to language models

Reinforcement learning is the branch of machine learning in which an agent learns to make a sequence of decisions by trial and error, guided only by a numerical reward. Nobody tells it the correct action; it acts, observes what happens and how much reward it got, and gradually shifts towards behaviour that collects more reward over time. The same idea trains game-playing programs, robot controllers and the post-training stage of large language models.

This guide builds the whole subject from one small example you can play with. You will see the vocabulary (states, actions, rewards, policies, values), the one equation that most algorithms are built on, the difference between planning with a known model and learning from experience, the main families of algorithms, and why reinforcement learning is harder in practice than its definition suggests. Every number below comes from a seeded simulation we ran, and each section links down to a full article on that topic.

A concrete example: a robot in a slippery grid#

Picture a robot on a 5 by 5 grid, starting in the bottom-left corner. The top-right cell holds a charger worth +1 reward. One cell near the charger is a pit worth −1. Two short walls block parts of the grid. The floor is slippery: when the robot tries to move in a direction, it goes that way 80% of the time and slides to one of the two perpendicular directions 10% of the time each. Reaching the charger or the pit ends the episode. Every other move earns nothing.

We call this the reference gridworld and use it across the reinforcement learning articles on this site. It is tiny (20 cells the robot can stand in, four actions each), yet it already has everything that makes the problem interesting. The reward arrives only at the end, so the robot must work out which of its many earlier moves deserved the credit. The shortest route passes next to the pit, so there is a real trade-off between speed and risk. And the slip means the same action does not always have the same result, so a good plan has to be a policy (what to do in every cell) rather than a fixed list of moves.

If you knew the slip probabilities and the layout, you could compute the best policy directly. That is planning. If you only get to try moves and see what happens, you have to learn the policy from experience. That is reinforcement learning in the strict sense, and the gap between the two is the subject of this guide.

The elements of reinforcement learning#

Every reinforcement learning problem has the same parts. The textbook treatment is Sutton and Barto [1]; here is the short version, mapped onto the grid.

• Agent and environment. The agent is the learner and decision maker (the robot's controller). The environment is everything else (the grid, the walls, the slippery floor).
• State $s$: what the agent knows about the situation now. In the grid, its cell.
• Action $a$: what it can do. Up, right, down or left.
• Reward $r$: a number the environment sends after each step. +1, −1 or 0 here. The reward defines the goal; everything else is up to the agent.
• Policy $\pi (a\mid s)$: the agent's behaviour, the probability of each action in each state. Learning means improving the policy.
• Return $G_t$: the total reward from time $t$ onwards, usually discounted.
• Value function $V^{\pi }(s)$: the expected return from state $s$ if you follow policy $\pi$. It is a prediction of the future, and most algorithms are built around estimating it.
• Model: the transition and reward probabilities. Some agents learn one and plan with it; many never do.

The return weights future rewards by a discount factor $\gamma$ between 0 and 1:

Here $r_{t+k+1}$ is the reward received $k+1$ steps after time $t$ and $\gamma$ controls how much the agent cares about the distant future. A reward $k$ steps away is worth ${\gamma }^k$ of a reward now. With $\gamma =0.9$, reaching the charger in 8 steps is worth $0.9^7\approx 0.48$ at the start, because the +1 arrives on the eighth step. The discount keeps infinite sums finite and makes the agent prefer sooner rewards over later ones.

The value of a state under a policy is the expected return, and the quantity most learning algorithms actually estimate is its action version, the Q-function:

$V^{\pi }(s)$ answers "how good is it to be here if I keep behaving like this?" and $Q^{\pi }(s,a)$ answers "how good is it to take action $a$ here and then behave like this?". If you know $Q$ for the best policy, acting well is trivial: in every state pick the action with the highest Q-value. The formal framework that ties these pieces together, including the Markov property that makes "state" meaningful, is covered in the Markov decision process article.

How does reinforcement learning work?#

Almost every RL algorithm alternates between two jobs: evaluate how good the current behaviour is, and change the behaviour to do better. What makes this possible is a recursive relationship that values satisfy, the Bellman equation [2]:

Here $V^{\ast }$ is the value under the best possible policy, $P(s^{\prime }\mid s,a)$ is the probability of landing in $s^{\prime }$ after taking $a$ in $s$, and $R(s,a,s^{\prime })$ is the reward for that transition. In words: the value of a state is the best you can do by taking one action, collecting its immediate reward and then continuing optimally from wherever you land. The value of a cell is defined in terms of its neighbours' values, so information about the charger flows backwards through the grid one step at a time. The Bellman equation article derives it and works through a backup with real numbers.

Take the cell directly below the charger. Moving up reaches the charger with probability 0.8 (+1), slides left into the pit with probability 0.1 (−1) and slides right into the wall with probability 0.1 (staying put). The optimal values come out as 0.769 for this cell and 0.648 for the one below it, and the Bellman equation checks out: moving up is worth $0.8\times 1+0.1\times (-1)+0.1\times 0.9\times 0.769=0.769$. The cell's value equals the value of its best action, which is exactly what the equation says.

There are three broad ways to use this relationship:

1. Dynamic programming. If you know $P$ and $R$, apply the Bellman equation as an update rule over all states until the values stop changing. This is value iteration and policy iteration. It is planning, not learning.
2. Temporal-difference learning. If you do not know $P$, replace the sum over next states with the one next state you actually observed, and nudge your estimate towards it. Q-learning is the best-known example [3, 4].
3. Policy optimisation. Skip values as the main object, parameterise the policy directly and follow the gradient of expected return. This is the policy gradient family, starting with REINFORCE [5].

Underneath all three sits the exploration problem. An agent that always takes the action it currently believes is best never finds out whether another action is better. The cleanest setting to study this trade-off is the one-state problem, the multi-armed bandit, where epsilon-greedy, UCB and Thompson sampling make different bets on how to explore.

Planning versus learning, side by side#

The figure below runs both approaches on the reference gridworld. Value iteration is given the transition model and sweeps the Bellman update across every cell. Q-learning is given nothing but the ability to act: it starts at S, takes epsilon-greedy actions, and after each step moves its estimate $Q(s,a)$ a fraction $\alpha$ of the way towards the observed reward plus the discounted best Q-value of the next state. The right-hand curve shows the quantity that actually matters, the true value from the start cell of the policy each method would follow if you stopped it now (computed exactly from the model, which the learner never sees).

Three things stand out, and they are general lessons rather than quirks of the toy.

Planning is cheap when you have a model. Value iteration needs 32 sweeps for its values to converge to within $10^{-8}$, but its greedy policy is already worth 0.273 from the start after 3 sweeps (93% of the optimum of 0.293) and is exactly optimal from sweep 11. It uses no experience at all. Policy iteration, which evaluates a policy fully before improving it, finds the optimal policy in 4 iterations from a policy that always moves up.

Learning without a model works, but noisily. In our numpy simulation (20 seeds, $\alpha =0.1$, $\epsilon =0.1$), Q-learning's greedy policy was worth 0.22 from the start after 100 episodes (95% interval across seeds 0.19 to 0.25), against −0.033 for the uniformly random policy it started from. It then stalled: after 3,000 episodes, about 47,000 environment steps, the mean was 0.205 (0.171 to 0.238), roughly 70% of the optimum. Single runs swing; the seed shown in the figure hovers around 0.28 with occasional dips, then falls to −0.02 at episode 3,000.

The stall is about step size and exploration, not about Q-learning being wrong. Q-learning provably converges to the optimal Q-function if every state-action pair is tried infinitely often and the step size shrinks appropriately [4]. Neither condition holds above. With a constant $\alpha =0.1$ the estimates keep jittering, and in cells where two actions are nearly tied the greedy choice flips. With $\epsilon =0.1$ the agent rarely visits cells away from its favourite route, so their estimates stay poor, and under slip it does end up in them. When we kept everything else fixed and changed only those knobs, the picture changed: a smaller step size $\alpha =0.02$ with fully random behaviour ($\epsilon =1$) reached 0.286 (97% of optimal) by 250 episodes and 0.291 (99%) by 1,000. That run also illustrates a property worth remembering: Q-learning is off-policy. It learned the optimal policy while behaving completely at random, because its update target uses the best next action rather than the one actually taken.

How a policy gradient method compares#

REINFORCE learns from whole episodes: it raises the probability of every action in proportion to the return that followed it. On the grid it is slow to start, because a random policy reaches the charger rarely and the return signal is mostly noise; its policy was worth −0.015 after 100 episodes, 0.095 after 1,000 and 0.236 after 2,000. But it improves steadily and ended at 0.274 (0.272 to 0.276), 93% of optimal, with far less spread between seeds than Q-learning at the end. This is the classic trade-off in miniature: value-based methods bootstrap from their own estimates and learn fast but can be unstable; policy gradient methods use actual returns and are unbiased but high-variance. Most of the practical progress in policy gradients, covered in baselines and advantage functions and actor-critic methods and GAE, is about getting the stability of one with the speed of the other.

Reinforcement learning vs supervised and unsupervised learning#

Reinforcement learning is often listed as the third paradigm of machine learning alongside supervised and unsupervised learning. The differences are about what feedback you get and where the data comes from.

The third row of the table hides the deepest difference. In supervised learning the data distribution is fixed. In RL the agent's policy decides which states it sees, so improving the policy changes the training data, and a bad early policy can keep the agent away from the very states it most needs to learn about. If you do have expert demonstrations, you can skip much of this: behaviour cloning is supervised learning on expert actions, and inverse reinforcement learning goes a step further and infers the reward the expert seems to be optimising.

Types of reinforcement learning algorithms#

The algorithm zoo is large, but most methods can be placed with four questions.

Model-based vs model-free#

Does the agent use a model of the environment's transitions? Model-based methods learn or are given $P$ and $R$ and plan with them, as value iteration does in Figure 2 and as tree search does in game-playing systems [8]. Model-free methods, such as Q-learning and policy gradients, learn values or policies directly from experience. Model-based methods are usually far more sample-efficient when the model is accurate and can fail badly when it is not, because the planner exploits the model's errors.

Value-based, policy-based and actor-critic#

Value-based methods learn $Q$ and act greedily with respect to it: Q-learning and its deep version, DQN [6, 7]. Policy-based methods optimise the policy's parameters by gradient ascent on expected return: the policy gradient theorem gives the gradient and REINFORCE estimates it from sampled episodes. Actor-critic methods combine the two: a critic learns a value function that reduces the variance of the actor's policy gradient (actor-critic and GAE). Trust-region and clipped methods such as TRPO and PPO [9] constrain how far each update can move the policy, which makes large neural policies trainable; the TRPO and PPO article explains why that works.

On-policy vs off-policy#

Can the agent learn about one policy while following another? Q-learning is off-policy, as the random-behaviour run above showed. REINFORCE and PPO are on-policy: they need fresh data from the current policy, which is simpler and more stable but wastes old experience.

Online vs offline#

Does the agent collect its own data? Online RL interacts with the environment as it learns. Offline reinforcement learning learns from a fixed log of past interactions with no further exploration, which matters wherever exploring is expensive or dangerous (medicine, industrial control, recommendation). Its central problem is that the learned policy may prefer actions the log rarely contains, whose values are therefore guesses [10].

Why reinforcement learning is hard#

The definition is one sentence; getting it to work is another matter. Four problems show up in every serious application, and the grid already exhibits each of them.

Exploration#

The agent only learns about what it tries. Our $\epsilon =0.1$ Q-learner found one decent route in its first hundred episodes and then mostly stuck to it, which is why it plateaued at 70% of optimal while the fully exploring learner with a smaller step size reached 99%. In large problems you cannot explore uniformly, so the question becomes how to explore cleverly. Bandits are where the theory is cleanest; see multi-armed bandits for how UCB and Thompson sampling beat epsilon-greedy.

Credit assignment and the discount#

The only reward in the grid arrives at the end of an episode. Which of the dozen moves before it deserves credit? Bootstrapped values solve this by passing credit backwards one step at a time, and the discount sets how far it can usefully travel.

With $\gamma =0.5$ the optimal value at the start is 0.002; the charger is eight steps away and $0.5^7$ is under 1%. With $\gamma =0.99$ it is 0.813, and the policy changes qualitatively: the cell left of the charger stops moving right (where a slip could drop it into the pit) and instead pushes against the top edge, waiting for a sideways slip to deliver it to the charger. The discount is not a nuisance parameter. It is part of the problem specification, and changing it changes what "optimal" means.

Reward design#

The reward is the only channel through which you tell the agent what you want, and agents optimise what you wrote, not what you meant. A common trick to make the robot hurry is a small penalty per step. At −0.04 per step the optimal action changes in only two cells. At −0.8 per step the optimal policy in two cells next to the pit is to jump in: one −1 now beats a long walk of −0.8 steps. That is a correct solution to the problem as specified and an obvious failure of the specification. Real systems produce subtler versions of this, from game-playing agents exploiting scoring quirks to language models learning to please a reward model rather than the user; see reward hacking, specification gaming and the safe way to add hints, potential-based reward shaping [11].

Variance and evaluation#

Two runs of the same algorithm with different seeds can end in very different places. At episode 3,000 our individual Q-learning runs ranged from below zero to near-optimal. Deep RL results are notoriously sensitive to seeds, hyperparameters and implementation details [12], and reporting a single run or the best of a few is a common way to fool yourself. Reliable comparisons need several seeds, interval estimates and outlier-resistant aggregates such as the interquartile mean [13]; how to evaluate reinforcement learning agents covers the practice, and our note on evaluation sample sizes covers the arithmetic.

Deep reinforcement learning#

A table with one entry per state and action works for 20 cells. It does not work for an Atari screen or a board position in Go. Deep reinforcement learning replaces the table with a neural network: a Q-network that maps an image to action values, or a policy network that maps a state to action probabilities. The breakthrough demonstration was DQN, which learned to play dozens of Atari games from pixels using Q-learning with two stabilising tricks, an experience replay buffer and a slowly updated target network [6, 7]. Game-tree search with learned policy and value networks then beat a professional Go player [8], and earlier still, TD-Gammon had learned backgammon at expert level by temporal-difference self-play [14].

Function approximation buys generalisation (similar states get similar values) at the price of the guarantees. The combination of bootstrapping, off-policy data and function approximation can diverge, and much of deep RL engineering is about keeping that combination stable. The Q-learning article covers DQN's fixes and the overestimation bias that motivates double Q-learning; the TRPO and PPO article covers the policy side.

Reinforcement learning for language models#

The most widely used application of RL today is training language models. A pretrained model is the policy, a prompt is the state, a generated response is the action (or a sequence of token-level actions), and a reward comes from a learned reward model trained on human preferences, from an AI judge, or from a program that checks the answer. Learning a reward from human preference comparisons was shown to work for deep RL on games and simulated robots [15] and then used to make language models follow instructions [16]; reinforcement learning with verifiable rewards uses checkable outcomes such as passing unit tests or a correct maths answer [17].

The ideas in this guide carry over directly. The policy gradient is the workhorse. A KL penalty keeps the policy close to its starting point, which is a trust region by another name. Reward hacking is the reward-design problem above at scale: optimise a learned reward model hard enough and the true quality falls while the proxy score climbs, a pattern we measured in our note on reward model overoptimisation. The reinforcement learning for LLMs guide covers this side of the field in full, with RLHF and RLVR as the two main recipes.

Reinforcement learning and AI safety#

An optimiser that finds whatever maximises a number will find the gaps in that number. That makes RL central to AI safety in two ways. First, as a source of risk: reward hacking, specification gaming, unsafe exploration and agents that pursue a proxy in situations the designer never anticipated were catalogued early as concrete problems [18]. Second, as a tool: safe reinforcement learning adds explicit constraints (for example a cap on the probability of entering the pit, solved as a constrained MDP), and RL from human or AI feedback is one of the main ways models are trained to refuse harmful requests. Either way, the discipline that keeps it honest is measurement: you cannot tell whether a trained policy is safe without evaluations that probe for the failure, and red teaming that searches for it adversarially. The AI safety guide places these pieces in context.

A minimal reinforcement learning program#

Here is the reference gridworld and a tabular Q-learning agent in plain numpy. It runs in a few seconds and prints the learned values and greedy policy.

import numpy as np

LAYOUT = ["....G", ".#.X.", ".#...", "...#.", "S...."]
MOVES = [(-1, 0), (0, 1), (1, 0), (0, -1)]          # up, right, down, left
GAMMA, SLIP, ALPHA, EPS = 0.9, 0.2, 0.1, 0.1
rng = np.random.default_rng(0)

def step(pos, a):
    """Environment: slip sideways with probability SLIP, reward on entering a cell."""
    u = rng.random()
    d = a if u < 1 - SLIP else (a + 1) % 4 if u < 1 - SLIP / 2 else (a + 3) % 4
    r, c = pos[0] + MOVES[d][0], pos[1] + MOVES[d][1]
    if not (0 <= r < 5 and 0 <= c < 5) or LAYOUT[r][c] == "#":
        r, c = pos                                   # bumped into a wall: stay put
    cell = LAYOUT[r][c]
    reward = 1.0 if cell == "G" else -1.0 if cell == "X" else 0.0
    return (r, c), reward, cell in "GX"

Q = np.zeros((5, 5, 4))
for episode in range(3000):
    pos, done = (4, 0), False
    while not done:
        if rng.random() < EPS:                       # explore
            a = int(rng.integers(4))
        else:                                        # exploit, ties broken at random
            q = Q[pos]; a = int(rng.choice(np.flatnonzero(q == q.max())))
        nxt, r, done = step(pos, a)
        target = r if done else r + GAMMA * Q[nxt].max()
        Q[pos][a] += ALPHA * (target - Q[pos][a])    # temporal-difference update
        pos = nxt

print(np.round(Q.max(axis=2), 2))                    # learned state values
print(np.array([["↑→↓←"[a] for a in row] for row in Q.argmax(axis=2)]))

With seed 0 this agent settles on the route along the bottom row and up the right-hand column. Its estimate at the start is 0.28 against the true optimum of 0.293, and the cells along its route are within a few hundredths of their optimal values. The top-left corner, which it almost never visits, is still near zero: the exploration lesson in a dozen numbers. (Arrows printed for walls and terminal cells are meaningless.) Try EPS = 1.0 and ALPHA = 0.02 to see off-policy learning fill in the whole grid.

How to learn reinforcement learning: a reading path#

If you are starting out, this order builds each idea on the last. Each link is a full article with its own interactive figure.

1. Multi-armed bandits: exploration with a single state.
2. Markov decision processes: the formal problem.
3. The Bellman equation: the recursion everything uses.
4. Value iteration vs policy iteration: solving a known MDP.
5. Q-learning and DQN: learning values without a model.
6. The policy gradient theorem and REINFORCE: optimising the policy directly.
7. Baselines and advantages, then actor-critic and GAE: making policy gradients practical.
8. TRPO and PPO: stable updates for large policies.
9. Reward shaping, inverse RL and offline RL: where rewards and data come from.
10. Evaluating RL agents: how to tell whether any of it worked.

Sutton and Barto's textbook [1] is free online and remains the best single reference; the older survey by Kaelbling, Littman and Moore [19] is a compact map of the classical field.

Before you reach for reinforcement learning#

RL is the right tool when decisions affect future situations and you can define a reward that you are willing to have optimised hard. It is often the wrong tool when a single prediction suffices (use supervised learning), when you have good demonstrations (start with behaviour cloning), or when you cannot afford the failures exploration produces (consider offline RL or a simulator first). A short checklist we use:

1. Write down the MDP. What is the state, what are the actions, when does an episode end, what is the discount? If you cannot answer, the problem is not ready.
2. Try to break your reward. Ask what the cheapest behaviour is that scores well. If it is not what you want, fix the reward before training, not after.
3. Get a planning baseline on a small version where the model is known. It tells you what optimal looks like.
4. Run several seeds from the start and report intervals, not best runs.
5. Evaluate the policy, not the training curve. Measure the final policy on held-out conditions, including the ones you are worried about.

Frequently asked questions#

Sources#

1. R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction (2nd ed.), 2018. incompleteideas.net
2. R. Bellman. Dynamic Programming, 1957. press.princeton.edu
3. C. J. C. H. Watkins. Learning from Delayed Rewards (PhD thesis), 1989. cs.rhul.ac.uk
4. C. J. C. H. Watkins, P. Dayan. Q-learning, 1992. doi.org
5. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, 1992. doi.org
6. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. Playing Atari with Deep Reinforcement Learning, 2013. arxiv.org
7. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, et al.. Human-level control through deep reinforcement learning, 2015. doi.org
8. D. Silver, A. Huang, C. J. Maddison, A. Guez, et al.. Mastering the game of Go with deep neural networks and tree search, 2016. doi.org
9. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal Policy Optimization Algorithms, 2017. arxiv.org
10. S. Levine, A. Kumar, G. Tucker, J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020. arxiv.org
11. A. Y. Ng, D. Harada, S. Russell. Policy invariance under reward transformations: theory and application to reward shaping, 1999. people.eecs.berkeley.edu
12. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, D. Meger. Deep Reinforcement Learning that Matters, 2018. arxiv.org
13. R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, M. G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice, 2021. arxiv.org
14. G. Tesauro. Temporal difference learning and TD-Gammon, 1995. doi.org
15. P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, D. Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org
16. L. Ouyang, J. Wu, X. Jiang, D. Almeida, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
17. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, et al.. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024. arxiv.org
18. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané. Concrete Problems in AI Safety, 2016. arxiv.org
19. L. P. Kaelbling, M. L. Littman, A. W. Moore. Reinforcement Learning: A Survey, 1996. doi.org