Reinforcement learning
Reinforcement learning is the study of agents that learn by acting. Instead of copying labelled answers, an agent tries something, receives a reward, and adjusts its behaviour so that future rewards are larger. The same loop drives game-playing systems, robot controllers and the fine-tuning stage of modern language models, which is why the ideas on these pages keep reappearing in AI safety and evaluation work.
The material is ordered so each page builds on the last. Markov decision processes give the vocabulary: states, actions, transitions, rewards and discounting. The Bellman equation turns that vocabulary into a recursion, and value iteration and policy iteration solve it when the model is known. Bandits isolate the exploration problem, Q-learning removes the need for a model, and the policy-gradient pages derive REINFORCE from first principles before fixing its variance with baselines, critics and generalised advantage estimation. Trust regions and clipping explain why PPO is stable. The last three pages cover learning from logged data, recovering rewards from demonstrations, and shaping rewards without changing the optimal policy.
Every article includes worked numbers from small simulations we ran, and most include an interactive figure you can change.
What is reinforcement learning? A guide from gridworlds to language models
Reinforcement learning explained with a gridworld you can play with: agents, rewards, policies, values, planning vs Q-learning, algorithm types, pitfalls.
Suggested reading order
- 1What is reinforcement learning? A guide from gridworlds to language models
Reinforcement learning explained with a gridworld you can play with: agents, rewards, policies, values, planning vs Q-learning, algorithm types, pitfalls.
- 2Markov decision process (MDP): definition, example and how to solve one
What a Markov decision process is, its five components, the Markov property and discounting, with an interactive gridworld and seeded simulations.
- 3The Bellman equation: derivation, intuition and a worked example
Bellman expectation and optimality equations derived step by step, a backup worked with real numbers, matrix form, contraction and an interactive demo.
- 4Value iteration vs policy iteration: two ways to solve an MDP
Value iteration and policy iteration side by side: equations, a worked gridworld, stopping rules, modified policy iteration and timed runs on large grids.
- 5Multi-armed bandits: epsilon-greedy, UCB and Thompson sampling, explained and measured
What a multi-armed bandit is, how regret works, and how epsilon-greedy, UCB and Thompson sampling compare in seeded simulations, including vs A/B tests.
- 6Q-learning explained: the update rule, worked examples and DQN
Q-learning from first principles: the update rule worked by hand, a live agent, seeded results on step size, SARSA on the cliff, double Q-learning and DQN.
- 7The policy gradient theorem: a full derivation, checked numerically
Derive the policy gradient theorem from the log-derivative trick, see why dynamics drop out, and check every step on a two-state MDP you can solve by hand.
- 8The REINFORCE algorithm: derivation, pseudocode, numpy code and its variance problem
Williams's REINFORCE explained: the update, pseudocode, runnable numpy code, and measured gradient noise that shows why plain REINFORCE learns slowly.
- 9Policy gradient baselines and the advantage function: why they reduce variance without bias
Why subtracting a baseline leaves the policy gradient unbiased, what the advantage function is, the optimal baseline, and measured variance reductions.
- 10Actor-critic methods and generalized advantage estimation (GAE), with the λ trade-off measured
How actor-critic methods use a critic and the TD error, how GAE blends n-step advantages with lambda, and measured bias and variance for every lambda.
- 11PPO in reinforcement learning, from first principles: trust regions, natural gradients and clipping
PPO from first principles: the surrogate objective, TRPO's KL trust region, the natural gradient, the clipped objective, and measured KL per update.
- 12Offline reinforcement learning: learning from logged data without falling off the cliff
What offline RL is, why distribution shift makes Q-learning overrate unseen actions, and how CQL, IQL and behaviour regularisation fix it. Live demo.
- 13Inverse reinforcement learning: inferring the reward behind the behaviour
What inverse RL is, why many rewards explain the same demonstrations, how max-entropy IRL works, and how it leads to learning from preferences.
- 14Reward shaping: why potential-based shaping is safe and other bonuses can break your agent
What reward shaping is, why potential-based shaping (Ng et al. 1999) keeps optimal policies, how other bonuses create loops, and its effect on learning.
Guides and explainers
Reward model overoptimization: why gold reward peaks while the proxy keeps climbing
Why gold reward rises then falls as you optimise a reward model, how best-of-n and RL differ in KL, and what KL penalties and ensembles really fix.
Reward hacking with verifiable rewards: measuring and hardening the verifier
How policies exploit RLVR verifiers, why a small false-positive rate lifts training reward while true accuracy falls, and how to audit and harden them.
Process vs outcome supervision: right answers, wrong reasoning
Why outcome reward models cannot reduce right-answer, wrong-reasoning rates, what step labels cost, and when PRM-guided search amplifies hidden errors.
Research notes on this topic
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.