An action is what a reinforcement learning agent chooses to do at time step , and the action space is the set of choices available to it. The action, together with the current state, determines the reward and the distribution of the next state in a Markov decision process.
Discrete vs continuous action spaces
A discrete action space has finitely many options, such as the moves in a board game. A policy over discrete actions is usually a softmax over learned scores :
A continuous action space contains real-valued vectors, such as joint torques for a robot arm. There a policy typically outputs the mean and standard deviation of a Gaussian, and an action is sampled as with .
The split matters for algorithms. Methods that take a maximum over actions, such as Q-learning, are easy with ten discrete actions and awkward in continuous spaces, where is itself an optimisation problem. Policy gradient methods handle both, because they only need to sample actions and compute .
Tokens as actions
When reinforcement learning is applied to a language model, each generated token is usually treated as an action and the vocabulary is the action space. The space is discrete but very large, and what matters is the space of whole responses. With a vocabulary of 50,000 tokens, there are possible 20-token sequences, which is why exploration in language-model RL relies on the pretrained model to propose sensible candidates rather than on random choice.
Why it matters
The action space defines what an agent can do, and therefore what it can do wrong. Safety work often restricts it directly: masking actions that violate a rule, clipping continuous controls to safe ranges, or requiring approval before high-impact tool calls. A smaller, well-designed action space also makes learning faster, because there are fewer options to explore and credit assignment is easier.
Related terms
Sources
- Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net