Safe exploration: Definition and Examples

Safe exploration is the problem of letting a reinforcement learning agent try new actions to learn, while avoiding actions that cause unacceptable harm during the learning process itself. A policy that is safe after training is not enough if reaching it required crashing the robot a thousand times.

Why it is hard

Exploration means taking actions whose consequences the agent does not yet know. Standard strategies such as epsilon-greedy or an entropy bonus randomise without regard to danger, and optimism-based methods actively seek out the unknown. In an environment with irreversible failures, any strategy that treats all unknowns as worth trying will eventually try a catastrophic one.

Worked example

An epsilon-greedy agent with $ϵ = 0.1$ and four actions walks along a cliff edge where one action falls off. Each step it picks a random action with probability 0.1, and that random action is the fatal one with probability 1/4, so the per-step risk is 0.025. Over 100 steps next to the cliff, the chance of never falling is

(1 - 0.025)^{100} \approx 0.08.

A greedy policy that already knows the cliff is dangerous still falls with 92 percent probability, purely because of how it explores.

Main approaches

Constraints: formulate the task as a constrained MDP and keep expected cost under a limit, using Lagrangian or trust-region methods.
Shielding: a separate component, often built from a formal model of the safe states, checks each proposed action and replaces unsafe ones before they execute.
Prior knowledge and safe sets: start from a known-safe policy or region and expand it only where a model is confident, as in Lyapunov and Gaussian-process methods.
Learning from logs or simulators: do the risky exploration in simulation or skip it entirely with offline RL.
Human oversight: require approval for actions above a risk threshold.

Safe exploration for AI agents

Language model agents trained or deployed in live environments face the same issue: an agent exploring a filesystem, a codebase or an external API can delete data or send messages that cannot be recalled. The practical answers mirror the RL ones: sandboxes instead of production, tool-call policies that act as shields, and human approval for irreversible actions.

Why it matters

Safe exploration is one of the five problems in the widely cited Concrete Problems in AI Safety agenda because it is where the learning process itself, not just its result, can cause harm. Evaluating it means counting violations during training or deployment, not just measuring the final policy.

Sources

Javier García, Fernando Fernández. A Comprehensive Survey on Safe Reinforcement Learning, 2015. jmlr.org
Mohammed Alshiekh, Roderick Bloem, Ruediger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu. Safe Reinforcement Learning via Shielding, 2017. arxiv.org
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Concrete Problems in AI Safety, 2016. arxiv.org