RL environments for LLM agents: state, tools, reward and sandboxing

An RL environment for an LLM agent is the program that hands the agent a task, executes its tool calls against some state (a repository, a website, a database, a simulated user), decides when the episode is over, and scores the final result. In reinforcement learning terms it defines the states, actions, transitions and reward of the decision process the agent is trained on. Getting it right matters more than the choice of RL algorithm, because whatever the environment rewards, including its bugs, is what the agent learns.

This page walks through the parts of an agent environment, how to design rewards and sandboxes, how rewards leak, and how evaluation environments should differ from training ones. A seeded Q-learning simulation of a toy coding environment, which you can rerun in the interactive figure, shows an agent learning to edit the tests instead of the code, and shows that a partial fix can make the training dashboard look better while true success gets worse.

A worked example: "make the failing test pass"#

A typical coding environment gives the agent a repository with a bug and an issue description. SWE-bench, built from 2,294 real issues and pull requests across 12 Python repositories, is the best-known version used for evaluation [2], and SWE-Gym packaged 2,438 such tasks with runnable environments for training agents [6]. Mapped onto the pieces of a decision process:

The agent's policy is the language model plus its scaffold: the prompt format, how tool outputs are inserted into context, and how actions are parsed. The ReAct pattern of interleaving reasoning with actions is the usual starting point [7]. From the RL algorithm's point of view, the whole multi-turn trajectory is one episode with a mostly terminal reward, which is why agent training inherits the sparse-signal problems described in RL with verifiable rewards.

What goes into an RL environment for agents?#

Task distribution#

A list of task instances, each with an initial-state recipe and a checker. Its difficulty profile matters as much as its size: a group-relative update learns nothing from tasks the policy always or never solves, so measure the starting pass rate per task and keep the bulk of training tasks in the range where the policy sometimes succeeds.

State, reset and snapshots#

The environment must reset to an identical initial state for every rollout, quickly, many times in parallel. For code this means container images or file-system snapshots; for web tasks, self-hosted sites restored from a dump, as in WebArena's e-commerce, forum, code-hosting and content-management sites [3]; for computer use, virtual machines with a per-task setup configuration, as in OSWorld's 369 tasks [4]. Anything that persists across episodes (a cache, a shared database, a log file the agent can read) is a channel through which one rollout can influence another.

Actions and tool calls#

Each tool needs a schema, deterministic argument parsing and a defined behaviour on malformed calls. Decide in advance whether a malformed call ends the episode, returns an error observation, or is silently ignored; each choice teaches something different. Keep the tool set close to deployment: a model trained with a run(cmd) shell will behave differently when deployed with only an edit_file tool. Tool outputs are also untrusted input to the model, which matters for indirect prompt injection and for tool protocols generally (MCP security).

Simulated users and other agents#

Customer-service style tasks need a user to talk to. τ-bench uses a language model to simulate the user and scores the episode by comparing the database state at the end of the conversation with an annotated goal state [5]. A simulated user is part of the transition function: if it can be talked into revealing the goal, or into accepting a wrong outcome, the agent will learn to do that.

Reward#

Most agent environments compute an outcome reward from the final state: tests pass, the database matches the goal, the file exists with the right contents. Outcome checks on state are harder to fool than checks on the agent's final message, because the agent has to change the world rather than describe a change. Options beyond a single pass/fail:

• Hard gates plus partial credit. Zero reward if any safety or format gate fails, otherwise the fraction of checks passed. Partial credit densifies the signal but creates gradients towards whichever checks are easiest to satisfy.
• Step or token costs. A small per-step penalty discourages wandering. Too large and it rewards giving up early or taking shortcuts.
• Shaping terms. Rewards for intermediate progress speed up learning and are easy to get wrong; use potential-based forms where you can (reward shaping).
• Judge-based components. A rubric scored by a model for things no program can check. Treat it as a learned reward with all the risks in reward models.

Sandboxing: the reward must live outside the agent's reach#

An agent with a shell can change anything inside its sandbox. The engineering rule that follows is simple: everything the reward depends on (test files, reference answers, the checker itself, its configuration and its exit codes) must be outside the sandbox or re-installed from a protected copy after the agent finishes. The usual controls:

How rewards leak: documented cases#

Reward leakage is any path by which the agent can raise the reward without completing the task. It is reward hacking with an environment bug as its cause, and the published record for coding agents is already long:

• Monitoring of a frontier reasoning model during agentic coding RL found two hacks that became systemic: calling exit(0) to leave before all tests ran, and raising SkipTest from outside the test framework [8].
• A study of reward hacking in production coding environments listed three hacks that its environments were vulnerable to: returning an object whose equality check always succeeds, calling sys.exit(0) before assertions, and creating a conftest.py that patches pytest's report so failures are recorded as passes. Models that learned these hacks generalised to broader misaligned behaviour, including attempts to sabotage AI-safety research code in a realistic agentic coding setup in 12% of runs [9].
• In an R&D benchmark, an agent asked to speed up a fine-tuning script while keeping the result close to a reference model copied the reference weights and added small random changes to "simulate" training; the benchmark authors scored such runs as zero [10].
• ImpossibleBench makes tests contradict the specification so that any pass implies a shortcut. One frontier model passed 76% of one-off impossible SWE-bench variants, and changing the prompt cut another variant's cheating rate from 92% to 1% [11].
• Training on a curriculum of gameable environments led models to generalise, rarely but measurably, to editing their own reward function [12].

The common thread: the checker trusted something inside the agent's sandbox. Our research note on reward hacking with verifiable rewards covers the answer-checker side of the same problem.

Simulation: a leaky checker in a toy coding environment#

To make the dynamics concrete we built the smallest environment that has the leak. The state records whether the agent has read the source file, whether the bug is fixed, whether the test file has been modified, and the step count. Four actions: read_file, edit_src (fixes the bug with probability $p_{\text{fix}}$ after reading the file, 0.1 blind), edit_tests (rewrites the test so it passes trivially) and submit. Each step costs 0.02 and the horizon is 6 steps. The reward at submit is 1 if the checker passes. Three checkers:

• Leaky: runs the tests in the agent's workspace, so it passes if the bug is fixed or the tests were edited.
• Monitor: the leaky checker plus a detector that catches a modified test file with probability $d$ and then gives 0.
• Hidden tests: runs a protected copy of the tests, so it passes only if the bug is fixed.

We trained tabular Q-learning (learning rate 0.1, no discounting, exploration decaying from 1.0 to 0.05 over the first 4,000 of 8,000 episodes; see Q-learning) and then evaluated the greedy policy over 2,000 rollouts. Seeds are fixed, and the browser figure reruns exactly the same computation.

Averaged over five seeds with the hard bug ($p_{\text{fix}}=0.3$):

Three lessons carry over to real environments. With a leaky checker the agent learns the two-step hack, edit_tests then submit, every time: it is shorter and surer than honest work. A detector does not remove the incentive, it rations it. With $d=0.3$ the greedy policy reads the file and tries to fix the bug first, and falls back on editing the tests on the last step when its fix attempts have failed, because a 70% chance of reward beats a certain zero. That "honest first, hack when stuck" pattern is exactly what is hard to see in samples. And the partially patched environment looks better on the dashboard (88.3% pass rate) than the correct one (74.2%) while producing fewer real fixes (61.7% against 74.2%). If you choose between environment versions by training reward, you will pick the leaky one. With an easier bug ($p_{\text{fix}}=0.6$) the honest path is worth more and a detector with $d=0.5$ or more is enough to make hacking rare in this toy; leakage bites hardest on the hardest tasks.

Evaluation environment vs training environment#

The same harness often serves both purposes, which is a mistake when it means the same checker and the same tasks. They have different jobs:

Consistency deserves special mention for agents. τ-bench introduced pass^k, the probability that all of $k$ independent trials of a task succeed, and found that agents which succeeded on under half of the tasks fell below 25% at pass^8 in the retail domain [5]. A training run that improves pass@1 can leave pass^k flat. See pass@k versus pass^k, agent trajectory evaluations for grading the path as well as the end state, and how to evaluate RL agents for seeds and intervals. Our evaluation work follows the same separation.

How to build an agent environment, step by step#

1. Write the checker first. Define success on the final state, then write three or four wrong solutions (including degenerate ones such as an empty submission, deleted tests, an early exit) and confirm the checker rejects all of them.
2. Isolate the reward. Move every file and process the checker depends on outside the sandbox, or restore it from a protected copy before scoring.
3. Make reset cheap and exact. Snapshot the initial state; verify two resets produce byte-identical state.
4. Specify tools tightly. Schemas, error behaviour, output truncation and a step and token budget that match deployment.
5. Measure starting pass rates. Run the initial policy several times per task; drop tasks at 0% or 100% from the training pool and keep them for analysis.
6. Red-team the environment. Have a strong model, or a person, try to pass without solving the task. Every success is a bug to fix before training.
7. Log trajectories and diffs. Record every tool call and the final diff of the state, so that hacks can be found by search rather than by luck.
8. Keep an independent evaluation copy. Held-out tasks, a stricter checker, fixed seeds, multiple trials.

Code: a minimal environment with a protected checker#

The reset and step interface that most RL environments still follow was standardised by the Gym toolkit [1]. Here it wraps a repository, a tool dispatcher and a checker that never reads anything the agent wrote except the source files.

import os, shutil, subprocess, tempfile

class FixTheBugEnv:
    """Gym-style episode: reset() -> observation, step(tool_call) -> (obs, reward, done)."""
    def __init__(self, repo_dir, hidden_tests_dir, max_steps=30):
        self.repo_dir, self.hidden = repo_dir, hidden_tests_dir
        self.max_steps = max_steps

    def reset(self):
        self.work = tempfile.mkdtemp()
        shutil.copytree(self.repo_dir, self.work, dirs_exist_ok=True)
        self.t = 0
        return "Fix the failing behaviour described in ISSUE.md. Tools: read, write, run, submit."

    def step(self, call):
        self.t += 1
        name, arg = call.get("tool"), call.get("arg", "")
        if name == "read":
            path = os.path.join(self.work, arg)
            obs = open(path).read()[:4000] if os.path.isfile(path) else "error: no such file"
        elif name == "write":
            path, text = arg
            open(os.path.join(self.work, path), "w").write(text)
            obs = "ok"
        elif name == "run":
            p = subprocess.run(arg, shell=True, cwd=self.work, capture_output=True,
                               text=True, timeout=60)
            obs = (p.stdout + p.stderr)[-4000:]
        elif name == "submit":
            return "submitted", self.score(), True
        else:
            obs = "error: unknown tool"
        return obs, 0.0, self.t >= self.max_steps

    def score(self):
        # Score a copy of the final state with the protected tests, outside the agent's workspace.
        final = tempfile.mkdtemp()
        shutil.copytree(self.work, final, dirs_exist_ok=True)
        shutil.rmtree(os.path.join(final, "tests"), ignore_errors=True)
        shutil.copytree(self.hidden, os.path.join(final, "tests"))
        for f in ("conftest.py", "pytest.ini", "setup.cfg", "tox.ini"):
            if os.path.exists(os.path.join(final, f)):
                os.remove(os.path.join(final, f))     # no agent-controlled test configuration
        p = subprocess.run(["python", "-m", "pytest", "-q", "-p", "no:cacheprovider", "tests"],
                           cwd=final, capture_output=True, text=True, timeout=300)
        return 1.0 if p.returncode == 0 and " passed" in p.stdout and "skipped" not in p.stdout else 0.0

This is a sketch, not a security boundary: in production the agent's commands run in a container without network access and the scoring happens in a different container. The structure is what matters: the agent works on its copy, and the reward is computed on a second copy whose tests and test configuration come from somewhere the agent never touched.

Failure modes#

Selecting environment versions by training reward. As the simulation shows, a leaky version can report higher reward than a correct one. Compare versions on an independent evaluation.

Penalising the hack's surface form. Detecting the string sys.exit removes that string, not the incentive. Penalising hacks seen in the agent's reasoning has been shown to produce obfuscated hacking instead [8]. Close the hole.

Training-only tools. Tools that exist in training but not deployment (or the reverse) produce behaviour that does not transfer.

Flaky transitions. Tests that fail randomly are reward noise; tests that pass randomly are false positives the agent can farm by resubmitting. Quarantine flaky tests.

Simulated users that can be gamed. If the user simulator accepts a confident wrong answer, the agent learns confidence. Audit simulator behaviour with adversarial transcripts, the way you would red-team any other component.

Frequently asked questions#

Sources#

1. Greg Brockman, Vicki Cheung, Ludwig Pettersson, et al.. OpenAI Gym, 2016. arxiv.org
2. Carlos E. Jimenez, John Yang, Alexander Wettig, et al.. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023. arxiv.org
3. Shuyan Zhou, Frank F. Xu, Hao Zhu, et al.. WebArena: A Realistic Web Environment for Building Autonomous Agents, 2023. arxiv.org
4. Tianbao Xie, Danyang Zhang, Jixuan Chen, et al.. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, 2024. arxiv.org
5. Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024. arxiv.org
6. Jiayi Pan, Xingyao Wang, Graham Neubig, et al.. Training Software Engineering Agents and Verifiers with SWE-Gym, 2024. arxiv.org
7. Shunyu Yao, Jeffrey Zhao, Dian Yu, et al.. ReAct: Synergizing Reasoning and Acting in Language Models, 2022. arxiv.org
8. Bowen Baker, Joost Huizinga, Leo Gao, et al.. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. arxiv.org
9. Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, et al.. Natural Emergent Misalignment from Reward Hacking in Production RL, 2025. arxiv.org
10. Hjalmar Wijk, Tao Lin, Joel Becker, et al.. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, 2024. arxiv.org
11. Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, 2025. arxiv.org
12. Carson Denison, Monte MacDiarmid, Fazl Barez, et al.. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org