What Are RL Environments? How LLM Agents Train

The short answer

In Epoch AI's framing, a model gets a task and a set of actions it can take. A grader, often a unit test or an LLM judging against a rubric, scores each attempt, and the scores update the model's weights. The environment is everything around the task: the actions, the file system, environment variables and simulated app state. It is "often a Docker container, but not always" (Epoch AI, January 2026).

Two caveats from the same source. The boundary between environment and task is "somewhat fuzzy," and the terminology is not standardized. When a vendor says "environment," ask whether they mean one container, one task, or a catalog of both.

Anatomy of an RL environment

Strip away the branding and every environment we have seen has the same four parts, plus the harness that connects the model to them.

Task. A prompt that states the goal: "fix the failing test," "find the cheapest 2-bedroom listing," "build a pivot table of Q3 sales." Epoch defines a task as prompt plus grader.
State. What the model can read and change: a Git repository, a database behind a cloned web app, a virtual desktop, a spreadsheet. State must reset cleanly between attempts.
Tools (the action space). Shell commands, file edits, API or MCP-style tool calls, browser actions, or mouse and keyboard on screenshots.
Verifier and reward. Code or a model that checks the outcome and emits a reward. Examples are hidden unit tests, a check of the final database state, or an LLM judge with a rubric. See verifier in our glossary.
Harness. The scaffold that runs the model: system prompt, tool schema, turn loop and limits. Prime Intellect's verifiers library packages an environment as a task dataset, a harness and a rubric.

Terminal-Bench gives a clean template for one task: an instruction, a Dockerfile, a set of tests, and an oracle solution that proves the task is solvable. The tests are copied into the container and run after the agent finishes (arXiv:2601.11868). Copying tests in only at grading time is a small detail that stops the most common hack, an agent editing the tests.

Hugging Face and Meta's OpenEnv spec reduces the interface to three calls: reset(), step() and close(). That is the same shape as the classic Gym API, which we cover in our RL gym guide.

The loop, drawn

One rollout is many action and observation turns, then one reward. The trainer batches rewards from many rollouts before it updates the policy.

Types of RL environments, with examples

Categories overlap, and vendors slice them differently. These are the groups that show up across Epoch AI, SemiAnalysis and the research literature.

Type	What the agent does	How it is graded	Public examples
Coding and SWE	Edits a real repository to resolve an issue	Hidden unit tests	SWE-bench (2,294 issues, 12 Python repos), SWE-Gym (2,438 tasks)
Terminal and CLI	Completes system tasks in a shell	Tests run after the agent stops	Terminal-Bench 2.0 (89 tasks)
Browser and computer use	Clicks and types through websites or a desktop OS	Final page or system state	WebArena, OSWorld (369 tasks)
Enterprise app replicas	Works inside clones of Slack, Salesforce, Gmail or Excel	Backend state, reference output	TheAgentCompany, WorkArena
Tool use and conversation	Calls domain APIs while talking to a simulated user	Final database state against a goal	tau-bench
Math and verifiable reasoning	Produces a checkable answer	Exact or symbolic match	Tulu 3 RLVR data
Rubric-based domains	Writes answers in medicine, law, finance	LLM judge with an instance-specific rubric	Rubrics as Rewards

Coding

SemiAnalysis called coding the highest-demand category in January 2026 and reported that DeepSeek used 24,667 coding tasks extracted from GitHub (SemiAnalysis). Coding environments are also growing past "make the tests pass." One founder told Epoch the newer ones include "GitHub, they have Linear, they have a code IDE." The hard part is grading quality beyond unit tests.

Computer use and UI gyms

A computer-use environment can host hundreds of tasks against one app. SemiAnalysis calls mocked website replicas "UI gyms," with clones of sites like DoorDash or Uber Eats, and reports that OpenAI "has purchased hundreds of sites" for ChatGPT Agent work. Labs replicate sites instead of training on the live ones partly because some companies block agents: Epoch cites Amazon suing Perplexity.

Enterprise workflows

Expense reports, pivot tables, slides and CRM updates, exposed as MCP-style tools, Playwright-style browser actions or screenshot computer use. "I think enterprise workflows are going to explode this year," one founder told Epoch in January 2026. TheAgentCompany, a simulated software company with internal sites and simulated coworkers, is the research version: its best agent finished 30% of tasks at release (Xu et al., 2024).

Math, rubrics and long-horizon tasks

Math tasks are cheap to make because they only need a checkable answer. Founders told Epoch that math "might be shrinking" and transfers poorly. Domains without a single right answer use rubrics as rewards. The direction most interviewees pointed to is multi-turn, long-horizon work: "Long horizon is I think the future direction."

How they differ from classic RL gyms

OpenAI Gym (2016) was "a toolkit for reinforcement learning research" built around Atari, control and robotics benchmarks (Brockman et al.). TechCrunch treats those gyms as the precursor and argues today's environments are harder to build because agents act in open-ended software with many ways to fail (TechCrunch, September 2025).

	Classic gym (CartPole, Atari)	LLM agent environment
Actions	A few discrete moves or continuous torques	Free text, tool calls, shell commands, clicks
Reward	Dense, hand-coded each step	Sparse, from a verifier at the end
State	A small vector or a frame	A repo, a database, a desktop
Runtime	One process, microseconds per step	Containers or app replicas, mostly on CPU servers next to GPU training
Main failure	Policy does not learn	Policy learns to game the grader

The table is our synthesis. The CPU point is SemiAnalysis's: "most environments... run on CPUs only servers and not on GPUs" (June 2025). It matters for planning because environment hosting scales with rollout count, not with model size.

How they differ from RLHF preference data

In RLHF as described in InstructGPT, labelers rank model outputs and those rankings train a reward model. The policy then optimizes against that learned model of human preference (Ouyang et al., 2022). The reward is a guess about what people like.

Environments mostly use verifiable rewards. Tulu 3 named this RLVR: the reward comes from checking an answer or a constraint, not from a preference model (Lambert et al., 2024). DeepSeek-R1 then showed reasoning can be trained through pure RL without human-labeled reasoning traces (DeepSeek-AI, 2025).

The practical difference for a buyer: preference data is a static dataset you consume once. An environment is software that produces new rollouts and rewards every time you run it. a16z described the shift as a move to "dynamic, engineered systems that generate high-quality training signals at scale" (March 2026). Rubric-graded environments sit in between, since an LLM judge is still a learned grader.

Why labs buy them now

RL on verifiable tasks works. Tulu 3 and DeepSeek-R1 showed that outcome-graded rewards can train reasoning. Agents need the same kind of graded tasks, set inside real software.
Spend is large. The Information reported in September 2025 that Anthropic leaders discussed spending over $1 billion on RL environments over the following year, as cited by TechCrunch. That was a reported discussion, not a confirmed budget. Epoch notes compute still dwarfs it: OpenAI's 2026 R&D compute is projected at about $19 billion.
Supply is new. SemiAnalysis counted more than 35 companies building environments in January 2026, and believes Anthropic works with more than a dozen of them.
Public benchmarks saturate. a16z cited OSWorld scores of 72.7% for Claude Opus 4.6 and 75% for GPT-5.4 in March 2026, around the 72.36% human baseline. At release in 2024 the best model scored 12.24%.

Labs also build. Epoch reports "substantially more in-housing" recently, to avoid vendor margins and keep training priorities private. Most big labs do both. Our pricing guide covers what buying costs.

How a lab uses an environment

Rollouts

A rollout is one attempt from reset to termination: the model reads the task, takes actions, sees observations, stops, and gets a reward. Labs run many rollouts per task in parallel. Epoch's interviewees spoke in batches of 64 to 128.

Pass rates

A task that the model always fails or always solves teaches nothing. Group-relative methods like GRPO compute each rollout's advantage against the other rollouts on the same prompt (Shao et al., 2024). If all 64 rewards are 0, or all are 1, every advantage is zero and the gradient from that task vanishes. That is why buyers want a minimum pass rate of about 2 to 3%, or at least one success in 64 to 128 rollouts (Epoch AI). At a true 2% pass rate, the chance of at least one success is about 73% in 64 rollouts and 92% in 128.

Curriculum

Researchers told Epoch they want "a smooth gradient: diversity in the difficulty of the tasks." In practice that means a mix: some tasks at 0%, some at 5%, some at 30%. Tasks near 70% may be dropped. As the policy improves, tasks drift toward 100% and stop paying for their compute, so the set needs periodic re-measuring and refilling. A 0% task may simply be impossible, which is why Epoch's sources check 0% tasks with a blind human solve.

Watching for reward hacking

Policies find grader bugs. A third-party evaluator found Claude 3.7 Sonnet "would directly edit the 'tests' file to cause all tests to pass" (SemiAnalysis). Our evaluation checklist covers how to test for this before you buy.

Beyond RL

The same environment can serve as an eval, and successful trajectories can be filtered into SFT data. Epoch reports RL use is perhaps 10 to 20 times benchmarking use. One founder said trailing labs lean on SFT because large-scale RL pipelines are hard to run.

Where to start

To try the loop yourself, the open-source options include SWE-Gym, BrowserGym, OSWorld, tau-bench and Prime Intellect's Environments Hub. Terms on this page are defined in the glossary.

Who builds them

Epoch groups suppliers into specialized startups (SWE, computer use, math, finance), traditional data providers such as Mercor, Surge, Handshake and Turing, in-house teams at frontier labs and neolabs, and product companies that partner with labs, such as Benchling with Anthropic and Shopify and Stripe with OpenAI. We compare the vendors that sell to labs, with a source for every fact, on our RL environment vendor comparison.

Common questions

What is an RL environment?

It is the software world a model acts in during reinforcement learning: tasks, the state the model can change, the tools it can call, and a grader that scores each attempt. Epoch AI notes it is often a Docker container, but not always.

What is the difference between an RL environment and a benchmark?

Mostly the use. The same container, tasks and grader can train a model or measure it. Epoch AI reported in January 2026 that labs use environments for RL perhaps 10 to 20 times more than for benchmarking.

Is an RL environment the same as RLHF data?

No. RLHF data is a static set of human rankings used to train a reward model. An environment is running software with a programmatic or rubric-based grader, and it can produce fresh attempts and rewards for as long as you run it.

What does a lab actually receive when it buys one?

Usually a containerized app or repository state, a set of task prompts, and the graders for those tasks. Computer-use environments can hold hundreds of tasks each, while coding environments often hold one.

What makes a good RL environment?

A grader that cannot be gamed, tasks the target model solves sometimes but not always, enough diversity to generalize, and infrastructure that runs many parallel rollouts quickly. Our evaluation guide turns this into a buyer checklist.

Which kinds of environments are in most demand?

SemiAnalysis called coding the highest-demand category in January 2026. Founders interviewed by Epoch AI expected enterprise workflows to grow fast in 2026, and described math as possibly shrinking.

How many rollouts does a lab run per task?

Epoch AI reports buyers think in batches of 64 to 128 rollouts per task, and want at least one success in that many attempts. That works out to a minimum pass rate of roughly 2 to 3 percent.

Can environments be used for supervised fine-tuning too?

Yes. Successful trajectories can be filtered and used for SFT, and Epoch AI reports this use is growing, especially for interleaved thinking and tool calls.

RL environments: what they are and how labs use them