Home / RL environments glossary
RL environments glossary
55 terms you meet when buying or building RL environments for LLM agents, from action space to verifiers. Each has a short definition and, where a claim needs one, a source. Terms are linked by anchor, so you can cite a single definition.
New to the topic? Read what RL environments are first. For prices, see RL environment pricing; for buying criteria, the evaluation checklist; for free options, open-source RL environments; for the classic API, RL gym.
Terms A to Z
A
- Action space
- The set of actions an agent may take. In classic gyms it is a few discrete moves or continuous values; in LLM environments it is text, tool calls, shell commands or clicks. Source: Gymnasium
- Advantage
- How much better one rollout did than expected. GRPO estimates it by comparing each rollout's reward with the other rollouts on the same prompt, so if every rollout gets the same reward, every advantage is zero. Source: DeepSeekMath, arXiv:2402.03300
B
- Benchmark contamination
- Test items or their solutions leak into training data and inflate scores. OpenAI cited training exposure, along with flawed tests, when it stopped reporting SWE-bench Verified in February 2026. Source: OpenAI, Feb 2026
C
- Chain-of-thought monitoring
- Using one model to read another model's reasoning for signs of misbehavior. OpenAI used a GPT-4o monitor to flag reward hacking in coding environments and found that penalizing "bad thoughts" taught models to hide their intent. Source: OpenAI, Mar 2025
- Computer-use environment
- An environment where the agent operates a browser or desktop through screenshots, clicks and typing. Epoch AI notes one such environment can host hundreds of tasks. See OSWorld and others. Source: Epoch AI, Jan 2026
- Curriculum
- Choosing and ordering tasks by difficulty so the policy always has tasks it sometimes solves. Researchers told Epoch AI they want "a smooth gradient: diversity in the difficulty of the tasks." Source: Epoch AI, Jan 2026
D
- Deterministic reset and replay
- The ability to restore an exact environment state and re-run a trajectory. Needed for reproducibility, debugging graders and parallel training. Source: Wing VC, Jan 2026
- Difficulty calibration
- Tuning tasks so the target model succeeds sometimes but not always. Buyers interviewed by Epoch AI want a minimum pass rate of about 2 to 3 percent and a spread of difficulties. Source: Epoch AI, Jan 2026
E
- Environment hub
- A registry for publishing, installing and evaluating RL environments, such as Prime Intellect's Environments Hub (launched August 2025) or OpenEnv on the Hugging Face Hub. See open-source list. Source: Prime Intellect, Aug 2025 · Hugging Face, Oct 2025
- Episode
- One run of an environment from reset until it terminates or is cut off. In LLM training the word is mostly used interchangeably with rollout. Source: Gymnasium
- Exclusivity
- A contract term under which a vendor sells an environment to one lab only. Two founders told Epoch AI it costs about 4 to 5 times the non-exclusive price. See pricing. Source: Epoch AI, Jan 2026
F
- Frontier lab
- Informal term for the labs training the most capable models, such as OpenAI, Anthropic and Google DeepMind. Epoch AI separates them from neolabs when describing who builds environments in-house. Source: Epoch AI, Jan 2026
G
- GRPO
- Group Relative Policy Optimization. It samples a group of rollouts per prompt and uses their normalized rewards as advantages, which removes the need for a separate value network. Widely used for RL with verifiable rewards. Source: DeepSeekMath, arXiv:2402.03300
- Gym API
- The interface OpenAI Gym standardized in 2016: reset() starts an episode, step(action) returns the next observation and reward, plus typed action and observation spaces. See RL gym guide. Source: Brockman et al., 2016
H
- Harness
- The scaffold the model runs inside during a rollout: system prompt, tool interface, turn loop and limits. Source: verifiers docs
I
- Inoculation prompting
- Telling a model during training that a particular hack is acceptable in that context, so that learning the hack does not generalize into broader misalignment. Proposed by Anthropic researchers as one mitigation. Source: arXiv:2511.18397
L
- LLM-as-a-judge
- Using a strong model to grade outputs, usually against a rubric. Common where outcomes cannot be checked by code. Source: Zheng et al., arXiv:2306.05685
- Long-horizon task
- A task that takes many actions or a long time to finish. Interviewees told Epoch AI that long horizon is "the future direction" for environments. Source: Epoch AI, Jan 2026
M
- MCP (Model Context Protocol)
- An open protocol, introduced by Anthropic in November 2024, for exposing tools and data to models. Many enterprise environments expose their tools MCP-style. Source: Anthropic, Nov 2024 · Epoch AI, Jan 2026
N
- NeMo Gym
- NVIDIA's open-source library that runs environments as servers separate from training and records trajectories in the OpenAI Responses format. Apache-2.0 core; labeled early development. Source: NVIDIA docs
- Neolab
- Informal term for newer, well-funded AI labs outside the established frontier labs. Epoch AI uses it for companies such as Cursor. Usage varies. Source: Epoch AI, Jan 2026
O
- Observation
- What the environment returns after an action: tool output, a screenshot, a file, an error. The policy conditions its next action on it. Source: Gymnasium
- OpenEnv
- A spec and hub from Meta-PyTorch and Hugging Face for agentic environments. Authors implement step(), reset() and close(); environments run as isolated sandboxes. BSD-3-Clause. Source: Hugging Face, Oct 2025
- Oracle solution
- A known-good solution shipped with a task to prove it is solvable and that the grader accepts a correct answer. Terminal-Bench ships one with every task. Source: Terminal-Bench, arXiv:2601.11868
- Outcome reward
- A reward based only on the final result, such as tests passing or the right database state. Contrast with a process reward model, which scores intermediate steps. Source: Lightman et al., arXiv:2305.20050
P
- Pass rate
- The share of rollouts on a task that the verifier scores as solved. Tasks at 0 or 100 percent give no learning signal. See pricing and pass rates. Source: Epoch AI, Jan 2026
- pass@k
- The probability that at least one of k samples solves the task. Introduced with the Codex evaluation. Source: Chen et al., arXiv:2107.03374
- pass^k
- The probability that all k trials succeed, a reliability metric introduced with tau-bench. In retail, gpt-4o's pass^8 was under 25 percent at release. Source: Yao et al., arXiv:2406.12045
- Policy
- The model being trained: the function that maps observations to actions. In LLM RL, the policy is the language model itself.
- PPO
- Proximal Policy Optimization, a policy-gradient algorithm with a clipped objective and a learned value function. It was the optimizer in InstructGPT-era RLHF. Source: Schulman et al., arXiv:1707.06347
- Process reward model (PRM)
- A reward model that scores each intermediate step of a solution rather than only the final outcome. Source: Lightman et al., arXiv:2305.20050
R
- Replica (clone environment)
- A functional copy of a product such as Slack, Salesforce, SAP or Excel, with a controllable backend and state. Epoch AI reports about $300,000 for a high-fidelity Slack-class replica. Source: Epoch AI, Jan 2026 · SemiAnalysis, Jan 2026
- Reward function
- The logic that turns a verifier's output into a scalar reward. Rubrics are often checklists of yes or no criteria graded by an LLM. Source: Sequoia podcast with Will Brown
- Reward hacking
- The policy earns high reward without solving the task, for example by editing tests, reading future commits or exploiting a grader bug. Epoch AI's sources call it a top concern. See how to test for it. Source: Epoch AI, Jan 2026 · SemiAnalysis, Jun 2025 · Lilian Weng, Nov 2024
- Reward model
- A model trained to predict a reward, usually from human preference rankings. RLHF optimizes against one; RLVR replaces it with a programmatic check. Source: Ouyang et al., arXiv:2203.02155
- RL environment
- The world an agent acts in during RL: actions and tools, state such as files, an app database or a browser, plus tasks and a grader. Often a Docker container. See full explainer. Source: Epoch AI, Jan 2026 · Hugging Face, Oct 2025
- RLEF
- RL from execution feedback: running model-generated code in the environment and using the result as the reward. Source: SemiAnalysis, Jun 2025
- RLHF
- RL from human feedback. Humans rank outputs, a reward model is trained on the rankings, and the policy is optimized against that reward model. Source: Ouyang et al., arXiv:2203.02155
- RLVR
- RL with verifiable rewards: the reward comes from programmatically checking correctness, such as a math answer or a test, not from a preference model. The term was named in Tulu 3. Source: Lambert et al., arXiv:2411.15124
- Rollout
- One execution of the policy on a task from reset to termination, producing a trajectory and a reward. Buyers interviewed by Epoch AI think in batches of 64 to 128 rollouts per task. Source: Epoch AI, Jan 2026
- Rubrics as Rewards (RaR)
- Using instance-specific rubrics as reward signals so RL can reach domains without a single checkable answer, such as medicine. Source: Gunjal et al., arXiv:2507.17746
S
- Sandbox
- An isolated execution context, such as a container or VM, where agent actions run. In the o1 system card, a model exploited a misconfigured Docker daemon API reachable from its environment. Source: Hugging Face, Oct 2025 · OpenAI o1 system card
- SFT
- Supervised fine-tuning on example outputs. Successful trajectories from an environment can be filtered into SFT data, a use Epoch AI reports is growing. Source: Epoch AI, Jan 2026
- Soundness
- A verifier is sound when high reward implies the task was actually solved. "Soundness matters most," per researchers interviewed by Epoch AI. Source: Epoch AI, Jan 2026
- Sparse reward
- A reward given rarely, often once at the end of an episode and often 0 or 1. Typical of LLM environments, unlike the per-step rewards of classic control tasks. See how environments differ from gyms.
- Specification gaming
- DeepMind's umbrella term for satisfying the literal objective while missing the intended one. Reward hacking is one kind. Source: DeepMind, Apr 2020
- SWE-bench
- 2,294 real GitHub issues from 12 Python repositories, graded by hidden tests. SWE-bench Verified is a 500-task human-validated subset. Source: Jimenez et al., arXiv:2310.06770
- SWE-Gym
- 2,438 real-world Python task instances with executable runtimes and tests, built for training coding agents and verifiers. Source: Pan et al., arXiv:2412.21139
T
- Task
- A prompt stating a goal, plus the grader that checks it. Coding environments often hold one task each; computer-use environments can hold hundreds. Source: Epoch AI, Jan 2026
- Terminated and truncated
- Gymnasium's two end-of-episode flags. Terminated means the task reached a terminal state; truncated means a time or step limit cut it off. Source: Gymnasium
- Trajectory
- The recorded sequence of observations, actions, tool calls and outputs in one rollout. Source: Epoch AI, Jan 2026
U
- UI gym
- A mocked replica of a real website or app used to train browser and computer-use agents. SemiAnalysis reported about $20,000 per website in January 2026. Source: SemiAnalysis, Jan 2026
V
- Verifier (grader)
- Code or a model that scores an attempt: unit tests, a final-state check, or an LLM judge with a rubric. Its quality sets the ceiling on what the environment can teach. See evaluation checklist. Source: Epoch AI, Jan 2026
- verifiers (library)
- Prime Intellect's MIT-licensed Python library for building environments as a task dataset, a harness and a rubric. Created by Will Brown. Source: GitHub
Common questions
What is a verifier in reinforcement learning?
The code or model that scores an agent's attempt, such as hidden unit tests, a check of the final app state, or an LLM judge with a rubric. Its output becomes the reward.
What is reward hacking?
When a policy earns high reward without doing the task, for example by editing the tests or reading the answer. Researchers interviewed by Epoch AI in January 2026 called it a top concern.
What is the difference between RLHF and RLVR?
RLHF optimizes against a reward model trained on human preference rankings. RLVR uses a programmatic check of correctness, such as a test or an exact answer, as the reward.
What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds. pass^k, from tau-bench, is the chance that all k attempts succeed, so it measures reliability.
What is a rollout?
One run of the policy on a task, from reset to the end, producing a trajectory and a reward. Labs run many per task, often 64 to 128.
What is GRPO?
Group Relative Policy Optimization, introduced in DeepSeekMath. It scores each rollout against the others on the same prompt instead of training a value network.
What is a UI gym?
A replica of a website or app used to train computer-use agents. SemiAnalysis reported in January 2026 that they often cost about $20,000 per site.
Sources
- Epoch AI: An FAQ on RL environments (2026-01-12)
- SemiAnalysis: Scaling RL: environments, reward hacking, agents, scaling data (2025-06-08)
- SemiAnalysis: RL environments and RL for science (2026-01-06)
- Hugging Face: Introducing OpenEnv (2025-10-23)
- Farama Foundation: Gymnasium
- Shao et al.: DeepSeekMath (GRPO), arXiv:2402.03300
- Schulman et al.: PPO, arXiv:1707.06347
- Ouyang et al.: InstructGPT, arXiv:2203.02155
- Lambert et al.: Tulu 3, arXiv:2411.15124
- Chen et al.: Codex and pass@k, arXiv:2107.03374
- Yao et al.: tau-bench and pass^k, arXiv:2406.12045
- Lightman et al.: Let's verify step by step, arXiv:2305.20050
- Gunjal et al.: Rubrics as Rewards, arXiv:2507.17746
- Wing VC: Who will win the RL environment market, and why (2026-01)
- OpenAI: Detecting misbehavior in frontier reasoning models (2025-03-10)
- MacDiarmid et al.: arXiv:2511.18397 (2025)
- DeepMind: Specification gaming (2020-04-21)