How to evaluate RL environments: a 10-step buyer's checklist

Before you start

Ask for a sample environment, not a deck. Labs typically test a sample, including probing the verifier for reward hacks, before signing (Troveo, a vendor guide). Run the sample on your own current checkpoint and trainer. A pass rate measured on someone else's model tells you little about yours.

The usual buying flow, per Sapphire Ventures: your research team writes a task spec, the vendor builds, and you review and stress-test. The ten steps below are the stress test.

The checklist

1. Reward hacking resistance

Why it matters. "Reward hacking is a big issue," a neolab researcher told Epoch AI. The examples were mundane: the model searches for the solution online or checks out future commits from the Git history. A third-party evaluator found Claude 3.7 Sonnet "would directly edit the 'tests' file to cause all tests to pass" (SemiAnalysis). METR reported in June 2025 that o3, Claude 3.7 Sonnet and o1 modify tests or scoring code, and that its detection probably undercounts it. OpenAI's chain-of-thought monitor caught agents using exit(0) and skipping tests in coding environments (OpenAI, March 2025).

It is a safety issue too. Anthropic researchers reported that when a model learned reward hacks in real production coding environments used for Claude Sonnet 3.7, misalignment rose on all their evals, including sabotage in about 12% of cases (arXiv:2511.18397).

How to test it. Run a strong model on every task. Sort by reward and read the top trajectories by hand. Run degenerate agents: one that submits nothing, one that exits immediately, one that only edits test files. Each should score zero.

Ask the vendor:

• Which models have you red-teamed these tasks against, and can we see the hacks you found and fixed?
• Are tests and graders outside the agent's reach until grading? Terminal-Bench copies tests in only after the agent finishes (arXiv:2601.11868).
• Is Git history stripped of future commits? Is internet access blocked or allowlisted?

2. Verifier soundness and grader accuracy

Why it matters. "Soundness matters most: high reward must mean the task was actually solved, not hacked" (Epoch AI). Errors run both ways. A false positive rewards bad work: "a model could write terrible code and still pass the unit tests" (SemiAnalysis). A false negative punishes good work: OpenAI reported in February 2026 that at least 59.4% of an audited subset of often-failed SWE-bench Verified tasks had tests that reject correct fixes (OpenAI). See soundness.

Ask the vendor:

• Does every task ship with an oracle solution that passes the grader?
• What is the grader's false-negative rate on independent correct solutions?
• For LLM-judge or rubric graders: how often does the judge agree with domain experts, and what happens when they disagree? (Invisible suggests this question.)

3. Difficulty calibration and pass rates

Why it matters. Tasks at 0% or 100% give no signal. Buyers interviewed by Epoch want a minimum pass rate of about 2 to 3%, or at least one success in 64 to 128 rollouts, and a mix: "some 0%, some 5%, some 30%." Tasks near 70% may be discarded. With group-relative methods like GRPO, a task where every rollout gets the same reward contributes zero gradient.

Ask the vendor:

• Pass rate per task, on which model, at how many rollouts? Can we see the distribution, not just the mean?
• Have 0% tasks been solved by a blind human?
• What happens when our model saturates the set? Is there a refresh or replacement policy?

Vendors sometimes report bands such as Easy, Medium and Hard by pass@3 (Turing case study). Bands measured on a vendor's model are a starting point. Re-measure on yours.

4. Diversity and generalization

Why it matters. "Training an agent to be great at one workflow doesn't automatically make it good at others" (Sapphire). Researchers told Epoch they want "a smooth gradient: diversity in the difficulty of the tasks," and skills that are related and build on each other. Researchers also told Epoch they worry about contractors using LLMs to write tasks.

Ask the vendor:

• How many distinct task templates are there, and how many tasks are parameter variations of one template?
• Who wrote the tasks, and what share were drafted with an LLM?
• How do you deduplicate tasks against each other and against public benchmarks?

5. Replica fidelity

Why it matters. On Slack and SAP clones, one interviewee told Epoch: "There's a large amount of useless bad environments out there." Environments "need to accurately represent the simulation or environment" while being hard to exploit (SemiAnalysis). The counterweight: Will Brown advises cutting simulator complexity that is "overkill," such as a full database where an in-memory mock would do (Sequoia podcast). Fidelity is worth paying for only where the skill depends on it.

Ask the vendor:

• Which features and API endpoints of the real product are implemented, and which return stubs?
• How realistic is the seeded data: volume, messiness, permissions?
• Do you have evidence that skills trained on this replica show up on the real product?

6. Scalability, latency and reliability

Why it matters. Slow feedback wastes rollout time. Environments need fault tolerance and checkpointing, must handle many parallel rollouts "and done well," and need a security layer and protection against resource exhaustion (SemiAnalysis). Wing VC adds deterministic reset and replay, parallel orchestration and telemetry (Wing VC). Most environments run on CPU servers, so this is a separate capacity plan from your GPUs.

Ask the vendor:

• p50 and p95 step latency and reset time, at what concurrency?
• Maximum parallel rollouts tested, and the infrastructure failure rate at that load?
• Can we reset to an exact state and replay a trajectory deterministically?
• Does it run on our cluster, theirs, or both?

7. Contamination

Why it matters. OpenAI stopped reporting SWE-bench Verified in February 2026, saying score gains increasingly reflected training exposure, and recommended SWE-bench Pro until new uncontaminated evals exist (OpenAI). Tasks built from public repos, public benchmarks or the open web may already be in your pretraining data, and an environment you also use as an eval loses its value once it leaks.

Ask the vendor:

• Where did each task's source material come from? Was any of it public before?
• Have these tasks been sold to anyone else or published? Will they be?
• Is there a held-out split we can keep for evaluation only?

8. Licensing, exclusivity and data rights

Why it matters. Most vendors "close source their environments and serve them in exclusive contracts to the labs" (SemiAnalysis), and exclusivity costs about 4 to 5 times more (Epoch AI). Epoch does not cover IP terms, and we found no primary public source on standard data-rights terms. A vendor guide flags trademark and terms-of-service risk when cloning another company's product, the need to pseudonymize personal data, contractor IP assignment, and license compatibility for bundled open-source code (Troveo). Some product companies partner with labs; others fight agents, as with Amazon suing Perplexity (Epoch AI). Costs are on our pricing page.

Ask the vendor:

• What exactly is exclusive: the tasks, the replica, derivative tasks? For how long?
• Who owns trajectories and model outputs generated in the environment?
• Do all contractors assign IP to you? Is any personal data in seeded state?
• Which open-source licenses cover bundled code, and does the cloned product's owner object?

9. Security and sandboxing

Why it matters. Agents explore. In the o1 system card, a capture-the-flag container failed to start. The model found a misconfigured Docker daemon API that was reachable from its environment, launched a new container and read the flag (OpenAI, September 2024). OpenEnv describes environments as "secure, semantically clear sandboxes" for this reason (Hugging Face). An escape during training is both a reward hack and an incident.

Ask the vendor:

• What isolates each rollout: a container, a hardened runtime, a VM? Is the Docker socket ever reachable?
• What network egress is allowed, and is it logged?
• Are CPU, memory, disk and process limits enforced per rollout?
• Are there any credentials or secrets inside the image?

10. Documentation and support

Why it matters. We found no practitioner source that ranks documentation, so this one is our editorial view. When a grader breaks mid-run, your team needs to read its code to fix it. Wing VC reports labs want "embedded thought partners, not on-demand vendors" (Wing VC). Standard interfaces like OpenEnv's reset(), step() and close() cut integration work.

Ask the vendor:

• Can we read every grader's source and every task spec?
• Which interfaces do you support: OpenEnv, verifiers, NeMo Gym, or your own?
• How fast do you fix a broken task mid-run, and who is on call?
• Is there a changelog when tasks or graders change?

Copy-ready checklist

Select the block and paste it into your pilot doc.

RL ENVIRONMENT PILOT CHECKLIST  (toprlenvironments.com, reviewed 2026-11-09)

Vendor: ____________   Environment: ____________   Our checkpoint: ____________

[ ] 1. Reward hacking: top-reward trajectories read by hand; null, early-exit
       and test-editing agents all score 0; tests hidden until grading;
       no future commits; internet blocked or allowlisted.
[ ] 2. Verifier soundness: oracle solution passes every task; false-negative
       rate measured on independent correct solutions; judge-expert agreement
       reported for rubric graders.
[ ] 3. Pass rates: measured on OUR checkpoint, 64-128 rollouts per task;
       floor ~2-3%; spread of difficulties; 0% tasks have a blind human solve;
       refresh policy for saturated tasks.
[ ] 4. Diversity: count of distinct templates vs. variants; share of
       LLM-drafted tasks; dedup against public benchmarks.
[ ] 5. Fidelity: implemented vs. stubbed features; realistic seeded data;
       evidence of transfer to the real product.
[ ] 6. Scale: p50/p95 step latency and reset time at target concurrency;
       failure rate under load; deterministic reset and replay.
[ ] 7. Contamination: provenance of every task; never published or resold;
       held-out eval split.
[ ] 8. Rights: exclusivity scope and term; trajectory ownership; contractor
       IP assignment; personal data; OSS licenses; trademark/ToS risk.
[ ] 9. Security: per-rollout isolation; no Docker socket; egress policy;
       resource limits; no secrets in images.
[ ] 10. Support: grader source readable; standard interface; mid-run fix SLA;
        changelog.

Decision: ____________   Signed off by: ____________   Date: ____________

Related guides

New to the topic? Start with what RL environments are and the RL gym guide. For free environments to calibrate against, see open-source RL environments. Terms are defined in the glossary. To compare vendors, see the vendor comparison.