RL gym: the classic API and the new gyms for LLM agents

Two meanings of "RL gym"

• The classic meaning. An environment that implements the Gym interface. OpenAI released Gym in 2016 as "a toolkit for reinforcement learning research," with Atari, control and robotics tasks and a common API (Brockman et al.). The Farama Foundation now maintains it as Gymnasium.
• The 2025 to 2026 meaning. Any environment used to post-train an LLM agent. SemiAnalysis calls website replicas "UI gyms" (January 2026). Deeptune describes its products as "training gyms" that simulate jobs such as accountant or DevOps engineer across tools like Slack and Salesforce (Fortune, March 2026). Turing sells "RL gyms" for sales workflows on replicas of HubSpot, Outreach and Calendly (Turing).

The two meanings share a shape. Both have a reset, a step, an observation and a reward. They differ in what an action is, how the reward is computed, and what it costs to run one step. Our RL environments explainer covers the second meaning in depth.

The Gym API in brief

The Gym contract is small, which is why it spread. An environment exposes:

• reset(seed): start a new episode and return the first observation and an info dict.
• step(action): apply one action and return the next observation, the reward, and whether the episode ended.
• action_space and observation_space: typed descriptions of valid inputs and outputs.
• close(): release resources.

This is the loop from the Gymnasium README, checked November 9, 2026:

import gymnasium as gym
env = gym.make("CartPole-v1")

observation, info = env.reset(seed=42)
for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)

    if terminated or truncated:
        observation, info = env.reset()
env.close()

Two details trip up people porting old code. Gymnasium's step() returns five values. The old single done flag is split into terminated (the task reached a terminal state) and truncated (a time or step limit cut it off). The split matters for value bootstrapping: a truncated episode did not really end, so its last state still has value.

Status and license

The openai/gym repository is archived on GitHub. Gymnasium is MIT licensed and had its latest push on November 8, 2026. Its paper describes it as the maintained successor with a standard API that extends Gym's (Towers et al.).

From Gym to gyms for LLM agents

Put an LLM in the Gym loop and three things change.

• The action is text. A tool call, a shell command, a patch, a click on a screenshot. There is no small discrete action space to sample from.
• The reward is sparse and comes from a verifier. Hidden tests, a check of the app's database, or an LLM judge with a rubric, usually once at the end. See verifier.
• One step is expensive. A step can mean a model forward pass of thousands of tokens plus a container executing a command. Environments run on CPU servers, while the policy runs on GPUs.

The newer frameworks keep Gym's verbs. OpenEnv, from Meta-PyTorch and Hugging Face, asks environment authors to implement step(), reset() and close() and serves environments as isolated sandboxes (Hugging Face, October 2025). Prime Intellect's verifiers library frames an environment as a dataset, a harness and a rubric. NVIDIA's NeMo Gym separates environment servers from the training loop and records trajectories in the OpenAI Responses format (NeMo Gym docs).

Today's gyms and hubs

The full list, with licenses checked on GitHub, is in our open-source RL environments guide.

Commercial gyms are a different market. Most vendors keep environments closed and sell them under exclusive contracts. SemiAnalysis names Prime Intellect, which open-sources, as the exception. Prices are private, but reported ranges are on our pricing page.

How to pick an RL gym

1. Start from the skill, not the framework. Pick the environment whose tasks look like the work you want the model to do. A gym that matches the target workflow beats a better-engineered one that does not.
2. Check the grader before anything else. Read the verifier code. Can the agent edit the tests, read the answer, or reach the internet? Our evaluation checklist lists the probes.
3. Measure pass rates on your model. Run 64 rollouts per task on your current checkpoint. Epoch AI reports buyers want at least about 2 to 3% success, since tasks at 0% or 100% give no learning signal.
4. Match your trainer. If you train with TRL or verl, OpenEnv integrations save glue code. If you use prime-rl, verifiers environments plug in directly. On NeMo, use NeMo Gym.
5. Check contamination. Gyms built from public repos or benchmarks may overlap with pretraining data. OpenAI stopped reporting SWE-bench Verified in February 2026 for this and other reasons.
6. Read the license, including the data. Code licenses do not always cover bundled datasets, Docker images or the websites a replica copies.
7. Price the compute, not just the gym. Mechanize estimates about $2,400 of compute is spent per task during RL training, per Epoch. A free gym with broken tasks is not free.