1  # Hermes-Agent Atropos Environments
  2  
  3  This directory contains the integration layer between **hermes-agent's** tool-calling capabilities and the **Atropos** RL training framework. It provides everything needed to run agentic LLMs through multi-turn tool-calling loops, score their output with arbitrary reward functions, and feed results into Atropos for training or evaluation.
  4  
  5  ## Architecture Overview
  6  
  7  ```
  8                          Atropos Framework
  9                      ┌───────────────────────┐
 10                      │       BaseEnv          │  (atroposlib)
 11                      │  - Server management   │
 12                      │  - Worker scheduling   │
 13                      │  - Wandb logging       │
 14                      │  - CLI (serve/process/ │
 15                      │    evaluate)           │
 16                      └───────────┬───────────┘
 17                                  │ inherits
 18                      ┌───────────┴───────────┐
 19                      │  HermesAgentBaseEnv    │  hermes_base_env.py
 20                      │  - Terminal backend    │
 21                      │  - Tool resolution     │
 22                      │  - Agent loop          │
 23                      │  - ToolContext          │
 24                      │  - Async patches       │
 25                      └───────────┬───────────┘
 26                                  │ inherits
 27                ┌─────────────────┼─────────────────┐
 28                │                 │                  │
 29       TerminalTestEnv     HermesSweEnv    TerminalBench2EvalEnv
 30       (stack testing)     (SWE training)   (TB2 benchmark eval)
 31  ```
 32  
 33  ### Inheritance Chain
 34  
 35  **BaseEnv** (from `atroposlib`) is the Atropos base class. It provides:
 36  - Server management (OpenAI-compatible API servers, VLLM, SGLang)
 37  - Worker scheduling for parallel rollouts
 38  - Wandb integration for metrics and rollout logging
 39  - CLI interface with three subcommands: `serve`, `process`, `evaluate`
 40  - `evaluate_log()` for saving eval results to JSON + samples.jsonl
 41  
 42  **HermesAgentBaseEnv** (`hermes_base_env.py`) extends BaseEnv with hermes-agent specifics:
 43  - Sets `os.environ["TERMINAL_ENV"]` to configure the terminal backend (local, docker, modal, daytona, ssh, singularity)
 44  - Resolves hermes-agent toolsets via `_resolve_tools_for_group()` (calls `get_tool_definitions()` which queries `tools/registry.py`)
 45  - Implements `collect_trajectory()` which runs the full agent loop and computes rewards
 46  - Supports two-phase operation (Phase 1: OpenAI server, Phase 2: VLLM ManagedServer)
 47  - Applies monkey patches for async-safe tool operation at import time
 48  
 49  Concrete environments inherit from `HermesAgentBaseEnv` and implement:
 50  - `setup()` -- Load dataset, initialize state
 51  - `get_next_item()` -- Return the next item for rollout
 52  - `format_prompt()` -- Convert a dataset item into the user message
 53  - `compute_reward()` -- Score the rollout using ToolContext
 54  - `evaluate()` -- Periodic evaluation logic
 55  
 56  ## Core Components
 57  
 58  ### Agent Loop (`agent_loop.py`)
 59  
 60  `HermesAgentLoop` is the reusable multi-turn agent engine. It runs the same pattern as hermes-agent's `run_agent.py`:
 61  
 62  1. Send messages + tools to the API via `server.chat_completion()`
 63  2. If the response contains `tool_calls`, execute each one via `handle_function_call()` (which delegates to `tools/registry.py`'s `dispatch()`)
 64  3. Append tool results to the conversation and go back to step 1
 65  4. If the response has no tool_calls, the agent is done
 66  
 67  Tool calls are executed in a thread pool (`run_in_executor`) so backends that use `asyncio.run()` internally (Modal, Docker) don't deadlock inside Atropos's event loop.
 68  
 69  Returns an `AgentResult` containing the full conversation history, turn count, reasoning content per turn, tool errors, and optional ManagedServer state (for Phase 2).
 70  
 71  ### Tool Context (`tool_context.py`)
 72  
 73  `ToolContext` is a per-rollout handle that gives reward/verification functions direct access to **all** hermes-agent tools, scoped to the rollout's `task_id`. The same `task_id` means the terminal/browser session is the SAME one the model used during its rollout -- all state (files, processes, browser tabs) is preserved.
 74  
 75  ```python
 76  async def compute_reward(self, item, result, ctx: ToolContext):
 77      # Run tests in the model's terminal sandbox
 78      test = ctx.terminal("pytest -v")
 79      if test["exit_code"] == 0:
 80          return 1.0
 81  
 82      # Check if a file was created
 83      content = ctx.read_file("/workspace/solution.py")
 84      if content.get("content"):
 85          return 0.5
 86  
 87      # Download files locally for verification (binary-safe)
 88      ctx.download_file("/remote/output.bin", "/local/output.bin")
 89  
 90      return 0.0
 91  ```
 92  
 93  Available methods:
 94  - **Terminal**: `terminal(command, timeout)` -- run shell commands
 95  - **Files**: `read_file(path)`, `write_file(path, content)`, `search(query, path)`
 96  - **Transfers**: `upload_file()`, `upload_dir()`, `download_file()`, `download_dir()` -- binary-safe file transfers between host and sandbox
 97  - **Web**: `web_search(query)`, `web_extract(urls)`
 98  - **Browser**: `browser_navigate(url)`, `browser_snapshot()`
 99  - **Generic**: `call_tool(name, args)` -- call any hermes-agent tool by name
100  - **Cleanup**: `cleanup()` -- release all resources (called automatically after `compute_reward`)
101  
102  ### Patches (`patches.py`)
103  
104  **Problem**: Some hermes-agent tools use `asyncio.run()` internally (e.g., the Modal backend). This crashes when called from inside Atropos's event loop because `asyncio.run()` cannot be nested.
105  
106  **Solution**: `ModalEnvironment` uses a dedicated `_AsyncWorker` background thread with its own event loop. The calling code sees a sync interface, but internally all async Modal SDK calls happen on the worker thread so they don't conflict with Atropos's loop. This is built directly into `tools/environments/modal.py` — no monkey-patching required.
107  
108  `patches.py` is now a no-op (kept for backward compatibility with imports).
109  
110  ### Tool Call Parsers (`tool_call_parsers/`)
111  
112  Client-side parsers that extract structured `tool_calls` from raw model output text. Used in **Phase 2** (VLLM server type) where ManagedServer's `/generate` endpoint returns raw text without tool call parsing.
113  
114  Each parser is a standalone reimplementation of the corresponding VLLM parser's `extract_tool_calls()` logic. No VLLM dependency -- only standard library (`re`, `json`, `uuid`) and `openai` types.
115  
116  Available parsers:
117  - `hermes` -- Hermes/ChatML `<tool_call>` XML format
118  - `mistral` -- Mistral `[TOOL_CALLS]` format
119  - `llama3_json` -- Llama 3 JSON tool calling
120  - `qwen` -- Qwen tool calling format
121  - `qwen3_coder` -- Qwen3 Coder format
122  - `deepseek_v3` -- DeepSeek V3 format
123  - `deepseek_v3_1` -- DeepSeek V3.1 format
124  - `kimi_k2` -- Kimi K2 format
125  - `longcat` -- Longcat format
126  - `glm45` / `glm47` -- GLM model formats
127  
128  Usage:
129  ```python
130  from environments.tool_call_parsers import get_parser
131  
132  parser = get_parser("hermes")
133  content, tool_calls = parser.parse(raw_model_output)
134  ```
135  
136  In Phase 1 (OpenAI server type), these parsers are not needed -- the server handles tool call parsing natively.
137  
138  ## Two-Phase Operation
139  
140  ### Phase 1: OpenAI Server (Evaluation / SFT Data Generation)
141  
142  Uses `server.chat_completion()` with `tools=` parameter. The server (VLLM, SGLang, OpenRouter, OpenAI) handles tool call parsing natively. Returns `ChatCompletion` objects with structured `tool_calls`.
143  
144  - Good for: evaluation, SFT data generation, testing
145  - Run with: `serve` (with `run-api`), `process`, or `evaluate` subcommands
146  - Placeholder tokens are created for the Atropos pipeline
147  
148  ### Phase 2: VLLM ManagedServer (Full RL Training)
149  
150  Uses ManagedServer for exact token IDs + logprobs via `/generate`. Client-side tool call parser (from `tool_call_parsers/`) reconstructs structured `tool_calls` from raw output.
151  
152  - Good for: full RL training with GRPO/PPO
153  - Run with: `serve` subcommand
154  - Real tokens, masks, and logprobs flow through the pipeline
155  
156  ## Directory Structure
157  
158  ```
159  environments/
160  ├── README.md                     # This file
161  ├── __init__.py                   # Package exports
162  ├── hermes_base_env.py            # Abstract base (HermesAgentBaseEnv)
163  ├── agent_loop.py                 # Multi-turn agent engine (HermesAgentLoop)
164  ├── tool_context.py               # Per-rollout tool access for reward functions
165  ├── patches.py                    # Async-safety patches for Modal backend
166  │
167  ├── tool_call_parsers/            # Phase 2 client-side parsers
168  │   ├── __init__.py               # Registry + base class
169  │   ├── hermes_parser.py
170  │   ├── mistral_parser.py
171  │   ├── llama_parser.py
172  │   ├── qwen_parser.py
173  │   ├── qwen3_coder_parser.py
174  │   ├── deepseek_v3_parser.py
175  │   ├── deepseek_v3_1_parser.py
176  │   ├── kimi_k2_parser.py
177  │   ├── longcat_parser.py
178  │   ├── glm45_parser.py
179  │   └── glm47_parser.py
180  │
181  ├── terminal_test_env/            # Stack validation environment
182  │   └── terminal_test_env.py
183  │
184  ├── hermes_swe_env/               # SWE-bench style training environment
185  │   └── hermes_swe_env.py
186  │
187  └── benchmarks/                   # Evaluation benchmarks
188      ├── terminalbench_2/          # 89 terminal tasks, Modal sandboxes
189      │   └── terminalbench2_env.py
190      ├── tblite/                   # 100 calibrated tasks (fast TB2 proxy)
191      │   └── tblite_env.py
192      └── yc_bench/                 # Long-horizon strategic benchmark
193          └── yc_bench_env.py
194  ```
195  
196  ## Concrete Environments
197  
198  ### TerminalTestEnv (`terminal_test_env/`)
199  
200  A self-contained environment with inline tasks (no external dataset needed) for validating the full stack end-to-end. Each task asks the model to create a file at a known path, and the verifier checks the content matches.
201  
202  ```bash
203  # Serve mode (needs run-api)
204  run-api
205  python environments/terminal_test_env/terminal_test_env.py serve
206  
207  # Process mode (no run-api, saves to JSONL)
208  python environments/terminal_test_env/terminal_test_env.py process \
209      --env.data_path_to_save_groups terminal_test_output.jsonl
210  ```
211  
212  ### HermesSweEnv (`hermes_swe_env/`)
213  
214  SWE-bench style training environment. The model gets a coding task, uses terminal + file + web tools to solve it, and the reward function runs tests in the same Modal sandbox.
215  
216  ```bash
217  python environments/hermes_swe_env/hermes_swe_env.py serve \
218      --openai.model_name YourModel \
219      --env.dataset_name bigcode/humanevalpack \
220      --env.terminal_backend modal
221  ```
222  
223  ### TerminalBench2EvalEnv (`benchmarks/terminalbench_2/`)
224  
225  **Eval-only** environment for the Terminal-Bench 2.0 benchmark (89 tasks). Each task gets a pre-built Docker Hub image, a natural language instruction, and a test suite. The agent uses terminal + file tools to solve the task, then the test suite verifies correctness.
226  
227  Follows the standard Atropos eval pattern (like GPQA, MMLU, etc.):
228  - Run via `evaluate` subcommand (no `run-api` needed)
229  - `setup()` loads the dataset, `evaluate()` runs all tasks
230  - `rollout_and_score_eval()` handles per-task agent loop + test verification
231  - Downloads verifier output locally for reliable reward checking (Harbor pattern)
232  
233  ```bash
234  # Run full benchmark
235  python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
236      --openai.model_name anthropic/claude-opus-4.6
237  
238  # Run subset of tasks
239  python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
240      --openai.model_name anthropic/claude-opus-4.6 \
241      --env.task_filter fix-git,git-multibranch
242  
243  # Skip specific tasks
244  python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
245      --openai.model_name anthropic/claude-opus-4.6 \
246      --env.skip_tasks heavy-task,slow-task
247  ```
248  
249  ## Creating a New Environment
250  
251  ### Training Environment
252  
253  1. Create a new directory under `environments/`
254  2. Create your env file inheriting from `HermesAgentBaseEnv`
255  3. Implement the four abstract methods + `evaluate()`
256  
257  ```python
258  from environments.hermes_base_env import HermesAgentBaseEnv, HermesAgentEnvConfig
259  
260  class MyEnvConfig(HermesAgentEnvConfig):
261      pass  # Add custom fields as needed
262  
263  class MyEnv(HermesAgentBaseEnv):
264      name = "my-env"
265      env_config_cls = MyEnvConfig
266  
267      @classmethod
268      def config_init(cls):
269          env_config = MyEnvConfig(
270              enabled_toolsets=["terminal", "file"],
271              terminal_backend="modal",
272              # ... other config
273          )
274          server_configs = [APIServerConfig(...)]
275          return env_config, server_configs
276  
277      async def setup(self):
278          self.dataset = load_dataset(...)
279          self.iter = 0
280  
281      async def get_next_item(self):
282          item = self.dataset[self.iter % len(self.dataset)]
283          self.iter += 1
284          return item
285  
286      def format_prompt(self, item):
287          return item["instruction"]
288  
289      async def compute_reward(self, item, result, ctx):
290          # ctx gives you full tool access to the rollout's sandbox
291          test = ctx.terminal("pytest -v")
292          return 1.0 if test["exit_code"] == 0 else 0.0
293  
294      async def evaluate(self, *args, **kwargs):
295          # Periodic evaluation logic
296          ...
297  
298  if __name__ == "__main__":
299      MyEnv.cli()
300  ```
301  
302  ### Eval-Only Environment (Benchmark)
303  
304  For eval benchmarks, follow the pattern in `terminalbench2_env.py`:
305  1. Create under `environments/benchmarks/your-benchmark/`
306  2. Inherit from `HermesAgentBaseEnv`
307  3. Set eval-only config: `eval_handling=STOP_TRAIN`, `steps_per_eval=1`, `total_steps=1`
308  4. Stub the training methods (`collect_trajectories`, `score`)
309  5. Implement `rollout_and_score_eval()` and `evaluate()`
310  6. Run with `evaluate` subcommand
311  
312  ## Key Config Fields
313  
314  | Field | Description | Default |
315  |-------|-------------|---------|
316  | `enabled_toolsets` | Which hermes toolsets to enable | `None` (all) |
317  | `disabled_toolsets` | Toolsets to disable | `None` |
318  | `distribution` | Probabilistic toolset distribution name | `None` |
319  | `max_agent_turns` | Max LLM calls per rollout | `30` |
320  | `agent_temperature` | Sampling temperature | `1.0` |
321  | `terminal_backend` | `local`, `docker`, `modal`, `daytona`, `ssh`, `singularity` | `local` |
322  | `system_prompt` | System message for the agent | `None` |
323  | `tool_call_parser` | Parser name for Phase 2 | `hermes` |
324  | `eval_handling` | `STOP_TRAIN`, `LIMIT_TRAIN`, `NONE` | `STOP_TRAIN` |