File size: 7,415 Bytes
0b2c941 5845692 0b2c941 5845692 0b2c941 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | # `openenv.core.harness`
Session-based runtime for driving rollouts against an OpenEnv environment
through an LLM + MCP-style tool loop.
- `__init__.py` — runtime primitives (from RFC 005 / PR #471):
`ResourceSession`, `ResourceSessionFactory`, `StepEnvSessionAdapter`,
`HarnessAdapter`, `MCPHarnessAdapter`, `CLIHarnessAdapter`,
`build_harness_rollout_func`, etc.
- `environment.py`, `adapter.py` — `HarnessEnvironment` and
`AgenticHarnessAdapter` (RFC 005), to run an external agentic harness inside
an environment, one conversational turn per `step()`. Worked example:
[`examples/claude_code_harness_eval`](../../../../examples/claude_code_harness_eval).
- `collect.py` — synthetic dataset generation on top of the runtime:
`EpisodeRecord`, `RolloutSerializer`, `CollectRunner`, `build_model_step`,
`push_to_hf_hub`.
## Quick start
The `openenv collect` CLI wires everything together. See `openenv collect --help`.
```bash
# Smoke test (no API key, no Docker — scripted teacher picks the first legal action)
openenv collect openspiel:tic_tac_toe \
--base-url https://<user>-<space>.hf.space \
--output-dir /tmp/ttt-scripted \
-n 10 --provider scripted
# Real LLM teacher + push the dataset to the Hub
OPENAI_API_KEY=... openenv collect openspiel:tic_tac_toe \
--base-url https://<user>-<space>.hf.space \
--output-dir /tmp/ttt-sft-v1 \
-n 200 --provider openai --model gpt-5-mini \
--push-to-hub <user>/ttt-sft-v1
# Self-hosted teacher (vLLM/TGI/Ollama) via any OpenAI-compatible base URL
openenv collect openspiel:tic_tac_toe \
--base-url https://<user>-<space>.hf.space \
--output-dir /tmp/ttt-sft-local \
-n 200 --llm-endpoint http://localhost:8000 --model Qwen/Qwen3-1.7B
```
Programmatic use:
```python
from openenv.core.harness import HarnessRunLimits, MCPHarnessAdapter
from openenv.core.harness.collect import (
CollectRunner,
RolloutSerializer,
build_model_step,
push_to_hf_hub,
)
from openenv.core.llm_client import create_llm_client
from openspiel_env.client import OpenSpielEnv
from openspiel_env.harness import OpenSpielSessionFactory
factory = OpenSpielSessionFactory(
lambda: OpenSpielEnv(base_url="https://<user>-<space>.hf.space"),
game_name="tic_tac_toe",
)
llm = create_llm_client(provider="openai", model="gpt-5-mini", api_key=...)
runner = CollectRunner(
session_factory=factory,
harness_adapter=MCPHarnessAdapter(),
serializer=RolloutSerializer("/tmp/ttt-sft-v1"),
limits=HarnessRunLimits(max_turns=9),
)
result = runner.run(
model_step=build_model_step(llm),
num_episodes=200,
should_keep=lambda r: r.reward >= 0.0,
)
push_to_hf_hub("/tmp/ttt-sft-v1", "<user>/ttt-sft-v1")
```
## Output schema
`results.jsonl` — one JSON object per episode:
| field | type | description |
| --- | --- | --- |
| `episode_id` | string | Stable id for resume (`<prefix>-<index>`) |
| `messages` | list | Chat transcript (user/assistant/tool). TRL `SFTTrainer`-compatible |
| `reward` | float | Env reward (derived via `_resolve_env_reward`, never synthesized) |
| `done` | bool | Whether the episode terminated |
| `tool_trace` | list | Structured tool calls with args & results |
| `metrics` | struct | Harness-level metrics (turns, tool_calls) |
| `verify_metrics` | struct | Session-level metrics (step_count, …) |
| `artifacts` | struct | Verification artifacts (final state, etc.) |
| `task` | any | Optional task spec passed by `CollectRunner.tasks` |
| `extra` | struct | Free-form annotations supplied by the caller |
`metadata.json` — run-level info (env, model, n, temperature, filter flags).
`README.md` is auto-generated by `push_to_hf_hub` with YAML front-matter so the
HF Dataset Viewer renders `results.jsonl` as `split=train`.
## Design notes
- **Thin envs** — `OpenSpielSessionFactory` and friends live in the env package
(`envs/<env>/harness.py`) and wrap the existing `EnvClient`. The collector
treats the env as a black box.
- **Rewards in env** — `EpisodeRecord.from_rollout` reuses the runtime's
`_resolve_env_reward` so any mismatch between the reward emitted by a tool
result and the one forwarded by `session.verify()` raises immediately.
- **Provider-agnostic teacher** — `build_model_step` adapts any
`openenv.core.llm_client.LLMClient` (OpenAI, Anthropic, OpenAI-compatible
self-hosted endpoints) into a `ModelStep`. No schema code per provider.
- **Resume** — `RolloutSerializer.collected_episode_ids()` reads prior runs
from disk; `CollectRunner.run(resume=True)` skips ids already serialized.
- **Filtering** — `should_keep` is a `Callable[[EpisodeRecord], bool]`. Defaults
to keeping everything. The CLI filters `reward >= 0` unless `--keep-losses`
is set.
## Training captures
A loop-owning session can implement `TrainableSession.fetch_training_trace()` to
return a `TrainingTrace`. Harbor implements this API. Raw
`fetch_proxy_trace()` remains available for diagnostics and older consumers.
```python
from openenv.core.harness import TrainingTrace, TrainingTurn
trace = TrainingTrace(turns=[TrainingTurn(
node_id="call-0",
prompt_token_ids=[10, 11],
completion_token_ids=[20, 21, 22],
per_token_logps=[-0.2, -0.4, -0.1],
loss_mask=[0, 0, 1, 0, 1],
)])
restored = TrainingTrace.model_validate_json(trace.model_dump_json())
```
The mask covers prompt plus completion. Prompt positions are zero; completion
positions select the sampled tokens that receive loss. A zero-masked turn stays
available as context. Tokens, masks and finite nonpositive logprobs are validated
when the object is built or deserialized. Missing masks and duplicate call IDs
are errors. Tokens are never reconstructed from text.
The environment selects agent calls and excludes auxiliary calls and discarded
retries before returning this object. Shared graph nodes appear once. Selection
uses the existing capture classifier; its tool-manifest and retry heuristics are
not a substitute for explicit harness evidence. The raw graph remains available
for auditing. Changing a task's supervision means changing its producer mask,
not adding a turn-selection callback in the trainer.
`verify()` remains the source of the task reward. `None` is ungraded and `0` is an
incorrect answer. A single-call rollout produces a diagnostic warning, not a
capture failure. Masks do not alter the verifier score. Trainers own advantages,
loss weighting, packing and weight updates. Graph identity does not by itself
normalize a rollout that expands into multiple training rows.
For training, construct the session factory with the trainer's `sampling` policy.
Harbor applies it before inference, so the trainer need not compare sampling
metadata after generation. When logprob entries identify their tokens
(`token_id:N`), training export checks that each one pairs with its token. A disagreement on an agent turn rejects the training trace; raw
capture stays available for inspection. Plain token text is not
re-tokenized to infer an ID.
Migration from TRL selection hooks: use `fetch_training_trace()` instead of a raw
trace, move whole-turn exclusions into the producer's `loss_mask`, and keep final
answers eligible unless your task explicitly masks them. The old Harbor export
continues to read legacy captures; the strict API requires explicit masks and
node IDs. Evaluation-only captures cannot be used for training.
|