Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Black-Box: Train Real Agents with Harbor","local":"black-box-train-real-agents-with-harbor","sections":[{"title":"Try One Rollout","local":"try-one-rollout","sections":[],"depth":2},{"title":"Train on the Captures","local":"train-on-the-captures","sections":[],"depth":2},{"title":"Learn More","local":"learn-more","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/entry/start.V8QGNxBq.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/ZrEo-onz.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/BSuxAqoA.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/entry/app.DO1X4r2J.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/Ds_1Ru_R.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/C61ky1zr.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/BAUKCucN.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/nodes/0.dsTx2rYi.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/CR5HsSXT.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/nodes/67.COVWVdUS.js" rel="modulepreload"> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/chunks/C07d6Hje.js" rel="modulepreload"> | |
| <!--fuejr8--><meta name="hf:doc:metadata" content="{"title":"Black-Box: Train Real Agents with Harbor","local":"black-box-train-real-agents-with-harbor","sections":[{"title":"Try One Rollout","local":"try-one-rollout","sections":[],"depth":2},{"title":"Train on the Captures","local":"train-on-the-captures","sections":[],"depth":2},{"title":"Learn More","local":"learn-more","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/openenv/pr_1333/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="black-box-train-real-agents-with-harbor" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#black-box-train-real-agents-with-harbor"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Black-Box: Train Real Agents with Harbor</span></h1><!--]--><!----> <p>Some agents can’t be reimplemented as a trainer’s tool loop. A coding agent such as OpenCode, Claude Code or Codex has its own planner, tools, context management and stop condition, and the point is to train the model that drives <em>that</em> agent. This is the <strong>black-box</strong> harness path: the agent owns its loop, and OpenEnv records every model call it makes. Episodes are tasks: an instruction goes in, the agent works on its own, and a verifier checks the result. If the job is a conversation instead, or the agent has to use your environment’s own tools, see <a href="claude-code-harness">Evaluate Claude Code in an Environment</a>. <a href="harnesses">Harnesses in OpenEnv</a> compares the paths.</p> <p>OpenEnv does this through <a href="../environments/harbor">Harbor</a>, which supplies the tasks, the sandboxes, the agents and the verifiers. One <code>harbor_env</code> server runs any of 16 validated agents on any Harbor dataset, in any of Harbor’s sandbox backends, and its capture proxy returns the token ids and logprobs of each model call together with the task’s reward.</p> <!--[1--><h2 class="relative group"><a id="try-one-rollout" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#try-one-rollout"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Try One Rollout</span></h2><!--]--><!----> <p>With an OpenAI-compatible endpoint in <code>$LLM</code> and Python 3.12 or later:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-bash "><!---->pip install <span class="hljs-string">"openenv[harbor]"</span> | |
| openenv harbor rollout \ | |
| --llm-url <span class="hljs-variable">$LLM</span> \ | |
| --dataset AdithyaSK/data_agent_rl_environment_eval \ | |
| --task-index 0 --harness opencode --sandbox e2b \ | |
| --out rollout.json<!----></pre></div><!----> <p>Against a hosted provider you get an evaluation rollout: the reward and the full trace. Against vLLM or SGLang you also get the token ids and logprobs that training needs. <code>openenv harbor info</code> checks first which agents, sandboxes and capture level this machine can use.</p> <!--[1--><h2 class="relative group"><a id="train-on-the-captures" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#train-on-the-captures"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Train on the Captures</span></h2><!--]--><!----> <p>Each rollout comes back as a <code>TrainingTrace</code>: the token ids, logprobs and loss masks of every model call, plus the task’s reward. It doesn’t depend on a trainer, so any framework can train on it. TRL has a worked example: <code>AsyncGRPOTrainer</code> with a <code>HarnessRolloutWorker</code> runs each task to completion, reads back the trace, and syncs the new weights into the same vLLM server.</p> <ul><li><a href="https://huggingface.co/docs/trl/main/en/openenv#training-on-harnesses-training-real-coding-agents-harbor" rel="nofollow">TRL’s guide to training on harnesses</a> explains how it works and how to wire it.</li> <li><a href="https://github.com/huggingface/trl/tree/main/examples/async_grpo_harbor" rel="nofollow"><code>examples/async_grpo_harbor</code></a> is the complete script, with a Hugging Face Jobs launcher.</li> <li><a href="https://huggingface.co/spaces/FineEnvs/multi-harness-rl" rel="nofollow">The ultimate guide to multi-harness RL</a> trains one model across four agents (OpenCode, Claude Code, Codex and Mini-SWE-Agent) and covers why the white-box loop isn’t enough, how the capture works and what the runs showed.</li></ul> <!--[1--><h2 class="relative group"><a id="learn-more" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#learn-more"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Learn More</span></h2><!--]--><!----> <ul><li><a href="../environments/harbor">The Harbor environment</a>: supported agents and sandboxes, the web UI, deploying to Spaces and troubleshooting.</li> <li><a href="https://huggingface.co/docs/trl/harbor" rel="nofollow">TRL’s Harbor integration</a> for the white-box path on Harbor tasks, where TRL’s own loop works through a Harbor task suite.</li> <li><a href="../guides/training">Training with OpenEnv</a> for the other ways to train and the frameworks that support them.</li></ul> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/openenv/blob/main/docs/source/tutorials/harbor-harness.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_7covk = { | |
| base: "/docs/openenv/pr_1333/en", | |
| assets: "/docs/openenv/pr_1333/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/openenv/pr_1333/en/_app/immutable/entry/start.V8QGNxBq.js"), | |
| import("/docs/openenv/pr_1333/en/_app/immutable/entry/app.DO1X4r2J.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 67], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 15 kB
- Xet hash:
- be53fec45dd4fec402a4a0644ee38a40c6ad91915cfb10c858c5362284b56272
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.