Hugging Face's logo

Spaces:
huggingmenfordays
/
sysadmin-env
Paused

App Files Community
sysadmin-env / README.md
huggingmenfordays's picture
huggingmenfordays
Upload folder using huggingface_hub
0ef174b verified
preview code
|
raw
history blame contribute delete
40.3 kB
metadata 
title: sysadmin env
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
tags:
  - openenv
base_path: /web

sysadmin-env

sysadmin-env is an openenv-style benchmark environment for openenv round 1: an agent connects to a live linux-like runtime, inspects a broken machine, issues one shell command at a time, receives stepwise observations and shaped rewards, and is judged on whether it restores the service safely and efficiently.

this repository is intentionally built around the round 1 submission contract:

  • a docker-deployable server with /health, /reset, /step, /state, /tasks, and /ws
  • a baseline agent entrypoint at inference.py
  • deterministic task definitions and graders under sysadmin_env/tasks/
  • structured reward shaping in sysadmin_env/rewards.py
  • openenv packaging shims at the repository root such as client.py, models.py, and __init__.py
  • deployment metadata in openenv.yaml, Dockerfile, server/Dockerfile, and pyproject.toml

the benchmark focuses on linux remediation rather than toy puzzle solving. the agent is not selecting from a fixed action list: it must decide which shell command to run, interpret command output, repair the underlying fault, and stop before wasting steps.

table of contents

why linux remediation is a meaningful benchmark

linux incident response is one of the few domains where agentic reasoning is both measurable and genuinely useful.

real operators routinely need to:

  • inspect logs and process state
  • debug a service that no longer starts
  • find why a filesystem is full
  • repair routes or dns inside a constrained runtime
  • avoid dangerous commands while working under time pressure

that makes remediation a strong benchmark for agent systems:

  1. the action space is realistic. the agent must generate shell commands, not pick from synthetic labels.
  2. observations are partially revealing. one command rarely solves the task; diagnosis matters.
  3. there is a safety dimension. destructive commands should be heavily penalized.
  4. partial progress is meaningful. fixing one component of a broken system should be worth something even before full recovery.
  5. success is operationally grounded. the grader checks system state, not just text output matching.

for round 1, this repository therefore benchmarks the full remediation loop: diagnose, repair, validate, and finish.

round 1 requirement mapping

the table below maps the repository to the practical requirements of the round 1 problem statement.

round 1 concern implementation in this repository
deployable environment server FastAPI app in sysadmin_env/server.py, cli wrapper in server/app.py, docker entrypoints in Dockerfile and server/Dockerfile
standard episode api POST /reset, POST /step, GET /state, GET /health, GET /tasks, WS /ws
deterministic tasks three fixed task modules in sysadmin_env/tasks/nginx_crash.py, sysadmin_env/tasks/disk_full.py, and sysadmin_env/tasks/network_broken.py
real command execution bubblewrap-based sandbox in sysadmin_env/sandbox.py with mutable task state layered over prepared filesystems
reward shaping RewardEngine in sysadmin_env/rewards.py combines health deltas, one-time diagnostic rewards, and penalties
agent entrypoint inference.py loads env vars, queries /tasks, connects to /ws, emits [START], [STEP], and [END] logs
packaging for openenv root shim files client.py, models.py, __init__.py, plus openenv.yaml and mirrored docker assets
validation path openenv validate, docker build, http health/reset probes, and scripts/validate-submission.sh (taken direclty from meta scaler website)

high-level architecture

at runtime the system looks like this:

  1. the server builds a task registry from sysadmin_env/tasks/.
  2. a client resets an episode by task id or lets the server choose the next task in round-robin order.
  3. the selected task prepares a deterministic lower filesystem.
  4. Sandbox creates an isolated execution root using OverlayFSManager.
  5. the client sends a shell command.
  6. the sandbox runs that command via bwrap under /bin/sh -c ....
  7. the task module updates any derived runtime state via observe_command() and synchronize().
  8. RewardEngine grades the resulting filesystem state and computes the per-step reward.
  9. the server returns an Observation and EnvironmentState.

that design splits the benchmark into clear responsibilities:

  • sysadmin_env/tasks/*.py: deterministic problem definitions and grading rules
  • sysadmin_env/sandbox.py: command execution and runtime isolation
  • sysadmin_env/overlayfs.py: resettable mutable filesystem layer
  • sysadmin_env/rewards.py: task-agnostic reward shaping and catastrophic command handling
  • sysadmin_env/server.py: http api, websocket flow, episode lifecycle, and web shim routes
  • inference.py: baseline agent and score logging

repository layout and file roles

the repository keeps the implementation under sysadmin_env/ and exposes a few required root-level shims for packaging workflows.

.
├── .env.example
├── README.md
├── messing-around-with-playbooks.md
├── __init__.py
├── client.py
├── Dockerfile
├── inference.py
├── models.py
├── openenv.yaml
├── pyproject.toml
├── requirements.txt
├── outputs/
│   └── output-*.txt
├── scripts/
│   └── validate-submission.sh
├── server/
│   ├── __init__.py
│   ├── app.py
│   └── Dockerfile
└── sysadmin_env/
    ├── __init__.py
    ├── models.py
    ├── overlayfs.py
    ├── rewards.py
    ├── sandbox.py
    ├── server.py
    └── tasks/
        ├── __init__.py
        ├── disk_full.py
        ├── network_broken.py
        └── nginx_crash.py

core package files under sysadmin_env/

  • sysadmin_env/server.py — main environment implementation. it defines EpisodeManager, http routes, websocket handling, per-step observation building, and the lightweight /web* shim endpoints.
  • sysadmin_env/sandbox.py — the execution sandbox. it uses bubblewrap (bwrap) to run commands in an isolated root, binds selected host binaries read-only, optionally unshares networking, and tracks command results.
  • sysadmin_env/overlayfs.py — mutable episode filesystem manager. it tries kernel overlayfs first, then fuse-overlayfs, then falls back to a plain directory copy strategy when overlay mounts are unavailable.
  • sysadmin_env/rewards.py — reward shaping engine shared across tasks. it applies per-step penalties, one-time diagnostic bonuses, health deltas from task graders, and catastrophic command penalties.
  • sysadmin_env/models.py — pydantic models for actions, observations, state, reset/step payloads, reward signals, task metadata, and grader state.
  • sysadmin_env/tasks/__init__.py — task registry assembly and module lookup.
  • sysadmin_env/tasks/nginx_crash.py — easy service-recovery task.
  • sysadmin_env/tasks/disk_full.py — medium disk-diagnosis/remediation task.
  • sysadmin_env/tasks/network_broken.py — hard routing-and-dns task with network isolation enabled.

root shims and openenv-facing files

  • client.py — thin root shim that re-exports main from inference.py. this keeps the repository shape friendly to packaging and submission tooling.
  • models.py — thin root shim that re-exports the canonical pydantic models from sysadmin_env.models.
  • __init__.py — root package shim that re-exports main, Action, Observation, and EnvironmentState.
  • inference.py — the baseline agent used as the submission entrypoint declared in openenv.yaml.
  • README.md — primary repository documentation covering architecture, tasks, reward shaping, setup, validation, and the current baseline behavior.
  • .env.example — sample environment-variable file for local configuration.
  • messing-around-with-playbooks.md — change log for the recent baseline prompt and network_broken guardrail adjustments, including observed local run results.
  • outputs/ — local captured baseline run logs used while tuning and validating the inference behavior.

deployment, packaging, and validation files

  • Dockerfile — primary container build for local docker runs and hugging face docker spaces.
  • server/Dockerfile — mirrored server build asset kept alongside server/app.py for openenv repository structure checks.
  • server/app.py — asgi/cli launcher that imports app from sysadmin_env.server and exposes the server console script.
  • openenv.yaml — openenv manifest: runtime entrypoints, endpoints, resources, and task metadata.
  • pyproject.toml — canonical packaging metadata, dependencies, python version bounds, and the server = "server.app:main" console script.
  • requirements.txt — mirrored runtime dependency list.
  • scripts/validate-submission.sh — local pre-submission validator that checks the live space, docker buildability, and openenv validate.

runtime model: actions, observations, state, and episode boundaries

the environment is turn-based. every turn consists of one shell command.

action model

the canonical action model is defined in sysadmin_env/models.py:

{
  "command": "string, min length 1",
  "reasoning": "string or null"
}
  • command is the single shell command executed with /bin/sh -c inside the sandbox.
  • reasoning is optional metadata for clients and logs. the server does not grade it.

for the http step route, the action is wrapped inside StepRequest:

{
  "action": {
    "command": "echo hello",
    "reasoning": null
  }
}

observation model

each step returns an Observation:

{
  "stdout": "string",
  "stderr": "string",
  "exit_code": 0,
  "working_directory": "/",
  "execution_time": 0.01,
  "reward": 0.0,
  "done": false,
  "step_number": 1,
  "max_steps": 40
}

important details:

  • reward is the reward for that step only, not a cumulative return.
  • done becomes true when the task grader declares success, a catastrophic action is detected, or the episode hits max_steps.
  • working_directory is / from the sandbox’s point of view.
  • if a command times out, the server appends command execution timed out to stderr.

state model

GET /state returns EnvironmentState:

{
  "episode_id": "string",
  "task_id": "nginx_crash",
  "step_count": 1,
  "max_steps": 40,
  "done": false,
  "reward": 0.0
}

again, reward here is the last step reward, mirroring the latest observation.

reset and task selection

POST /reset optionally accepts a task_id:

{
  "task_id": "disk_full"
}

if task_id is omitted, EpisodeManager selects the next task in round-robin registry order. in this repository that order is the registry insertion order:

  1. nginx_crash
  2. disk_full
  3. network_broken

episode boundaries

for an episode with step index t, the server marks the observation done when:

  • the task grader returns done = true, or
  • the reward engine flags the action as catastrophic, or
  • t >= max_steps

on the http path, when an episode ends the current sandbox is cleaned up immediately. the last state remains queryable through GET /state, but another POST /step requires a new POST /reset.

api reference

http routes

GET /health

health probe for validators and deployment smoke tests.

{"status": "ok"}

GET /tasks

returns the available task metadata that clients can iterate over.

{
  "tasks": [
    {
      "task_id": "nginx_crash",
      "difficulty": "easy",
      "description": "nginx crashed with stale pid and config syntax error",
      "max_steps": 40,
      "time_limit": 300.0
    }
  ]
}

POST /reset

starts a new episode and returns a StepResult consisting of:

  • an initial zero-reward observation at step_number = 0
  • the environment state with a fresh episode_id

POST /step

executes one action inside the active episode sandbox and returns:

{
  "observation": {
    "stdout": "...",
    "stderr": "...",
    "exit_code": 0,
    "working_directory": "/",
    "execution_time": 0.02,
    "reward": 0.07,
    "done": false,
    "step_number": 1,
    "max_steps": 40
  },
  "state": {
    "episode_id": "...",
    "task_id": "nginx_crash",
    "step_count": 1,
    "max_steps": 40,
    "done": false,
    "reward": 0.07
  }
}

if no episode has been initialized, the route returns http 409.

GET /state

returns the latest EnvironmentState. if no episode has been initialized yet, the route returns http 404.

websocket flow: WS /ws

the websocket route is the main agent interface used by inference.py.

connection behavior:

  1. connect to /ws or /ws?task_id=<task>.
  2. the server immediately starts an episode.
  3. the first message is:
{
  "type": "episode_started",
  "task": {
    "task_id": "network_broken",
    "difficulty": "hard",
    "description": "broken network namespace with corrupted routing and dns",
    "max_steps": 70,
    "time_limit": 480.0
  }
}
  1. the client sends raw Action json, not a StepRequest wrapper:
{
  "command": "ip route show",
  "reasoning": "inspect the default route"
}
  1. the server replies with observation messages:
{
  "type": "observation",
  "task_id": "network_broken",
  "observation": {
    "stdout": "default via 192.0.2.1 dev eth9\n",
    "stderr": "",
    "exit_code": 0,
    "working_directory": "/",
    "execution_time": 0.01,
    "reward": 0.06,
    "done": false,
    "step_number": 1,
    "max_steps": 70
  }
}

malformed or empty actions yield error messages such as:

{
  "type": "error",
  "code": "invalid_action",
  "message": "malformed action json"
}

once done becomes true, the server cleans up the sandbox and closes the episode loop for that websocket connection.

web shim routes

the server also exposes lightweight web shim routes intended for space uis and openenv web probing:

  • GET /web
  • GET /web/metadata
  • POST /web/reset
  • POST /web/step
  • GET /web/state

these routes do not replace the canonical http api; they wrap it.

useful details:

  • GET /web/metadata returns the benchmark name, a short description, a /docs url, and the contents of README.md.
  • POST /web/reset returns a json object with top-level observation, reward, done, and state fields.
  • POST /web/step accepts either:
    • {"action": {"command": "...", "reasoning": null}}, or
    • {"command": "...", "reasoning": null}
  • GET /web/state returns an initialized flag and null fields before the first reset.

sandbox and filesystem model

each task is defined as a prepared lower filesystem plus a mutable episode runtime.

Sandbox in sysadmin_env/sandbox.py:

  • verifies that bwrap is available
  • creates a writable overlay-backed runtime root
  • binds selected host binaries read-only into the sandbox
  • clears the environment and sets a small deterministic PATH
  • runs as uid 0 and gid 0
  • drops all linux capabilities
  • optionally unshares networking for tasks that require isolation

task modules write stub binaries into the lower filesystem, such as nginx, df, du, ip, ping, service, and systemctl. this gives the benchmark realistic command semantics while keeping the task fully deterministic and cheap to reset.

task suite

there are exactly three tasks, with increasing difficulty and fixed metadata also mirrored in openenv.yaml.

task difficulty max steps time limit objective
nginx_crash easy 40 300 s restore a broken nginx service with config and pid issues
disk_full medium 55 420 s identify and neutralize the hidden file exhausting /mnt/data
network_broken hard 70 480 s repair routing and dns so outbound connectivity is restored

determinism guarantees across tasks

all three tasks are deterministic in the current codebase:

  • the prepared filesystem contents are fixed
  • grader logic is pure filesystem-state inspection
  • diagnostic triggers are fixed regular-expression matches over commands
  • there is no random task generation, no stochastic log output, and no nondeterministic reward noise

the only source of behavioral variation is the agent’s command sequence.

task 1: nginx_crash

what is broken

  • /etc/nginx/nginx.conf is missing the semicolon after listen 8080
  • /var/run/nginx.pid contains a stale pid (424242)
  • /var/log/nginx/error.log contains the parse error text
  • the provided stub nginx binary refuses to start while the stale pid is present or the config is still broken

relevant task-local command stubs

  • nginx
  • curl
  • ps
  • pgrep
  • service
  • systemctl

difficulty progression

this is the easiest task because the failure is local to one service and the remediation path is short:

  1. inspect logs or config
  2. clear or repair the pid/config problem
  3. start nginx
  4. optionally verify with curl, service nginx status, or systemctl status nginx

grader behavior

the task health is:

H_nginx = 0.25 * I_stale_pid_removed
        + 0.35 * I_config_fixed
        + 0.40 * I_service_running

where:

  • I_stale_pid_removed = 1 if /var/run/nginx.pid is missing or contains 1234
  • I_config_fixed = 1 if the config contains listen 8080;
  • I_service_running = 1 if the config is fixed and /run/nginx.running says running

the episode ends successfully when I_service_running = 1.

diagnostic rewards

  • checking error.log: +0.05
  • running nginx -t: +0.08
  • reading the pid file: +0.04
  • checking process state via ps or pgrep: +0.04

these rewards are one-time only per episode.

task 2: disk_full

what is broken

  • the simulated mount is /mnt/data
  • capacity is fixed at 100
  • the hidden file /mnt/data/.cache/.rotated/app.trace is written with length 100
  • that makes used space equal capacity, so available space is 0

relevant task-local command stubs

  • df
  • du
  • lsof

difficulty progression

this task is harder than nginx_crash because the agent must identify where the space went before it can reclaim capacity. the intended trajectory is usually:

  1. establish that the filesystem is full
  2. search or summarize the mount contents
  3. identify the hidden offender
  4. truncate or remove the file
  5. verify free space returned

grader behavior

the task health is:

H_disk = 0.30 * I_filesystem_identified
       + 0.30 * I_hidden_file_found
       + 0.40 * I_capacity_free

where:

  • I_filesystem_identified = 1 once the task records diagnosis state full or found
  • I_hidden_file_found = 1 once the hidden file has either been removed/truncated away from existence or the discovery state is found
  • I_capacity_free = 1 if free capacity is greater than 0

the task uses .capacity, .usage, and .diagnosed files under /mnt/data to make the state explicit and deterministic.

the episode ends successfully when I_capacity_free = 1.

diagnostic rewards

  • df / df -h: +0.06
  • du: +0.05
  • find ... -type f or find ... -name: +0.06
  • lsof: +0.05

what counts as a repair

any non-catastrophic change that leaves the filesystem with available capacity works. for example, truncating or deleting the hidden file both satisfy the implemented grader.

task 3: network_broken

what is broken

  • /etc/network/routes/default starts as default via 192.0.2.1 dev eth9
  • /etc/resolv.conf starts as nameserver 0.0.0.0
  • eth0 itself is up and already has 10.0.2.15/24
  • the task definition sets requires_network_isolation = True, so the sandbox unshares networking

relevant task-local command stubs

  • ip
  • route
  • ping

difficulty progression

this is the hardest task because the agent must reason about multiple networking layers:

  1. inspect the route table
  2. inspect interface state and addresses
  3. inspect dns resolver configuration
  4. repair the default route
  5. repair resolv.conf
  6. validate connectivity

grader behavior

the task health is:

H_net = 0.20 * I_routing_issue_diagnosed
      + 0.30 * I_default_route_restored
      + 0.20 * I_dns_resolution_restored
      + 0.30 * I_outbound_connectivity_restored

where:

  • I_default_route_restored = 1 iff /etc/network/routes/default exactly equals default via 10.0.2.2 dev eth0\n
  • I_dns_resolution_restored = 1 iff /etc/resolv.conf exactly equals nameserver 1.1.1.1\n
  • I_outbound_connectivity_restored = 1 iff both fixes above are in place and the link state file still says up
  • I_routing_issue_diagnosed = 1 iff the route has already been fixed or the task’s network.ping flag has been marked diagnosed

the episode ends successfully when I_outbound_connectivity_restored = 1.

notably, the grader does not require an actual successful ping command after repair; success is determined from the repaired state files. a ping is still useful as evidence for the agent.

diagnostic rewards

  • ip route show or route -n: +0.07
  • ip addr or ifconfig: +0.05
  • ip link or ethtool: +0.05
  • ping or curl: +0.06
  • reading resolv.conf: +0.05

reward and scoring system

this section is based on the actual implementation in sysadmin_env/rewards.py, the per-task grade() functions, and the task summary logic in inference.py.

step reward formula

let:

  • H_t = task health after step t, as returned by the task module’s grade() function
  • H_(t-1) = health before the current step
  • K_t = one-time diagnostic reward earned on step t
  • P_step = -0.01

then for a normal, non-catastrophic action:

r_t = (H_t - H_(t-1)) + K_t + P_step

equivalently:

r_t = health_delta + knowledge_delta - 0.01

where:

  • health_delta = H_t - H_(t-1)
  • knowledge_delta = sum of newly unlocked diagnostic trigger rewards on this step

the reward engine stores known_fact_ids, so a diagnostic trigger only pays once. repeating the same diagnostic command later gives no extra knowledge reward.

catastrophic action penalty

if the command string matches one of the destructive regex patterns, the reward engine ignores any positive progress from that action and instead returns:

r_t = -1.0

and marks the episode done.

the default catastrophic patterns include commands matching behaviors such as:

  • rm -rf /
  • mkfs
  • shutdown, reboot, halt
  • kill 1 or kill -9 1
  • destructive dd/truncate writes targeting /etc or /boot
  • a shell fork bomb pattern

matching is regex-based and case-insensitive.

partial progress and telescoping health

because each task health is defined on [0, 1], cumulative health gain over an episode telescopes:

sum_t (H_t - H_(t-1)) = H_final - H_initial

all three tasks begin with H_initial = 0.0, so if the agent fully solves a task without catastrophic failure:

sum_t health_delta = 1.0

this is why task-specific partial repairs directly appear in reward:

  • removing only the stale nginx pid is worth +0.25 health before the step penalty
  • identifying the full disk is worth +0.30 health before the step penalty
  • fixing only the network route is worth +0.30 health before the step penalty

one-time knowledge rewards by task

the maximum knowledge reward available per task is:

task knowledge trigger sum
nginx_crash 0.05 + 0.08 + 0.04 + 0.04 = 0.21
disk_full 0.06 + 0.05 + 0.06 + 0.05 = 0.22
network_broken 0.07 + 0.05 + 0.05 + 0.06 + 0.05 = 0.28

so the maximum raw trajectory return before step penalties is:

1.0 + knowledge_sum

which is:

  • 1.21 for nginx_crash
  • 1.22 for disk_full
  • 1.28 for network_broken

after n non-catastrophic steps, the raw return becomes:

R_raw = H_final + K_total - 0.01 * n

for the common non-catastrophic case.

examples

example: useful diagnosis but no repair

if the agent runs nginx -t as the first command in nginx_crash, the command reveals the config fact and changes no system health:

health_delta = 0.00
knowledge_delta = 0.08
reward = 0.00 + 0.08 - 0.01 = 0.07

example: partial repair

if the agent removes the stale pid in nginx_crash and nothing else changes:

health_delta = 0.25
knowledge_delta = 0.00
reward = 0.25 - 0.01 = 0.24

example: repeated diagnosis

if the agent runs the same rewarded diagnostic command twice, the second step yields no extra knowledge reward:

reward_repeat = health_delta + 0.00 - 0.01

if no repair happened either, that means reward_repeat = -0.01.

how the inference script turns trajectory rewards into a reported score

inference.py accumulates the per-step rewards it receives from websocket observations:

R_episode = sum_t r_t

it then reports the task score as:

score = clamp(R_episode, 0.0, 1.0)

where:

clamp(x, 0, 1) = min(max(x, 0), 1)

important implications:

  1. this is a clamped trajectory sum, not a separate grader-normalized value.
  2. strong trajectories can exceed 1.0 before clamping because they combine full health (1.0) with diagnostic rewards.
  3. wasted steps reduce the score by 0.01 each.
  4. a catastrophic -1.0 step can wipe out prior gains or leave a small residual score if the previous raw total was already above 1.0.

how success is computed in inference.py

the baseline script’s success flag is distinct from the clamped score. on the final observation it computes:

success = (last_step_reward > 0.0) and (step_number < max_steps)

consequences:

  • a task completed with a positive final reward before the step cap is counted as success
  • a run that ends exactly on max_steps is marked unsuccessful by the baseline summary, even if the last action repaired the state
  • the server itself still reports done; this success flag is a client-side summary convention used by inference.py

local setup

the repository is designed around python 3.11 and uv.

recommended setup with uv 

uv python install 3.11
uv sync --python 3.11 --extra dev

if python 3.11 is already available:

uv sync --extra dev

pyproject.toml is the canonical dependency source, and uv.lock pins the resolved environment used by docker builds.

alternative setup with pip 

python -m pip install .
python -m pip install pytest

requirements.txt mirrors the runtime dependency set, but the packaging metadata lives in pyproject.toml.

running the server locally

the canonical launcher is the server console script declared in pyproject.toml and implemented by server/app.py.

uv run server --host 0.0.0.0 --port 8000

useful checks:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/tasks

manual http flow 

curl -X POST http://127.0.0.1:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id":"nginx_crash"}'

curl -X POST http://127.0.0.1:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action":{"command":"cat /var/log/nginx/error.log","reasoning":null}}'

curl http://127.0.0.1:8000/state

inference usage

the baseline agent entrypoint is inference.py.

uv run python inference.py

it will:

  1. probe /health
  2. query /tasks unless SYSADMIN_ENV_TASK_ID is set
  3. connect to /ws?task_id=<task>
  4. choose actions using the openai responses api if credentials exist
  5. fall back to a deterministic heuristic plan otherwise
  6. emit structured stdout logs

the required environment variables are:

HF_TOKEN="your_api_key_here"
MODEL_NAME="gpt-5.4"
API_BASE_URL="https://api.openai.com/v1"
OPENAI_REASONING_EFFORT="medium"
SYSADMIN_ENV_SERVER_URL="ws://127.0.0.1:8000/ws"
SYSADMIN_ENV_HEALTHCHECK_URL="http://127.0.0.1:8000/health"
SYSADMIN_ENV_TASKS_URL="http://127.0.0.1:8000/tasks"
SYSADMIN_ENV_TASK_ID=""
MODEL_API_TIMEOUT_SECONDS="20"
EPISODE_TIMEOUT_SECONDS="600"

notes:

  • API_BASE_URL and MODEL_NAME both have built-in defaults in inference.py.
  • HF_TOKEN is the required submission-facing variable name. in practical terms, the token value must match the provider behind API_BASE_URL: if you point at the hugging face router, use a hugging face token; if you point at another openai-compatible endpoint, use the credential that endpoint expects.
  • the script also accepts OPENAI_API_KEY and API_KEY as compatibility fallbacks for local runs, but the documented submission path should still provide HF_TOKEN.
  • SYSADMIN_ENV_TASK_ID="" means “run all tasks returned by /tasks in order”.
  • API_BASE_URL may point to any openai-compatible endpoint.
  • this baseline talks to the running environment server over http/websocket, so an extra LOCAL_IMAGE_NAME variable is not needed here unless you rewrite the client around a from_docker_image() flow.
  • by default, the script writes the flat submission-oriented [START], [STEP], and [END] records to stdout and diagnostics to stderr.
  • if you need the older json payload logs for local debugging, set SYSADMIN_ENV_LOG_FORMAT=json before running inference.py.

stdout output contract

the default stdout format is the flat key-value format expected by the latest submission notes:

[START] task=<task_name> env=<benchmark> model=<model_name>
[STEP] step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
[END] success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...,rn>

details:

  • score is normalized to stay strictly inside (0, 1) before logging, so boundary values are not emitted in submission summaries
  • reward and each entry in rewards are formatted to exactly two decimal places
  • done and success are lowercase booleans
  • error is null when there is no step error
  • all output stays on a single line per record

baseline behavior and current observations

the current baseline keeps the same high-level contract while tightening how the hard task is handled.

current baseline behavior

  • if HF_TOKEN or another supported api key is present, inference.py uses the openai responses api.
  • if no api key is present or the model call fails, the script falls back to the deterministic task plan described in inference.py.
  • for network_broken, the model prompt now uses a generic task playbook rather than embedding the exact hidden grader targets.
  • after enough route, interface, and dns diagnosis, the baseline applies a state-aware guardrail for network_broken so that unsupported guesses do not loop forever.
  • the guardrail emits concise stderr traces such as network guardrail dns repair and network guardrail route repair, which makes the baseline easier to debug without changing the wire protocol.

why the baseline was adjusted

the earlier prompt variant made network_broken too easy because the model could effectively recover the exact answer from the prompt rather than infer it from the environment. the current prompt removes that leakage and keeps the hard task benchmark-oriented while still allowing a reproducible baseline run.

current observed local baseline run

the latest local run against the repository server with MODEL_NAME="gpt-5.4-nano" produced the following episode summaries:

task success steps score notes
nginx_crash true 6 1.0 fixed config, cleared stale pid, then started nginx
disk_full true 4 1.0 diagnosed the full mount, inspected the hidden trace, then truncated it
network_broken true 7 1.0 gathered route/link/dns evidence first, then the guardrail applied dns repair followed by route repair

this is a current observed baseline, not a theoretical guarantee for every model provider or future model snapshot.

for the full debugging narrative behind those adjustments, see messing-around-with-playbooks.md.

validation flow

there are three useful validation layers.

1. python tests

run the full suite:

uv run pytest -q

for packaging, server-contract, and scoring-focused checks, a narrower command is:

uv run pytest -q tests/test_packaginge.py tests/test_server.py tests/test_rewards.py tests/test_inferenxe.py

for the recent baseline-planner and task-behavior checks used while tuning network_broken, a focused command is:

uv run pytest -q --import-mode=importlib tests/test_inferenxe.py tests/test_tasks.py

2. openenv manifest validation 

openenv validate

this checks the submission structure and endpoint declarations from openenv.yaml.

3. end-to-end submission helper

the repository includes an exact pre-submission helper script:

bash scripts/validate-submission.sh https://your-space.hf.space .

or, from the repository root:

bash scripts/validate-submission.sh https://your-space.hf.space

the script performs four checks in sequence:

  1. GET <space>/health
  2. POST <space>/reset
  3. local docker build
  4. local openenv validate

use the runtime url ending in .hf.space, not the repository page url under huggingface.co/spaces/....

docker and deployment flow

local docker build 

docker build -t sysadmin-env .
docker run --rm -p 18000:8000 sysadmin-env
curl http://127.0.0.1:18000/health
curl http://127.0.0.1:18000/tasks

both Dockerfile and server/Dockerfile:

  • start from python:3.11-slim
  • install bubblewrap, fuse-overlayfs, procps, iputils-ping, findutils, and curl
  • install uv
  • copy pyproject.toml and uv.lock
  • run uv sync --locked --no-dev --no-install-project
  • copy the project files including README.md, root shims, server/, sysadmin_env/, and assets/
  • run uv sync --locked --no-dev --no-editable
  • start the environment with uv run server --host 0.0.0.0 --port 8000

hugging face deployment

the repository is prepared for a hugging face docker space.

key points:

  • the readme front matter declares sdk: docker
  • Dockerfile is suitable for space runtime startup
  • openenv.yaml declares inference.py as the benchmark entrypoint and server.app:app as the server entrypoint
  • the root shims (client.py, models.py, __init__.py) and server/Dockerfile are present because openenv repository checks expect this structure after an openenv init style workflow

typical flow:

  1. build and test locally
  2. run openenv validate
  3. push the repository or space update
  4. wait for the hugging face space to become healthy
  5. run bash scripts/validate-submission.sh https://your-space.hf.space .
  6. run your agent against the live deployment via inference.py

openenv submission commands 

openenv validate
openenv push

this repository keeps the mirrored build assets and root shims needed for that workflow.

mathematical summary of each task’s total raw return

ignoring catastrophic termination, the raw episode return for each task can be written as:

R = H_final + K_total - 0.01 * n

where n is the number of executed steps.

for the fully solved case (H_final = 1.0):

task fully solved raw return
nginx_crash R = 1.0 + K_nginx - 0.01n, where 0 <= K_nginx <= 0.21
disk_full R = 1.0 + K_disk - 0.01n, where 0 <= K_disk <= 0.22
network_broken R = 1.0 + K_net - 0.01n, where 0 <= K_net <= 0.28

the score reported by inference.py is then transformed into an open-interval submission summary value:

score_clamped = min(max(R, 0.0), 1.0)
score_reported = 0.01 + 0.98 * score_clamped

so the benchmark strongly rewards:

  • solving the task at all
  • gathering useful evidence without repeating it
  • reaching the repair quickly
  • avoiding destructive commands entirely

limitations and portability notes

overlay mount constraints on hugging face and other managed runtimes

managed container platforms often restrict privileged mount operations. in practice, hugging face docker spaces may not allow kernel overlay mounts, and some environments may also lack a usable fuse-overlayfs path.

sysadmin_env/overlayfs.py handles this explicitly:

  1. try kernel overlayfs
  2. if that fails, try fuse-overlayfs
  3. if that also fails, use a plain directory copy fallback

the fallback is important because it preserves correctness even when the faster mount strategies are unavailable.

what the copy fallback means

in copy mode:

  • the prepared lower filesystem is copied into the merged runtime directory
  • resets rebuild that merged directory by copying from the lowerdir again
  • the environment remains deterministic and functional
  • resets are typically slower than true overlay copy-on-write resets

this is a deliberate portability tradeoff: the benchmark prefers “runs correctly in restricted environments” over “requires privileged overlay support”.

additional candid limitations

  • the tasks are realistic but still simplified; they use stub executables rather than full linux services.
  • grading is based on explicit filesystem state rather than black-box network/service behavior.
  • the baseline success flag in inference.py is a client summary heuristic, not an authoritative server-side evaluation primitive.
  • the environment currently models exactly three tasks; expanding benchmark breadth would require additional task modules and graders.

practical quickstart

if you just want the shortest useful path:

uv sync --extra dev
uv run server --host 0.0.0.0 --port 8000

in another shell:

uv run python inference.py

before submission:

openenv validate
bash scripts/validate-submission.sh https://your-space.hf.space .

that sequence exercises the main round 1 path from local development to deployment validation.

with love :

hatsune-miku-miku 200w kasane-teto-teto-kasane teto-tetoris 200