Hugging Face's logo

Spaces:
Madhav189
/
multi-agent-env
Sleeping

App Files Community
multi-agent-env / execution.md
Madhav189's picture
Madhav189
Sync council demo and docs
4455094
preview code
|
Raw
History Blame Contribute Delete
9.76 kB

SPECTRA Execution Guide

This is the concrete runbook for the current repo. It matches the real Makefile, demo.py, inference.py, and test status in /Users/madhav_189/Documents/Scalar_hackathon/Madhav_task.

Project root:

cd /Users/madhav_189/Documents/Scalar_hackathon/Madhav_task

1. Python Environment

The repo supports two Python paths.

Option A: create a local .venv 

make setup-venv

This creates:

/Users/madhav_189/Documents/Scalar_hackathon/Madhav_task/.venv

Option B: reuse the older shared environment

If there is no local .venv, the Makefile falls back to:

/Users/madhav_189/Documents/Scalar_hackathon/openenv/.venv/bin/python

Check what the repo is using 

make doctor
make which-python

2. Optional .env

The repo auto-loads a local .env file if it exists.

Create it from the template:

cp .env.example .env

Most useful keys:

  • HF_TOKEN
  • INFRA_SPECIALIST_MODEL
  • INFRA_SPECIALIST_PROVIDER
  • LOG_SPECIALIST_MODEL
  • LOG_SPECIALIST_PROVIDER
  • SEC_SPECIALIST_MODEL
  • SEC_SPECIALIST_PROVIDER
  • COMMANDER_PROVIDER
  • COMMANDER_MODEL
  • COMMANDER_HF_PROVIDER
  • API_BASE_URL
  • OPENAI_API_KEY

Important notes:

  • .env is ignored by git
  • hosted specialist runs need HF_TOKEN
  • local commander runs use an OpenAI-compatible endpoint, usually Ollama at http://127.0.0.1:11434/v1
  • the repo defaults to SPECIALIST_MODE=hybrid for hosted multi-agent runs and LOCAL_MULTI_SPECIALIST_MODE=deterministic for local smoke runs

3. First 5 Minutes

For a first-time user:

make doctor
./.venv/bin/python -m pytest tests -q
make scenarios
make demo SCENARIO=broken_auth_cascade

That verifies:

  • Python environment selection
  • test suite health
  • scenario catalog
  • council-style terminal experience

Current local test result:

  • 23 passed

4. The Main User Flows

Think of SPECTRA in five practical flows:

  1. make demo: single-scenario council story
  2. make council-local: all-scenario local council pipeline
  3. make multi-agent-local-smoke or make multi-agent-smoke: trace, dataset, and hint-pack collection
  4. make hinted and make hinted-check: blind versus hinted replay
  5. make train-smoke or make train: replay-backed GRPO workflow

If you only remember one sequence, use this:

make doctor
./.venv/bin/python -m pytest tests -q
make demo

5. Demo Experience

Single-scenario money demo 

make demo SCENARIO=broken_auth_cascade

This runs the three-phase story for one scenario:

  1. blind commander
  2. multi-agent council
  3. hinted commander

Default output root:

  • outputs/council_pipeline/single_broken_auth_cascade/

Convenience shortcuts:

make demo-easy
make demo-medium
make demo-hard
make demo-cache

Full local council pipeline 

make council-local

Default behavior:

  • COUNCIL_SCOPE=all
  • phases: untrained,multi_agent,hinted
  • local commander model: qwen2.5:3b
  • local multi-agent model: qwen2.5:3b
  • specialist mode: deterministic

Default output root:

  • outputs/council_pipeline/

Run only one scenario in the council UI 

make council-local COUNCIL_SCOPE=single SCENARIO=broken_auth_cascade

Run only selected phases 

make council-local COUNCIL_PHASES=untrained
make council-local COUNCIL_PHASES=multi_agent
make council-local COUNCIL_PHASES=hinted

Hosted council variant 

make council-hf

This keeps the untrained and hinted phases local, but switches the multi-agent council phase to the hosted HF commander stack.

6. Single-Agent Baselines

Blind local commander

This is the clean baseline: one full-state commander with no hint help.

Before running it, make sure your local OpenAI-compatible endpoint exists.

For Ollama:

ollama serve
ollama list

Run the blind baseline:

make untrained SCENARIO=broken_auth_cascade LOCAL_MODEL=qwen2.5:1.5b

Default output folder:

  • outputs/untrained/

Direct single-agent freeform inference 

make local-free SCENARIO=broken_auth_cascade LOCAL_MODEL=qwen2.5:1.5b

This goes through inference.py in single_agent mode without the council wrapper.

7. Multi-Agent Collection

This is the core benchmark artifact flow. One run can export:

  • step-level JSONL data
  • episode summary JSON
  • raw traces
  • a trace-derived hint pack

Local no-HF smoke run 

make multi-agent-local-smoke SCENARIO=broken_auth_cascade

Default local settings:

  • commander provider: openai
  • commander model: qwen2.5:3b
  • specialist mode: deterministic
  • output dir: outputs/multi_agent/

Hosted or hybrid smoke run 

make multi-agent-smoke SCENARIO=broken_auth_cascade COMMANDER_MODEL=Qwen/Qwen3-4B-Instruct-2507

Default hosted settings:

  • commander provider: hf
  • commander model: Qwen/Qwen3-4B-Instruct-2507
  • commander HF provider: nscale
  • specialist mode: hybrid

Larger collection run 

make multi-agent EPISODES=5 COMMANDER_MODEL=Qwen/Qwen3-4B-Instruct-2507

Real hosted specialist collection 

make real-collect-smoke
make real-collect EPISODES=5

These force SPECIALIST_MODE=llm.

Local model matrix 

make multi-agent-local-05b-smoke SCENARIO=broken_auth_cascade
make multi-agent-local-15b-smoke SCENARIO=broken_auth_cascade
make multi-agent-local-3b-smoke SCENARIO=broken_auth_cascade
make multi-agent-local-matrix SCENARIO=broken_auth_cascade

Artifacts to inspect

After a multi-agent run, look at:

  • outputs/multi_agent/data.jsonl
  • outputs/multi_agent/data.summary.json
  • outputs/multi_agent/traces/
  • outputs/multi_agent/hints.json

Specialist mode guidance

  • deterministic: cheapest and most reproducible
  • hybrid: hosted first, deterministic fallback on failure
  • llm: strict hosted specialists only

Use hybrid when you want the pipeline to keep moving even if provider calls are brittle.

8. Hinted Replay

This re-runs a full-state local commander with the exported hint pack.

Run the hinted commander 

make hinted SCENARIO=broken_auth_cascade LOCAL_MODEL=qwen2.5:1.5b

Default output folder:

  • outputs/hinted/

Compare blind versus hinted 

make hinted-check SCENARIO=broken_auth_cascade

Alias:

make hint-effect SCENARIO=broken_auth_cascade

Expected output folder:

  • outputs/hint_effect/

This is the simplest way to test whether the trace-derived cheat sheet actually improves the local commander.

9. Hint Building And Dataset Smoke Tests

Build a hint pack directly from deterministic heuristic collection:

make build-hints EPISODES=5

Minimal dataset smoke test:

make dataset-smoke

This is useful when you want to validate artifact wiring without burning hosted credits.

10. GRPO Workflow

Smoke test the training loop 

make train-smoke

This does two things:

  1. collects a tiny deterministic dataset
  2. dry-runs training/grpo_train.py against it

Dry-run training on an existing dataset 

make grpo-dry-run DATASET_PATH=outputs/multi_agent/data.jsonl

Real training run 

make train DATASET_PATH=outputs/multi_agent/data.jsonl TRAIN_STEPS=20

Training output root:

  • outputs/grpo_runs/

11. Direct Inference Commands

Single scenario:

make inference SCENARIO=database_sqli_outage

Pretty terminal rendering:

make inference-pretty SCENARIO=broken_auth_cascade

All scenarios:

make inference-all

The CLI itself supports:

./.venv/bin/python inference.py --help

Main choices:

  • runtime: local or remote
  • observation mode: multi_agent or single_agent
  • specialist mode: deterministic, hybrid, or llm
  • commander type: heuristic, random, llm, or single-agent

12. API Server And Remote Mode

Start the local server

Terminal 1:

make server

For auto-reload:

make dev

Run remote inference against the server

Terminal 2:

make remote REMOTE_SCENARIO=database_sqli_outage
make remote-pretty REMOTE_SCENARIO=database_sqli_outage

Default base URL:

  • http://127.0.0.1:8000

The top-level server wrapper is server/app.py, and the real FastAPI/OpenEnv implementation is in environments/pomir_env/server.py.

13. Docker Flow

Build the image:

make docker-build

Run it:

make docker-run

The Docker entrypoint serves:

  • uvicorn server.app:app --host 0.0.0.0 --port 8000

14. Validation Commands

Tests 

./.venv/bin/python -m pytest tests -q

Current local result:

  • 23 passed

OpenEnv validation 

./.venv/bin/openenv validate .

Current local result:

  • not yet ready for multi-mode deployment because uv.lock is missing

The repo no longer fails validation on the server wrapper shape:

  • server = "server.app:main" exists in pyproject.toml
  • server/app.py exposes a callable main()

To clear the remaining issue:

uv lock

If uv lock needs network access in your environment, run it outside the restricted sandbox and then re-run:

./.venv/bin/openenv validate .

15. Recommended Paths

If you want the judge-facing story:

make demo

If you want the full local product flow:

make council-local

If you want benchmark artifacts without HF credits:

make multi-agent-local-smoke

If you want the strongest hosted multi-agent collection:

make multi-agent-smoke COMMANDER_MODEL=Qwen/Qwen3-4B-Instruct-2507

If you want the shortest training sanity check:

make train-smoke