# autocodabench — User Guide This guide describes how to install, authenticate, and use autocodabench through each of its four surfaces: the command-line interface (CLI), the Python library, an MCP host (Claude Code or Claude Desktop), and the web user interface. Maintainer documentation is provided in [`architecture.md`](./architecture.md). --- ## 1. Installation ```bash pip install -e . # from a checkout (PyPI release pending) ``` Python 3.10 or later is required. The agentic features depend on the Claude Agent SDK, which is installed automatically. The validator and the demo command do not use the SDK and therefore operate in any environment, without a Node runtime, credentials, or network access. To verify the installation: 1. Run `autocodabench --version`. 2. Run `autocodabench demo --out /tmp/acb-demo` to execute a fully offline end-to-end test. ### Docker (required to *run* bundles) autocodabench executes every bundle program — scoring, ingestion, and the starting-kit notebook — inside the competition's Docker image, exactly as the Codabench compute worker does. **Docker must be installed and running** for the run phases: `plan-build-validate`'s build self-validation, and any direct call to the runner. Static `validate` and `demo` do not need it. Both `plan-build-validate` and `validate` open with a **Docker preflight banner** that reports the image that will run, its CPU architecture versus your host (native vs. slow QEMU emulation), and whether the daemon is up — so the runtime is never a surprise. On Apple silicon, prefer the multi-arch `codalab/codalab-legacy:py312` (Docker resolves it to arm64) for fast local testing: ```bash export AUTOCODABENCH_DOCKER_IMAGE=codalab/codalab-legacy:py312 ``` See `docs/post-create-pipeline.md` for exactly what runs after `create` and how to test each step, and `docker/README.md` for the autocodabench base images. --- ## 2. Authentication (agentic features only) Two authentication paths are supported, listed in recommended order. | Path | Intended use | Setup | |---|---|---| | **Claude subscription** (Pro or Max) | Local use; the recommended path for individual users | Install Claude Code; then run `autocodabench auth use subscription` — if no login is found it asks for consent and opens Claude Code's sign-in for you (equivalent to `claude auth login`). Subsequent autocodabench agent sessions draw from the plan's monthly Agent SDK credit. | | **`ANTHROPIC_API_KEY`** | Automation in CI-adjacent environments, users without a subscription, and any hosted multi-user deployment, for which it is required | `export ANTHROPIC_API_KEY=sk-ant-…`, or place the key in a `.env` file (see below). | To inspect or choose the active path: ```bash autocodabench auth status # report, pick, and verify (one live turn) autocodabench auth status --no-probe # report only; no live turn (offline / CI) autocodabench auth use subscription # prefer the subscription, even if a key is set autocodabench auth use api_key # prefer the key (and paste one if none is set) autocodabench auth use auto # default: the key if present, else the subscription ``` Both `auth status` and `auth use ` do more than report: by default they **realize the preference and authenticate the agent SDK with it** — one minimal live turn — and report whether the sign-in actually succeeded. Static detection only confirms that a credential file or variable is present on disk; the probe confirms the credential is accepted. Pass `--no-probe` to skip the live turn (for offline or CI use). ### Choosing between a key and a subscription The Claude Agent SDK prefers an exported `ANTHROPIC_API_KEY` over a stored subscription login. Rather than require you to delete a key to fall back to your plan, autocodabench stores an explicit **preference** and realizes it for you: choosing `subscription` hides any `ANTHROPIC_API_KEY` from the SDK for the run, so the subscription login is the one used — nothing to unset. The preference persists at `~/.config/autocodabench/auth.json` (override per-invocation with `AUTOCODABENCH_AUTH=auto|subscription|api_key`), and is set with `autocodabench auth use ` or the picker that `auth status` shows on a terminal. The default, `auto`, uses a key when one is present and the subscription otherwise. Every command that starts a live model session prints a one-line `INFO:` banner naming the auth in use (API key, subscription, or none) before any tokens are spent. `autocodabench auth status` also reports a masked preview of each configured credential, so you can confirm *which* value is set without revealing it. The `ANTHROPIC_API_KEY` shows its scheme prefix and last four characters; the Codabench publishing credentials (section 7) are listed in a second block, with the password and token masked and the username shown in full. A credential that is absent reads `(not set)`, distinct from `(set but empty)`. One rule still binds: 1. **Hosted multi-user deployments must use an API key.** Under Anthropic's terms of service, requests from other users may not be routed through one person's Free, Pro, or Max credentials. Running the web UI locally for personal use on a personal subscription is permitted; deploying it for other users (for example, as a Hugging Face Space) requires an API key. ### `.env` loading and the authentication preflight The CLI loads `/.env` at startup using a minimal parser that never overrides variables already present in the real environment, so `ANTHROPIC_API_KEY` may live in a `.env` file instead of being exported. You do not have to edit that file by hand: `autocodabench auth use api_key` (and the `auth status` picker) prompt for a key with hidden input and offer to save it to `./.env` (file mode 600) for you. The commands `autocodabench plan-build-validate` and `autocodabench validate --judged` perform an authentication preflight before starting a session. When no credentials are found and the command is running on an interactive terminal, the preflight offers the same two options — signing in to the subscription (after asking for consent, it opens Claude Code's sign-in in place; you may decline and run `claude auth login` yourself), or entering an API key. In non-interactive contexts, these commands exit with status 2 and print guidance instead. Keyless commands (`validate`, `demo`, `checks list`) do not consult authentication state at all. --- ## 3. Validating a bundle (keyless) The validator accepts any Codabench bundle, whether generated by autocodabench or written by hand, supplied as a directory or a zip archive: ```bash autocodabench validate path/to/bundle/ # or bundle.zip autocodabench validate bundle.zip --json # machine-readable report autocodabench checks list # every check, by tier, with citations ``` The report contains four sections with distinct meanings: - **Gate failures** — deterministic checks that block upload, such as missing referenced files, unparseable YAML, or leaderboard keys that the scoring program never writes. These produce exit code 1. - **Findings (advisory)** — design risks with citations into Pavão et al. (2024), *AI Competitions and Benchmarks*; examples include the absence of a daily submission cap, a development phase shorter than 40 days, or a test set too small to satisfy the 100/E rule. - **Attestations required** — launch criteria that only a human can certify, such as completion of external review, publication of a datasheet, or prize legality. The validator surfaces these criteria; it does not claim to have verified them. - **Skipped** — checks that require a declared fact that has not been provided. ### Declared facts (`competition_facts.yaml`) Some checks require context that the bundle cannot carry. Declare this context in a file placed next to `competition.yaml`, or supply it with `--facts path.yaml`: ```yaml anticipated_error_rate: 0.05 # enables the 100/E test-set sizing check test_set_size: 2400 # overrides counting reference_data rows unit_of_generalization: patient # what a data split must not straddle external_data_allowed: false prizes: false # resolves the game-of-skill attestation task_type: binary_classification_imbalanced ``` ### LLM-judged checks (authentication required) ```bash autocodabench validate path/to/bundle --judged ``` This option adds advisory checks graded by an LLM — for example, whether the participant-facing pages contradict the machine-readable configuration (the pages state five submissions per day while the YAML enforces ten). Judged results are findings, never gates: an LLM's assessment never blocks an upload, and an unparseable judge reply degrades to a skipped result, never to a silent pass. --- ## 4. Creating a competition agentically (authentication required) ```bash autocodabench plan-build-validate "AI-generated-text detection, balanced accuracy, \ two phases, result submission" \ --data ./sample_data/ # `create` remains as a shorter alias ``` Before any tokens are spent, `plan-build-validate` prints its full effective configuration — backend and auth path, model, the exact output directory, sample data, cost cap, output mode, and the three pipeline stages — then (on a terminal) asks where the output should go and confirms before starting. **Phase-1 research.** The banner also shows which external knowledge sources the planner may consult so the design is grounded in what already exists rather than the model's training data alone: - **OpenAlex** — recent related competition / benchmark papers (topic search, related works, top-AI-conference venue preset), via the external `openalex-research-mcp` server launched with `npx` (install [Node/npx](https://nodejs.org/); overridable with `AUTOCODABENCH_OPENALEX_MCP_CMD`). Keyless; OpenAlex appreciates a courtesy email (`OPENALEX_EMAIL`). - **Kaggle** — how similar competitions are hosted (metric, submission caps, team-size limits, deadlines, full rules pages), via first-party tools that wrap the Kaggle SDK (the `kaggle` package ships in the base install). Reads **public** competitions only and needs no key from you — a shared throw-away token is used unless you set `KAGGLE_API_TOKEN` (or have `~/.kaggle/`), with your own token from . - **Web search** — a last resort (single-source, easily biased); the planner is instructed to prefer OpenAlex and Kaggle for related-work discovery. All on by default; turn them off with `--no-research` (all) or `--no-openalex` / `--no-kaggle` / `--no-web-search` (individually). Research is a **Claude-only** capability — OpenAI-compatible / Ollama backbones cannot host the external MCP server or web tools, and the banner says so. A missing launcher/package marks that source unavailable and the plan proceeds without it. Phase 1 ends with a **provenance table** (✓ specified by your input · ⚠ partially · ✗ inferred by the planner) so you can see at a glance which decisions warrant your review. The run reports progress at one of three levels of detail: - **Default** — a concise, user-oriented narrative: a header per phase and the plain-language milestone messages the agent emits, including any *deviation* from the plan stated in non-technical terms (for example, that the plan named a scikit-learn argument removed in a recent release, that it was corrected, and the resulting metric). Raw tool calls, raw tool output, and the agent's internal reasoning are not shown. - **`--debug`** — the full developer trace: every tool call with its arguments, tool errors, and the agent's reasoning. A notice before the run explains that this mode is intended for diagnosing the pipeline rather than routine use. - **`--quiet`** — only the final summary. If the build phase departs from the locked plan in any way, it writes `specs/updated_implementation_plan.md`, which opens with a *Changes from the original plan* section enumerating each change (original specification → what changed → why); the original `implementation_plan.md` is preserved unchanged as the provenance record. Absence of the updated file means the bundle was built exactly as planned. Other useful flags: `--out DIR` (set the output location non-interactively), `--yes` (skip the confirmation), `--model` (override the model), and `--max-budget-usd` (a cost cap per phase). The pipeline proceeds as follows: 1. **Plan session.** An agent drafts the full competition design — task framing, data, metric, baselines, phases, and rules — and saves a locked `implementation_plan.md`. Headless runs make conservative assumptions and state each one explicitly in the plan. 2. Optionally, stop at this point, edit the plan by hand, and re-run the build. 3. **Build session.** A fresh agent reads only the plan and writes the bundle through the MCP tool surface: `competition.yaml`, pages, the scoring program, a baseline solution, and data; it then validates and zips the result. 4. The check framework described in section 3 runs over the resulting bundle. All artifacts are written to a per-run directory under `./.autocodabench/runs/`: the plan, the bundle and its zip archive, a complete `tool_calls/` audit trail of every authoring action, and message traces for both sessions. Relevant flags include `--model` and `--max-budget-usd` (a cost cap per phase). The same functionality is available from Python: ```python import autocodabench result = autocodabench.create("plankton image classification, two phases") print(result.bundle_dir, result.zip_path, result.total_cost_usd) print(result.validation.to_markdown()) report = autocodabench.validate("path/to/any_bundle.zip") print(report.ok, report.counts) ``` --- ## 5. Choosing an LLM backend `autocodabench plan-build-validate` (and the benchmarks under `benchmark/`) drive the model **hermetically through the backend seam** (`autocodabench.backends`): the autocodabench MCP tool surface is registered *programmatically* for each agent session, so there is **no `claude mcp add` and no `.mcp.json` to maintain** — install the package and it works, from any directory. Select a backend with `--backend` (and optionally `--model`): | spec | backbone | credentials | |------|----------|-------------| | `claude[:model]` (default) | Claude Agent SDK | subscription login or `ANTHROPIC_API_KEY` | | `ollama:` | local Ollama (offline) | none | | `openai:` | OpenAI or a proxy (`OPENAI_BASE_URL`) | `OPENAI_API_KEY` | | `#` | any OpenAI-compatible endpoint (vLLM, LiteLLM, …) | `AUTOCODABENCH_LLM_API_KEY` / `OPENAI_API_KEY` | The generic (OpenAI-compatible) backends require native tool calling and get the **same 20-tool surface and the same `tool_calls/` audit trail** as the SDK path (`autocodabench.backends.local_tools`) — that parity is what makes cross-backbone benchmarking commensurable. The same tool surface is still available as a standalone MCP stdio server (`python -m autocodabench.mcp.server`) for embedding in a custom MCP host, but it is **not** required for `plan-build-validate`, `validate`, or the benchmarks. --- ## 6. Web user interface The web UI is a Chainlit chat surface over the same plan-then-build flow, with a phase bar, cost tracking, and a Publish form. ```bash pip install -r web/requirements.txt cd web && chainlit run app.py --host 127.0.0.1 --port 8500 -h ``` A `.env` file at the repository root is required (see `.env.example`) with the following variables: `ANTHROPIC_API_KEY` (the web UI is API-key-only; see section 2), `SHARED_PASSWORD`, `CHAINLIT_AUTH_SECRET`, and `OPENALEX_MAILTO`. Operating and deploying the Space is documented in [`../web/README.md`](../web/README.md). --- ## 7. Publishing to Codabench Three equivalent routes are available; all are optional and all require an explicit action: - **Web UI Publish form.** Enter a Codabench username and password into the workspace panel; the upload is sent directly to codabench.org and never passes through the LLM. - **MCP tool** `autocodabench_upload_bundle`. This tool reads `CODABENCH_USERNAME` and `CODABENCH_PASSWORD` (or `CODABENCH_TOKEN`) from the environment. - **Script.** Run `python -m autocodabench.upload.codabench_api bundle.zip`. The four-step REST flow (token, dataset placeholder, signed PUT, unpack poll) is documented in [`codabench-upload-api.md`](./codabench-upload-api.md). --- ## 8. Troubleshooting | Symptom | Likely cause and resolution | |---|---| | `plan-build-validate` or `--judged` fails to start a session | Run `autocodabench auth status` — it verifies the agent SDK can actually sign in and reports the failure if it cannot. If no authentication is configured, log in through Claude Code or export `ANTHROPIC_API_KEY` (or place it in `./.env`); on an interactive terminal, the preflight described in section 2 offers these options directly. | | Usage is billed to the API instead of the subscription plan | A stale `ANTHROPIC_API_KEY` is exported; `auth status` warns about precisely this condition. Unset the variable. | | Bundle validates locally but is rejected by Codabench | Confirm that the uploaded archive is the zip produced by `zip_bundle` or by the pipeline — `competition.yaml` must reside at the zip root, not inside a subdirectory. | | Checks report `skipped … requires facts` | Add the named keys to `competition_facts.yaml` (section 3). | | Artifacts appear in an unexpected location | Roots default to `/.autocodabench/`; override with `AUTOCODABENCH_HOME` (or `AUTOCODABENCH_BUNDLES_ROOT` / `AUTOCODABENCH_RUNS_ROOT`). |