EVALUATION.md

# Evaluating these checkpoints in the CS2 environment

Each model is scored by **playing a task**: the harness drives the live game with the policy and
reports **score = mean total episode reward** (server-authoritative kills/damage minus penalties),
plus success rate. This doc is self-contained — pick a checkpoint, arrange it as a run dir, start the
game, run one command.

## What you need

- The **gamevla** codebase (contains `gamevla.envs.cs2` + `gamevla.envs.eval` + `examples/CS2/...`)
  and its conda env (PyTorch, gymnasium, websockets, the CS2 frameworks).
- A **CS2 dedicated server** for the scenario + a **rendered game client** on the eval machine
  (Linux with an NVIDIA GPU-backed X display, or Windows). The env owns screen capture + input.
- For `aimflow*`: the Qwen3-VL base model available to the model server.

## 1. Arrange a checkpoint as a run dir

Un-normalization reads `config.yaml` + `dataset_statistics.json` from **two dirs above** the weight
(`run_dir = ckpt.parents[1]`). Each folder here ships all three, so lay them out as:

```
run/<exp>/
  config.yaml
  dataset_statistics.json
  checkpoints/<weight>.pt        # e.g. steps_55000.pt  (or final_model.pt)
```
e.g.
```bash
mkdir -p run/aimflow_v3/checkpoints
cp aimflow_v3/config.yaml aimflow_v3/dataset_statistics.json run/aimflow_v3/
cp aimflow_v3/steps_55000.pt run/aimflow_v3/checkpoints/
```

## 2. Bring up the game (once)

```bash
export PYTHONPATH=<gamevla_repo>:<gamevla_repo>/gamevla/train/starVLA
python -m gamevla.envs.cs2.server.lifecycle up 5e_mirage_prefire --timeout 150
python -m gamevla.envs.cs2.client.connect <host:port> --timeout 600
```

## 3. Smoke-test the loop (no model)

Confirms server/client/capture/input/reward all work before wiring a model:
```bash
python -m gamevla.envs.eval.run_eval \
  --env cs2/5e_mirage_prefire/connector_to_a_site --endpoint <host:port> \
  --policy idle --episodes 1
```

## 4. Run a model

The eval process auto-launches a local model server for the checkpoint, then plays:
```bash
python -m gamevla.envs.eval.run_eval \
  --env cs2/5e_mirage_prefire/connector_to_a_site --endpoint <host:port> \
  --policy examples.CS2.aimflow_v3.evaluation.policy:build_policy \
  --policy-kwargs '{"ckpt":"run/aimflow_v3/checkpoints/steps_55000.pt","port":10093,"sensitivity":1.0}' \
  --episodes 3 --bot-count 5 --output result.json
# -> RESULT {... "score": <mean total reward>, "success_rate": ...}
```
Per-method factories + defaults: `examples.CS2.<exp>.evaluation.policy:build_policy` for
`<exp>` in `csbc, nitrogen_starvla, aimflow, aimflow_v2, aimflow_v3`. See each folder's `EVAL.md`.

## Mouse conversion (auto-derived, DPI-free)

Model mouse output is a screen fraction where `1.0 = 45°` of view rotation. In CS2 with raw input,
`Δdeg = counts · sensitivity · m_yaw`, so the harness derives
`mouse_gain = 45 / (sensitivity · m_yaw)` (m_yaw 0.022) and pins the matching client cvars
(`m_rawinput 1; m_customaccel 0; m_yaw 0.022; m_pitch 0.022; sensitivity <S>`) at each reset — no
DPI, no hand-tuning. Just pass your in-game `sensitivity` (default 1.0). Sanity-check one flick on
the box; if mirrored, add `"invert_dx":true` / `"invert_dy":true`.

## Agent interface (to add your own policy)

Anything implementing this plugs into `evaluate(env, policy, ...)`:
```python
class Policy:
    def reset(self): ...                       # clear history at episode start
    def act(self, obs, info) -> action_dict    # RGB frame -> env action Dict
```
Env action space: `{keys: MultiBinary(87), buttons: MultiBinary(5), mouse: int32(2), wheel: int32(1)}`.

## Tasks

Any `cs2/<scenario>/<task>` id works with `--env`:
`cs2/5e_dust2_prefire/{a_long_to_a_site, a_short_to_a_site, a_short_to_top_mid, ct_spawn_to_a_site,
ct_spawn_to_top_mid, lower_to_upper_tunnels, lower_tunnels_to_b_site, upper_tunnels_to_b_site}` and
`cs2/5e_mirage_prefire/{a1_to_ct, b_apartments_to_b_site, b_short_to_b_site, connector_to_a_site,
ct_to_a_site, market_to_b_site, mid_to_b_short, palace_to_a_site}`.

## Windows vs Linux

The env auto-selects the transport (Linux `mss` + `/dev/uinput`/XTEST; Windows `dxcam` + Win input),
so the same `run_eval` command works on both. On Windows, start the dedicated server in the
interactive desktop user's session and turn off HDR/Advanced Color before capture.