| --- |
| license: other |
| tags: |
| - counter-strike |
| - world-model |
| - v-jepa2 |
| - video |
| library_name: pytorch |
| --- |
| |
| # CS2 Match-Level Hierarchical World Model (scale300) |
|
|
| A causal, action-conditioned latent world model over frozen V-JEPA2 embeddings of |
| **single-POV Counter-Strike 2 gameplay**. Sequences span a player's whole match |
| (rounds concatenated, round-number embedding per token), so the belief state can |
| carry cross-round context. Trained on 300 matches of |
| [RekaAI/CS2-10k](https://huggingface.co/datasets/RekaAI/CS2-10k) |
| (mirage + dust2), 6.94M one-second windows at 0.5s stride. |
|
|
| Screen-only contract: the encoder sees POV pixels with no HUD, radar, or parsed |
| game state. Positions and actions are used as training targets and for |
| evaluation, never as inference inputs. |
|
|
| ## Files |
|
|
| | file | what it is | |
| |---|---| |
| | `hwm_ckpt_scale300.pt` | the world model (12.25M params) + latent norm stats + action norm | |
| | `demo_heads.pt` | readout heads on the same latents: position, position@+8s, behaviour cloning (9 keys), 30s zone intent, zone-transition prior | |
| | `reports/*.json` | every number below, as produced by the scripts | |
|
|
| ## Results (match-disjoint holdout, 450 sequences) |
|
|
| Latent MSE against the true future latent, versus a persistence ("nothing |
| changes") baseline: |
|
|
| | horizon | model | persistence | |
| |---|---|---| |
| | 0.5 s | 0.279 | 0.417 | |
| | 1 s | 0.436 | 0.765 | |
| | 2 s | 0.617 | 1.091 | |
| | 4 s | 0.765 | 1.369 | |
| | 8 s | 0.882 | 1.608 | |
| | 30 s | 0.976 | 1.893 | |
|
|
| ## The limitation that matters most |
|
|
| Shuffling the action input and re-measuring gives the share of the model's |
| headroom over persistence that actually comes from knowing the action: |
|
|
| | horizon | 0.5 s | 1 s | 2 s | 4 s | 8 s | 30 s | |
| |---|---|---|---|---|---|---| |
| | action share of headroom | **24.4%** | 11.6% | 6.6% | 3.1% | 1.2% | **−0.0%** | |
|
|
| At 30 s the model is **provably action-blind**, and its MSE of 0.976 on |
| ~unit-variance latents means it is close to a mean predictor there. It beats |
| persistence at 30 s only because persistence is a poor baseline at that range. |
|
|
| Structurally: `forward(z, a, rnd)` is causally masked, so token *t* sees actions |
| only up to *t* and each horizon head predicts z*t+h* directly — **there is no |
| slot for a candidate future action sequence**. Counterfactual "what if I do X" |
| requires autoregressive rollout through the h=1 head and is only meaningful for |
| ~0.5–2 s. Macro intent must come from the destination/intent heads, not rollouts. |
|
|
| Reproduce with `scripts/hwm/hwm_action_sensitivity.py`. |
|
|
| ## 30 s intent probe |
|
|
| | features | top-1 | top-3 | moved top-1 | |
| |---|---|---|---| |
| | latent | 0.420 | 0.781 | 0.269 | |
| | HWM context | 0.413 | 0.776 | 0.303 | |
| | latent + context | 0.407 | 0.771 | 0.299 | |
| | *persistence baseline* | *0.371* | — | — | |
| | *zone-transition prior (moved subset)* | — | — | *0.318* | |
|
|
| Latent top-1 clears the persistence baseline. On the **moved** subset (63% of |
| cases, where the player actually changes zone — the ones that matter for |
| next-action) all three feature sets still **lose to a trivial zone-transition |
| prior**. `beats_transition_prior` is false everywhere. Scaling 100 → 300 matches |
| narrowed that gap but did not close it, and improved the dynamics model only |
| ~2%, so scaling in this range is sublinear. |
|
|
| ## Demo: verified on holdout video |
|
|
| `scripts/hwm/demo_pov_video.py` replays a holdout match as real video through |
| the frozen encoder and this checkpoint, with the action input supplied by the BC |
| head reading the same pixels. Ground truth is overlaid only to check. |
|
|
| | metric | result | baseline | |
| |---|---|---| |
| | self-localization (median) | 1.99 m | — | |
| | 8 s position forecast | 6.03 m | 9.68 m stand-still | |
| | 30 s intent top-3 | 60.3% | 55.8% zone-transition prior | |
| | 2 s counterfactual, true action closest | 46.2% | 16.7% chance | |
|
|
| Caveats, in the interest of not overstating: the 8 s figure shown is a **direct** |
| forecast head — decoding the HWM's own h=16 latent scores slightly worse (6.53 m), |
| so the world model is not what drives that panel. The counterfactual is scored in |
| **latent space**; scored on map positions it falls to 18.6%, i.e. chance, because |
| the position head's ~99 u error swamps the 238 u candidate spread. It is an |
| aggregate claim, not a per-frame one. The position head is unconstrained and |
| emits off-map coordinates on ~1% of frames. |
|
|
| ## Intended use |
|
|
| Research on latent world models and intent prediction from egocentric video. |
| Not a cheat, not an aim assist: it consumes recorded POV video offline and |
| predicts where a player will be, at metre-to-zone granularity. |
|
|
| ## Provenance |
|
|
| Derived from RekaAI/CS2-10k; that dataset's terms apply upstream. Encoder is |
| `facebook/vjepa2-vitl-fpc64-256`, frozen. Match split is deterministic |
| (md5 of `match_id`), so the holdout is reproducible. The checkpoint itself is |
| **not** bit-reproducible: training seeds the RNGs but GPU nondeterminism means a |
| re-run yields a statistically equivalent, not identical, model. |
|
|
| Code: https://github.com/cristibctr/HWM-CS2 |
|
|
