Publish Pyramids-only Unity policy

README.md

@@ -7,266 +7,91 @@ tags:
   - unity-ml-agents
   - onnx
   - causal-gpt-rl
+  - pyramids
 ---
 
-# Causal GPT-RL — Unity ML-Agents (ONNX)
+# Causal GPT-RL — Pyramids
 
-GPT-style transformers (GPT-2, Llama) running as RL policies inside Unity
-ML-Agents scenes.
+A Causal GPT-RL policy trained for the Unity ML-Agents release-23 Pyramids
+environment.
 
-Both LLM generation and RL interaction are autoregressive:
+This repository contains one self-contained ONNX policy:
 
 ```text
-token           → next token                           (LLM generation)
-(state, action) → (next state from env, next action)   (RL rollout)
+pyramids/pyramids-b16.onnx
 ```
 
-Causal GPT-RL policies act stably under their own rollouts — long-horizon control
-without the drift that has historically kept transformers from being usable as RL
-agents. A single autoregressive model drives full-episode rollouts — no critic, no
-auxiliary networks at inference.
+The graph includes observation normalization and can run with ONNX Runtime or
+Unity Sentis without PyTorch at inference time.
 
-This repo holds ready-to-run Unity policies as single self-contained ONNX graphs.
-They run with any ONNX runtime — `onnxruntime` (Python) or Unity
-Sentis (C# / native) — with **no Python runtime, no PyTorch, and no ML-Agents
-Python stack required**. Observations go in raw; state normalization is baked into
-the graph. It is the Unity companion to the Gymnasium / MuJoCo policies at
-[ccnets/causal-gpt-rl](https://huggingface.co/ccnets/causal-gpt-rl).
+## Training data
 
-**Companion repos**
-- Unity build to run it against:
-  [ccnets/causal-gpt-rl-unity-envs](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs)
-- Recorded trajectories + reproduction recipe:
-  [ccnets/causal-gpt-rl-unity-datasets](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-datasets)
+This policy was trained using:
 
-**Links**
-- **Code (GitHub):** [ccnets-team/causal-gpt-rl](https://github.com/ccnets-team/causal-gpt-rl)
-- **Run logs (W&B, public):** [wandb.ai/junhopark/Causal GPT-RL](https://wandb.ai/junhopark/Causal%20GPT-RL)
-- **Hugging Face org:** https://huggingface.co/ccnets
-- Website: https://ccnets.org
-- LinkedIn: https://www.linkedin.com/company/ccnets
+- `unity/pyramids/simple-v0`
+- `unity/pyramids/medium-v0`
 
-## What's in this repo
+The `unity/pyramids/expert-v0` dataset was not used for training and is included
+below only as an evaluation reference.
 
-```text
-ccnets/causal-gpt-rl-unity/
-  pushblock.onnx
-  dungeon-escape/
-    dungeonescape-b36.onnx
-  pyramids/
-    pyramids-b16.onnx
-  soccer-twos/
-    soccertwos-b16.onnx
-  README.md
-```
-
-## ONNX I/O contract
-
-Every graph uses the same named windowed-policy interface; dimensions are
-model-specific:
-
-| Input | Shape | Meaning |
-|---|---|---|
-| `states` | `[B, T, state_size]` | last `T` **raw** observations |
-| `actions` | `[B, T, action_size]` | previous actions in model representation |
-| `is_bos` | `[B, T, 1]` | begin-of-sequence flag per window step |
-| `mask` | `[B, T]` | valid-step mask for the window |
-
-Output: `action` → `[B, action_size]`.
-
-- **Raw observations in.** State normalization is inside the graph — feed the
-  build's observations unchanged.
-- **Maintain a 32-step rolling window.** On reset, push `(s0, zero-action,
-  is_bos=1, mask=1)`; each step push `(s_next, previous raw action, is_bos=0,
-  mask=1)`; the window rolls.
-- **Decode.** The environment action is `clip(raw, -1, 1)`. Feed the **raw**
-  action back into the window (autoregressive feedback), not the clipped one.
-- **One context per agent.** A batch-1 graph is called once per agent; a fixed
-  full-scene batch evaluates all agents in one call.
-
-## Quick Start (onnxruntime)
-
-A single forward call:
-
-```python
-import numpy as np
-import onnxruntime as ort
-
-sess = ort.InferenceSession("pyramids/pyramids-b16.onnx", providers=["CPUExecutionProvider"])
-action = sess.run(
-    ["action"],
-    {
-        "states":  states,   # [16, 32, 172] float32, raw observations
-        "actions": actions,  # [16, 32, 5]   float32, one-hot feedback
-        "is_bos":  is_bos,   # [16, 32, 1]   float32
-        "mask":    mask,     # [16, 32]      float32
-    },
-)[0]                          # -> [16, 5] logits
-env_action = np.argmax(action, axis=-1)
-```
-
-A ready-made closed-loop evaluator downloads no training/runtime code, manages
-one rolling window per agent, supports continuous/discrete/hybrid actions, and
-reports agent and cooperative-group returns:
-[`examples/unity/evaluate_onnx.py`](https://github.com/ccnets-team/causal-gpt-rl/blob/main/examples/unity/evaluate_onnx.py).
-
-The accompanying
-[`DungeonEscape notebook`](https://github.com/ccnets-team/causal-gpt-rl/blob/main/examples/unity/dungeon_escape_hf.ipynb)
-shows the Hugging Face download and evaluation workflow.
-
-## SoccerTwos
-
-| Policy | Ctx | Batch | Obs | Action | Side-swapped W/D/L | Win rate |
-|---|---:|---:|---:|---|---:|---:|
-| `soccer-twos/soccertwos-b16.onnx` | 32 | 16 | 336 | `MultiDiscrete([3,3,3])` logits | 4/0/12 | 25.00% |
-
-This is the **completed training-run checkpoint**. The corrected smoke result
-uses the bundle's `bos_cache_mode=discard` serving convention and one
-side-swapped run against the release-23 stock policy: 4 wins, 0 draws, and 12
-losses (**25.00% win rate**). `--seed 0` seeded stock-policy action sampling;
-the current Unity wrapper used its deterministic launch seeds 100 and 101, so
-this is not a multi-environment-seed benchmark.
-
-The original `config.json` + `model.safetensors` bundle, evaluated through the
-cached `PolicyRunner` with BOS discard, produced the same 4/0/12 result. This
-validates the published ONNX at the closed-loop W/D/L level. Small numeric
-differences changed episode tick counts, so this is not a claim of bit-identical
-trajectories.
-
-One release-23 scene contains eight independent 2-vs-2 fields. The Causal graph
-controls both players on one team (`8 fields x 2 agents = 16` independent context
-rows) while the stock ML-Agents ONNX controls the opponent team. Side swapping
-reduces team-0/team-1 placement bias. The sample is still a smoke comparison,
-not a statistically complete benchmark; comparisons should report the exact
-environment build, opponent policy, action sampling, and number of seeds.
+## Evaluation
 
-Public offline datasets were collected from release-23 self-play:
-
-| Dataset | Policy generation | Matches | Ego episodes | Transitions |
-|---|---|---:|---:|---:|
-| `unity/soccer-twos/expert-v0` | stock ONNX self-play | 3,647 | 14,588 | 1,000,816 |
-| `unity/soccer-twos/medium-v0` | team epsilon in `{0.08, 0.09, 0.10}` | 3,260 | 13,040 | 1,001,776 |
-| `unity/soccer-twos/simple-v0` | team epsilon in `{0.08, 0.10, 0.15}` | 3,215 | 12,860 | 1,003,048 |
-
-For each noisy-policy match, each team independently samples one epsilon value.
-Both teammates share that noise level, while random action replacement is
-sampled independently for each agent. Unity's individual and group rewards are
-stored without reshaping or reward engineering. Each 2-vs-2 match becomes four
-ego-agent trajectories linked by `match_id`.
-
-The tiers were selected by side-swapped calibration against the fixed stock
-policy, not by mean dataset return: adversarial self-play naturally contains
-both winning and losing trajectories, so its mean agent return remains near
-zero. As a direct separation check, the medium policy pool scored `0.795`
-against the simple pool over 200 side-swapped matches.
-
-SoccerTwos ONNX contract:
-
-| Input/output | Shape |
-|---|---|
-| `states` | `[16, 32, 336]` |
-| `actions` | `[16, 32, 9]` (three one-hot branches) |
-| `is_bos` | `[16, 32, 1]` |
-| `mask` | `[16, 32]` |
-| `action` | `[16, 9]` (three groups of three logits; `argmax` per branch) |
-
-The public side-swapped evaluator and notebook are in
-[`examples/unity`](https://github.com/ccnets-team/causal-gpt-rl/tree/main/examples/unity).
-Batching is computational only; teammates do not cross-attend or share hidden
-state.
-
-## Pyramids
-
-| Policy | Ctx | Batch | Obs | Action | Return | Completion |
+| Policy | Context | Batch | Observation | Action | Return | Completed |
 |---|---:|---:|---:|---|---:|---:|
-| `pyramids/pyramids-b16.onnx` | 32 | 16 | 172 | `Discrete(5)` logits | **1.7738 ± 0.1278** | **128/128** |
+| `pyramids/pyramids-b16.onnx` | 32 | 16 | 172 | `Discrete(5)` | **1.7738 ± 0.1278** | **128/128** |
 
-The return is mean ± population standard deviation over 128 first-agent
-episodes: 16 independent agents in each of eight fresh Unity launches, using
-environment seeds `100..107`. The matching model-removed release-23 build was
-run headless with `time_scale=40`; each launch used a separate worker ID.
-Every measured ONNX episode completed successfully.
+The result is the mean ± population standard deviation over 128 episodes:
+16 agents across eight fresh Unity launches using environment seeds `100..107`.
 
-The four observation sensors are preserved by the dataset as
-`Tuple(Box(56), Box(56), Box(56), Box(4))` and concatenated in that order for
-the graph. The discrete output contains five logits; use `argmax` and feed its
-five-way one-hot representation back into the next action window. The graph
-uses the bundle's `bos_cache_mode=discard` convention.
+All episodes completed successfully on the matching model-removed ML-Agents
+release-23 Pyramids build.
 
-Public one-million-transition dataset references:
+## Dataset reference
 
-| Dataset | Episodes | Return |
-|---|---:|---:|
-| `unity/pyramids/simple-v0` | 3,040 | 1.6339 ± 0.3822 |
-| `unity/pyramids/medium-v0` | 4,267 | 1.7498 ± 0.2528 |
-| `unity/pyramids/expert-v0` | 5,348 | 1.7953 ± 0.2661 |
+| Dataset | Episodes | Mean return | Training use |
+|---|---:|---:|---|
+| `unity/pyramids/simple-v0` | 3,040 | 1.6339 ± 0.3822 | Used |
+| `unity/pyramids/medium-v0` | 4,267 | 1.7498 ± 0.2528 | Used |
+| `unity/pyramids/expert-v0` | 5,348 | 1.7953 ± 0.2661 | Not used |
 
-The published ONNX is above the simple and medium dataset means and 1.2% below
-the expert mean on this protocol. Dataset returns are sums of the recorded
-per-step rewards for each Minari episode.
+The ONNX policy exceeds the mean returns of both training datasets. It is also
+approximately 1.2% below the expert dataset mean, despite never being trained
+on the expert dataset.
 
-The original `config.json` + `model.safetensors` checkpoint was also measured
-with the cached `PolicyRunner`, BOS discard, cache length 32, and the same eight
-Unity seeds. It scored approximately `1.6425 ± 0.5076`, with 124/128 successful
-episodes. This is a real closed-loop runtime difference despite the exporter's
-single-window numerical check (`max_abs_error=9.537e-07`); the published metric
-therefore describes the stateless rolling-window ONNX artifact in this repo,
-not the cached PyTorch serving path.
+Dataset returns are sums of the recorded per-step rewards for each Minari
+episode.
 
-Pyramids ONNX contract:
+## ONNX interface
 
 | Input/output | Shape |
 |---|---|
 | `states` | `[16, 32, 172]` |
-| `actions` | `[16, 32, 5]` (one-hot feedback) |
+| `actions` | `[16, 32, 5]` |
 | `is_bos` | `[16, 32, 1]` |
 | `mask` | `[16, 32]` |
-| `action` | `[16, 5]` (logits; `argmax` per row) |
-
-## DungeonEscape
-
-| Policy | Ctx | Batch | Obs | Action | Agent return | Group success |
-|---|---:|---:|---:|---|---:|---:|
-| `dungeon-escape/dungeonescape-b36.onnx` | 32 | 36 | 371 | `Discrete(7)` logits | 0.2222 | 41.67% |
+| `action` | `[16, 5]` |
 
-This is an **intermediate training checkpoint** published to validate the
-end-to-end download and evaluation workflow. Its reported performance is
-provisional and does not represent the final trained policy. The artifact and
-metrics will be updated after training completes. The measurement above is one
-deterministic first-episode pass over the 36-agent scene (12 independent arenas
-× three cooperative agents), using a compatible local release-23 build. The
-matching model-removed build is published separately in the environment repo.
+The four observation sensors must be concatenated in this order:
 
-DungeonEscape is decentralized cooperative control. Each three-agent match is
-stored as three linked ego-agent trajectories, and the same shared policy is
-applied independently to every row. Batch inference is computational batching,
-not cross-agent attention or hidden-state communication. Cooperation comes from
-ego observations and demonstrations generated under the group reward.
-
-Dataset reference (`unity/dungeon-escape/expert-v0`): mean agent return
-0.606589; mean summed three-agent group return 1.819767; group success 95.32%.
-An agent can terminate with zero return before its teammates succeed, so agent
-zero is not equivalent to group failure.
+```text
+56 + 56 + 56 + 4
+```
 
-DungeonEscape ONNX contract:
+`action` contains five logits. Select the environment action with `argmax`, then
+feed its five-way one-hot representation into the next action window. Maintain
+one independent 32-step rolling context per agent.
 
-| Input/output | Shape |
-|---|---|
-| `states` | `[36, 32, 371]` |
-| `actions` | `[36, 32, 7]` (one-hot feedback) |
-| `is_bos` | `[36, 32, 1]` |
-| `mask` | `[36, 32]` |
-| `action` | `[36, 7]` (logits; `argmax` per row) |
+The graph uses the checkpoint's `bos_cache_mode=discard` serving convention.
 
-## Requirements
+## Companion resources
 
-- A matching model-removed Unity ML-Agents build (`.exe`) — see the
-  [envs repo](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs).
-- Any ONNX runtime: `onnxruntime` (Python) or Unity Sentis (C# / native).
-- For the Python runner: the ML-Agents → step wrapper (`unity_env.py`) on
-  `PYTHONPATH` (see the example script header).
+- [Unity environment build](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs)
+- [Recorded datasets](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-datasets)
+- [Evaluation code](https://github.com/ccnets-team/causal-gpt-rl/tree/main/examples/unity)
+- [Causal GPT-RL models for Gymnasium and MuJoCo](https://huggingface.co/ccnets/causal-gpt-rl)
 
 ## License
 
 Released under PolyForm Noncommercial License 1.0.0. See `LICENSE` for details.
-For commercial licensing, contact the maintainers via ccnets.org.
+For commercial licensing, contact the maintainers via
+[ccnets.org](https://ccnets.org).

dungeon-escape/dungeonescape-b36.onnx

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9e5592ee5ee856417c518291bd127350a42a89da794b424fa856cc90b0ad90a7
-size 18005378

pushblock.onnx

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e97d9dbc1b2d907c7d057d086a62d353c5fa4d93f719f03c02fb4422efab577b
-size 17807710

soccer-twos/soccertwos-b16.onnx

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:17cac9c9fba7ef1a769bdc344b5bfad4074a3403b03999e6ff5a6ffd8cf118ab
-size 17974556