README.md

---
license: other
license_name: polyform-noncommercial-1.0.0
license_link: https://polyformproject.org/licenses/noncommercial/1.0.0
tags:
  - reinforcement-learning
  - unity-ml-agents
  - onnx
  - causal-gpt-rl
  - pyramids
  - pushblock
---

# Causal GPT-RL — Unity ML-Agents Policies

Causal GPT-RL policies trained for Unity ML-Agents environments.

This repository contains self-contained ONNX policies:

```text
pyramids/pyramids-b16.onnx
pushblock/pushblock-b32.onnx
```

The graph includes observation normalization and can run with ONNX Runtime or
Unity Sentis without PyTorch at inference time.

## Pyramids

### Training data

This policy was trained using:

- `unity/pyramids/simple-v0`
- `unity/pyramids/medium-v0`

The `unity/pyramids/expert-v0` dataset was not used for training and is included
below only as an evaluation reference.

### Evaluation

| Policy | Context | Batch | Observation | Action | Return | Completed |
|---|---:|---:|---:|---|---:|---:|
| `pyramids/pyramids-b16.onnx` | 32 | 16 | 172 | `Discrete(5)` | **1.7738 ± 0.1278** | **128/128** |

The result is the mean ± population standard deviation over 128 episodes:
16 agents across eight fresh Unity launches using environment seeds `100..107`.

All episodes completed successfully on the matching model-removed ML-Agents
release-23 Pyramids build.

### Dataset reference

| Dataset | Episodes | Mean return | Training use |
|---|---:|---:|---|
| `unity/pyramids/simple-v0` | 3,042 | 1.6256 ± 0.4154 | Used |
| `unity/pyramids/medium-v0` | 4,159 | 1.7432 ± 0.2601 | Used |
| `unity/pyramids/expert-v0` | 5,348 | 1.7953 ± 0.2661 | Not used |

The ONNX policy exceeds the mean returns of both training datasets. It is also
approximately 1.2% below the expert dataset mean, despite never being trained
on the expert dataset.

Dataset returns are sums of the recorded per-step rewards for each Minari
episode.

### ONNX interface

| Input/output | Shape |
|---|---|
| `states` | `[16, 32, 172]` |
| `actions` | `[16, 32, 5]` |
| `is_bos` | `[16, 32, 1]` |
| `mask` | `[16, 32]` |
| `action` | `[16, 5]` |

The four observation sensors must be concatenated in this order:

```text
56 + 56 + 56 + 4
```

`action` contains five logits. Select the environment action with `argmax`, then
feed its five-way one-hot representation into the next action window. Maintain
one independent 32-step rolling context per agent.

The graph uses the checkpoint's `bos_cache_mode=discard` serving convention.

## PushBlock

| Policy | Ctx | Obs | Action | Batch | Validation |
|---|---:|---:|---|---:|---:|
| `pushblock/pushblock-b32.onnx` | 32 | 210 (`105+105`) | `Discrete(7)` | 32 | 159/160 successes (99.4%) |

The graph inputs are `states[32,32,210]`, `actions[32,32,7]`,
`is_bos[32,32,1]`, and `mask[32,32]`; output `action[32,7]` contains the
seven discrete-action logits. Use `argmax` per row, and feed the selected
action back as a 7-way one-hot vector. The Unity scene has 32 agents and a
decision period of 5: infer only on decision steps and repeat the selected
action on the four intervening physics ticks.

Closed-loop validation covered five Unity seeds (160 first episodes): 159
successes (99.4%). A final ONNX-only run completed 32/32 episodes with return
`4.968 ± 0.049`.

This policy was trained on **expert-only trajectories**. Data affected by the
dataset recall was excluded and was not used to train this release.

## Companion resources

- [Unity environment build](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs)
- [Recorded datasets](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-datasets)
- [Evaluation code](https://github.com/ccnets-team/causal-gpt-rl/tree/main/examples/unity)
- [Causal GPT-RL models for Gymnasium and MuJoCo](https://huggingface.co/ccnets/causal-gpt-rl)

## License

Released under PolyForm Noncommercial License 1.0.0. See `LICENSE` for details.
For commercial licensing, contact the maintainers via
[ccnets.org](https://ccnets.org).