ccnets/causal-gpt-rl-unity

Causal GPT-RL — Unity ML-Agents Policies

Causal GPT-RL policies trained for Unity ML-Agents environments.

This repository contains self-contained ONNX policies:

pyramids/pyramids-b16.onnx
pushblock/pushblock-b32.onnx

The graph includes observation normalization and can run with ONNX Runtime or Unity Sentis without PyTorch at inference time.

Pyramids

Training data

This policy was trained using:

• unity/pyramids/simple-v0
• unity/pyramids/medium-v0

The unity/pyramids/expert-v0 dataset was not used for training and is included below only as an evaluation reference.

Evaluation

Policy	Context	Batch	Observation	Action	Return	Completed
pyramids/pyramids-b16.onnx	32	16	172	Discrete(5)	1.7738 ± 0.1278	128/128

The result is the mean ± population standard deviation over 128 episodes: 16 agents across eight fresh Unity launches using environment seeds 100..107.

All episodes completed successfully on the matching model-removed ML-Agents release-23 Pyramids build.

Dataset reference

Dataset	Episodes	Mean return	Training use
unity/pyramids/simple-v0	3,042	1.6256 ± 0.4154	Used
unity/pyramids/medium-v0	4,159	1.7432 ± 0.2601	Used
unity/pyramids/expert-v0	5,348	1.7953 ± 0.2661	Not used

The ONNX policy exceeds the mean returns of both training datasets. It is also approximately 1.2% below the expert dataset mean, despite never being trained on the expert dataset.

Dataset returns are sums of the recorded per-step rewards for each Minari episode.

ONNX interface

Input/output	Shape
states	[16, 32, 172]
actions	[16, 32, 5]
is_bos	[16, 32, 1]
mask	[16, 32]
action	[16, 5]

The four observation sensors must be concatenated in this order:

56 + 56 + 56 + 4

action contains five logits. Select the environment action with argmax, then feed its five-way one-hot representation into the next action window. Maintain one independent 32-step rolling context per agent.

The graph uses the checkpoint's bos_cache_mode=discard serving convention.

PushBlock

Policy	Ctx	Obs	Action	Batch	Validation
pushblock/pushblock-b32.onnx	32	210 (105+105)	Discrete(7)	32	159/160 successes (99.4%)

The graph inputs are states[32,32,210], actions[32,32,7], is_bos[32,32,1], and mask[32,32]; output action[32,7] contains the seven discrete-action logits. Use argmax per row, and feed the selected action back as a 7-way one-hot vector. The Unity scene has 32 agents and a decision period of 5: infer only on decision steps and repeat the selected action on the four intervening physics ticks.

Closed-loop validation covered five Unity seeds (160 first episodes): 159 successes (99.4%). A final ONNX-only run completed 32/32 episodes with return 4.968 ± 0.049.

This policy was trained on expert-only trajectories. Data affected by the dataset recall was excluded and was not used to train this release.

Companion resources

• Unity environment build
• Recorded datasets
• Evaluation code
• Causal GPT-RL models for Gymnasium and MuJoCo

License

Released under PolyForm Noncommercial License 1.0.0. See LICENSE for details. For commercial licensing, contact the maintainers via ccnets.org.