reflex-2048 v0.1: 164k mean, no search, 14.5M params
Browse files- README.md +105 -0
- calibration_report.json +410 -0
- eval_10k.jsonl +1 -0
- eval_ply1.jsonl +1 -0
- reflex.pt +3 -0
- reflex/__init__.py +0 -0
- reflex/__pycache__/__init__.cpython-314.pyc +0 -0
- reflex/__pycache__/model.cpython-314.pyc +0 -0
- reflex/env.py +183 -0
- reflex/evaluate.py +136 -0
- reflex/model.py +87 -0
- reflex/play.py +48 -0
README.md
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: pytorch
|
| 3 |
+
tags:
|
| 4 |
+
- "2048"
|
| 5 |
+
- game
|
| 6 |
+
- imitation-learning
|
| 7 |
+
- distillation
|
| 8 |
+
- transformer
|
| 9 |
+
pipeline_tag: reinforcement-learning
|
| 10 |
+
model-index:
|
| 11 |
+
- name: reflex-2048
|
| 12 |
+
results:
|
| 13 |
+
- task:
|
| 14 |
+
type: reinforcement-learning
|
| 15 |
+
name: 2048 (standard, 90% twos, no undo)
|
| 16 |
+
metrics:
|
| 17 |
+
- type: mean_score
|
| 18 |
+
value: 164395
|
| 19 |
+
name: mean score, 10,000 games, greedy, no search
|
| 20 |
+
- type: median_score
|
| 21 |
+
value: 131064
|
| 22 |
+
name: median score
|
| 23 |
+
- type: rate_32768
|
| 24 |
+
value: 0.039
|
| 25 |
+
name: share of games reaching 32768
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
# reflex-2048
|
| 29 |
+
|
| 30 |
+
A 14.5M-parameter transformer that plays 2048 with **one forward pass per move and no search**. It's distilled from the [macroxue/2048-ai](https://github.com/macroxue/2048-ai) expectimax + lookup-table teacher on one A100, for $8.72 total.
|
| 31 |
+
|
| 32 |
+
Code, training pipeline and paper: [github.com/a3ro-dev/reflex-2048](https://github.com/a3ro-dev/reflex-2048).
|
| 33 |
+
|
| 34 |
+
This is not a record. It's a small, cheap data point.
|
| 35 |
+
|
| 36 |
+
## Results
|
| 37 |
+
|
| 38 |
+
10,000 fresh games, greedy argmax policy, end-of-schedule checkpoint (not selected by eval score):
|
| 39 |
+
|
| 40 |
+
| mean | median | max | 2048 | 4096 | 8192 | 16384 | 32768 | 65536 |
|
| 41 |
+
|---|---|---|---|---|---|---|---|---|
|
| 42 |
+
| 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
|
| 43 |
+
|
| 44 |
+
A second evaluation on 2,048 games from other seeds gave 166,767 mean. Per-game scores weren't logged, so there are no standard errors.
|
| 45 |
+
|
| 46 |
+
| agent | params | search | mean | 32768 |
|
| 47 |
+
|---|---|---|---|---|
|
| 48 |
+
| macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
|
| 49 |
+
| 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
|
| 50 |
+
| Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
|
| 51 |
+
| **reflex-2048** | **14.5M** | **none** | **164,395** | **3.9%** |
|
| 52 |
+
| reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
|
| 53 |
+
|
| 54 |
+
Speed: 9.1 ms per move median on a laptop CPU (Intel, 8 threads, batch 1, features included).
|
| 55 |
+
|
| 56 |
+
Calibration against teacher moves (`calibration_report.json`):
|
| 57 |
+
|
| 58 |
+
| boards | agreement | mean confidence | ECE |
|
| 59 |
+
|---|---|---|---|
|
| 60 |
+
| held-out depth-4 teacher games | 93.6% | 0.931 | 0.006 |
|
| 61 |
+
| held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
|
| 62 |
+
| boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
|
| 63 |
+
|
| 64 |
+
## Known failures
|
| 65 |
+
|
| 66 |
+
- **DAgger collapsed play.** Mixing teacher relabels of the student's own states into 30% of each batch dropped greedy play from 56,652 to 4,188 mean, while held-out agreement stayed at ~87%. The labels were verified correct, and the cause is not isolated. This checkpoint is trained without DAgger data.
|
| 67 |
+
- **The value head can't drive search.** It's an auxiliary head trained on log return-to-go. It tracks the return across a game (0.88 log-space correlation on held-out teacher boards) but can't rank the four moves of one turn. 1-ply search on it scores 4,189.
|
| 68 |
+
- **Overconfident off-distribution.** On boards it reaches itself, the model reports 88% confidence and agrees with the teacher 61% of the time.
|
| 69 |
+
|
| 70 |
+
## Use
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
pip install torch numpy huggingface_hub
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
```python
|
| 77 |
+
from huggingface_hub import snapshot_download
|
| 78 |
+
import sys
|
| 79 |
+
path = snapshot_download("a3ro-dev/reflex-2048")
|
| 80 |
+
sys.path.insert(0, path)
|
| 81 |
+
from reflex.model import load
|
| 82 |
+
|
| 83 |
+
m = load(f"{path}/reflex.pt") # cpu by default
|
| 84 |
+
# board as tile exponents (0 = empty, k = tile 2^k), row-major
|
| 85 |
+
m.decide([[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]])
|
| 86 |
+
# {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
To watch a full game in the terminal, run `python -m reflex.play --ckpt reflex.pt` from the snapshot folder.
|
| 90 |
+
|
| 91 |
+
The checkpoint is a plain `torch.save` dict (`cfg`, `model` state dict, `step`, `samples`). Loading it needs `weights_only=False`, so only load files you trust.
|
| 92 |
+
|
| 93 |
+
## Model
|
| 94 |
+
|
| 95 |
+
- **Tokens:** 16 tile tokens (tile-exponent + row + column embeddings), 4 action tokens built from each move's afterstate (legality, reward, empties, max tile and corner, monotonicity, smoothness, merges, follow-up rewards), and 1 value token.
|
| 96 |
+
- **Network:** 8 pre-norm bidirectional transformer blocks, d=384, 12 heads, MLP ratio 4. One logit per action token, with illegal moves masked.
|
| 97 |
+
- **Action order:** 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
|
| 98 |
+
|
| 99 |
+
## Training
|
| 100 |
+
|
| 101 |
+
- **Loss:** cross-entropy to the teacher's move plus a 0.25× Huber loss on log return-to-go. Random D4 symmetry per sample.
|
| 102 |
+
- **Data:** 360M depth-3 and 100M depth-4 teacher self-play decisions.
|
| 103 |
+
- **Schedule:** 261M samples total. The model starts from a first run's step-6000 checkpoint, taken before any DAgger data reached the trainer. It then trains 212M samples on depth-3/depth-4 data at 50/50 with AdamW, peak lr 5e-4 and a 1.9 h cosine schedule, in bf16 on one A100.
|
| 104 |
+
|
| 105 |
+
The teacher is GPL-3.0 code. The repo ships only a patch script for it, not the code itself.
|
calibration_report.json
ADDED
|
@@ -0,0 +1,410 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ckpt": "/root/runs/final/reflex.pt",
|
| 3 |
+
"step": 25915,
|
| 4 |
+
"d3": {
|
| 5 |
+
"n": 200000,
|
| 6 |
+
"acc": 0.9109599590301514,
|
| 7 |
+
"ece": 0.010334860303325798,
|
| 8 |
+
"classwise_ece": 0.005813934361734425,
|
| 9 |
+
"nll": 0.20451036095619202,
|
| 10 |
+
"brier": 0.12394937127828598,
|
| 11 |
+
"mean_conf": 0.9212714433670044,
|
| 12 |
+
"reliability": [
|
| 13 |
+
{
|
| 14 |
+
"lo": 0.20000001788139343,
|
| 15 |
+
"conf": 0.2616492807865143,
|
| 16 |
+
"acc": 0.0,
|
| 17 |
+
"n": 1
|
| 18 |
+
},
|
| 19 |
+
{
|
| 20 |
+
"lo": 0.2666666805744171,
|
| 21 |
+
"conf": 0.3161606788635254,
|
| 22 |
+
"acc": 0.1666666716337204,
|
| 23 |
+
"n": 12
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"lo": 0.3333333432674408,
|
| 27 |
+
"conf": 0.3777371644973755,
|
| 28 |
+
"acc": 0.38291141390800476,
|
| 29 |
+
"n": 316
|
| 30 |
+
},
|
| 31 |
+
{
|
| 32 |
+
"lo": 0.40000003576278687,
|
| 33 |
+
"conf": 0.43759623169898987,
|
| 34 |
+
"acc": 0.43310657143592834,
|
| 35 |
+
"n": 882
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"lo": 0.46666669845581055,
|
| 39 |
+
"conf": 0.5100187063217163,
|
| 40 |
+
"acc": 0.5065382122993469,
|
| 41 |
+
"n": 4359
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"lo": 0.5333333015441895,
|
| 45 |
+
"conf": 0.5668291449546814,
|
| 46 |
+
"acc": 0.566933810710907,
|
| 47 |
+
"n": 6738
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"lo": 0.5999999642372131,
|
| 51 |
+
"conf": 0.6327977180480957,
|
| 52 |
+
"acc": 0.6009564995765686,
|
| 53 |
+
"n": 6691
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"lo": 0.6666666269302368,
|
| 57 |
+
"conf": 0.7004483938217163,
|
| 58 |
+
"acc": 0.6584302186965942,
|
| 59 |
+
"n": 6880
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"lo": 0.7333333492279053,
|
| 63 |
+
"conf": 0.7675895094871521,
|
| 64 |
+
"acc": 0.7328316569328308,
|
| 65 |
+
"n": 7441
|
| 66 |
+
},
|
| 67 |
+
{
|
| 68 |
+
"lo": 0.800000011920929,
|
| 69 |
+
"conf": 0.8342754244804382,
|
| 70 |
+
"acc": 0.7924944758415222,
|
| 71 |
+
"n": 9060
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"lo": 0.8666666746139526,
|
| 75 |
+
"conf": 0.9024046659469604,
|
| 76 |
+
"acc": 0.8668531179428101,
|
| 77 |
+
"n": 12873
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"lo": 0.9333333373069763,
|
| 81 |
+
"conf": 0.9931983351707458,
|
| 82 |
+
"acc": 0.9901137351989746,
|
| 83 |
+
"n": 144747
|
| 84 |
+
}
|
| 85 |
+
],
|
| 86 |
+
"acc_by_stage": {
|
| 87 |
+
"<2048": {
|
| 88 |
+
"acc": 0.9084683060646057,
|
| 89 |
+
"n": 11974
|
| 90 |
+
},
|
| 91 |
+
"2048-4096": {
|
| 92 |
+
"acc": 0.9051666259765625,
|
| 93 |
+
"n": 32826
|
| 94 |
+
},
|
| 95 |
+
"8192": {
|
| 96 |
+
"acc": 0.9059008359909058,
|
| 97 |
+
"n": 39554
|
| 98 |
+
},
|
| 99 |
+
"16384": {
|
| 100 |
+
"acc": 0.9102802276611328,
|
| 101 |
+
"n": 60689
|
| 102 |
+
},
|
| 103 |
+
">=32768": {
|
| 104 |
+
"acc": 0.9193551540374756,
|
| 105 |
+
"n": 54957
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
},
|
| 109 |
+
"d4": {
|
| 110 |
+
"n": 200000,
|
| 111 |
+
"acc": 0.9362399578094482,
|
| 112 |
+
"ece": 0.005877783937850665,
|
| 113 |
+
"classwise_ece": 0.0037371709357283678,
|
| 114 |
+
"nll": 0.15541012585163116,
|
| 115 |
+
"brier": 0.09056096524000168,
|
| 116 |
+
"mean_conf": 0.930762767791748,
|
| 117 |
+
"reliability": [
|
| 118 |
+
{
|
| 119 |
+
"lo": 0.2666666805744171,
|
| 120 |
+
"conf": 0.3104710280895233,
|
| 121 |
+
"acc": 0.3333333432674408,
|
| 122 |
+
"n": 9
|
| 123 |
+
},
|
| 124 |
+
{
|
| 125 |
+
"lo": 0.3333333432674408,
|
| 126 |
+
"conf": 0.3777979910373688,
|
| 127 |
+
"acc": 0.32258063554763794,
|
| 128 |
+
"n": 248
|
| 129 |
+
},
|
| 130 |
+
{
|
| 131 |
+
"lo": 0.40000003576278687,
|
| 132 |
+
"conf": 0.43695154786109924,
|
| 133 |
+
"acc": 0.4291091561317444,
|
| 134 |
+
"n": 797
|
| 135 |
+
},
|
| 136 |
+
{
|
| 137 |
+
"lo": 0.46666669845581055,
|
| 138 |
+
"conf": 0.510391354560852,
|
| 139 |
+
"acc": 0.509703516960144,
|
| 140 |
+
"n": 3710
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"lo": 0.5333333015441895,
|
| 144 |
+
"conf": 0.56622713804245,
|
| 145 |
+
"acc": 0.5708888173103333,
|
| 146 |
+
"n": 5558
|
| 147 |
+
},
|
| 148 |
+
{
|
| 149 |
+
"lo": 0.5999999642372131,
|
| 150 |
+
"conf": 0.6333951950073242,
|
| 151 |
+
"acc": 0.6303596496582031,
|
| 152 |
+
"n": 5784
|
| 153 |
+
},
|
| 154 |
+
{
|
| 155 |
+
"lo": 0.6666666269302368,
|
| 156 |
+
"conf": 0.7004299759864807,
|
| 157 |
+
"acc": 0.7268487811088562,
|
| 158 |
+
"n": 6004
|
| 159 |
+
},
|
| 160 |
+
{
|
| 161 |
+
"lo": 0.7333333492279053,
|
| 162 |
+
"conf": 0.7674108147621155,
|
| 163 |
+
"acc": 0.7979346513748169,
|
| 164 |
+
"n": 6488
|
| 165 |
+
},
|
| 166 |
+
{
|
| 167 |
+
"lo": 0.800000011920929,
|
| 168 |
+
"conf": 0.8349699974060059,
|
| 169 |
+
"acc": 0.864363968372345,
|
| 170 |
+
"n": 8309
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"lo": 0.8666666746139526,
|
| 174 |
+
"conf": 0.903027355670929,
|
| 175 |
+
"acc": 0.9281561374664307,
|
| 176 |
+
"n": 12555
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"lo": 0.9333333373069763,
|
| 180 |
+
"conf": 0.9933969974517822,
|
| 181 |
+
"acc": 0.9946790933609009,
|
| 182 |
+
"n": 150538
|
| 183 |
+
}
|
| 184 |
+
],
|
| 185 |
+
"acc_by_stage": {
|
| 186 |
+
"<2048": {
|
| 187 |
+
"acc": 0.8899039030075073,
|
| 188 |
+
"n": 8429
|
| 189 |
+
},
|
| 190 |
+
"2048-4096": {
|
| 191 |
+
"acc": 0.9120897650718689,
|
| 192 |
+
"n": 24525
|
| 193 |
+
},
|
| 194 |
+
"8192": {
|
| 195 |
+
"acc": 0.9290807843208313,
|
| 196 |
+
"n": 31275
|
| 197 |
+
},
|
| 198 |
+
"16384": {
|
| 199 |
+
"acc": 0.9418529868125916,
|
| 200 |
+
"n": 58782
|
| 201 |
+
},
|
| 202 |
+
">=32768": {
|
| 203 |
+
"acc": 0.9476288557052612,
|
| 204 |
+
"n": 76989
|
| 205 |
+
}
|
| 206 |
+
}
|
| 207 |
+
},
|
| 208 |
+
"dag": {
|
| 209 |
+
"n": 23621,
|
| 210 |
+
"acc": 0.44477373361587524,
|
| 211 |
+
"ece": 0.21562865744482934,
|
| 212 |
+
"classwise_ece": 0.11597429518194752,
|
| 213 |
+
"nll": 2.2459492683410645,
|
| 214 |
+
"brier": 0.7532422542572021,
|
| 215 |
+
"mean_conf": 0.6604024171829224,
|
| 216 |
+
"reliability": [
|
| 217 |
+
{
|
| 218 |
+
"lo": 0.20000001788139343,
|
| 219 |
+
"conf": 0.2614020109176636,
|
| 220 |
+
"acc": 0.25,
|
| 221 |
+
"n": 12
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"lo": 0.2666666805744171,
|
| 225 |
+
"conf": 0.31140047311782837,
|
| 226 |
+
"acc": 0.24229075014591217,
|
| 227 |
+
"n": 681
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"lo": 0.3333333432674408,
|
| 231 |
+
"conf": 0.3702302873134613,
|
| 232 |
+
"acc": 0.31361210346221924,
|
| 233 |
+
"n": 2248
|
| 234 |
+
},
|
| 235 |
+
{
|
| 236 |
+
"lo": 0.40000003576278687,
|
| 237 |
+
"conf": 0.43376410007476807,
|
| 238 |
+
"acc": 0.3314346671104431,
|
| 239 |
+
"n": 2809
|
| 240 |
+
},
|
| 241 |
+
{
|
| 242 |
+
"lo": 0.46666669845581055,
|
| 243 |
+
"conf": 0.5007346272468567,
|
| 244 |
+
"acc": 0.37642714381217957,
|
| 245 |
+
"n": 2978
|
| 246 |
+
},
|
| 247 |
+
{
|
| 248 |
+
"lo": 0.5333333015441895,
|
| 249 |
+
"conf": 0.5658344626426697,
|
| 250 |
+
"acc": 0.39647752046585083,
|
| 251 |
+
"n": 2555
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"lo": 0.5999999642372131,
|
| 255 |
+
"conf": 0.6301718950271606,
|
| 256 |
+
"acc": 0.44249120354652405,
|
| 257 |
+
"n": 1991
|
| 258 |
+
},
|
| 259 |
+
{
|
| 260 |
+
"lo": 0.6666666269302368,
|
| 261 |
+
"conf": 0.6996316313743591,
|
| 262 |
+
"acc": 0.4875076115131378,
|
| 263 |
+
"n": 1641
|
| 264 |
+
},
|
| 265 |
+
{
|
| 266 |
+
"lo": 0.7333333492279053,
|
| 267 |
+
"conf": 0.76542067527771,
|
| 268 |
+
"acc": 0.46928200125694275,
|
| 269 |
+
"n": 1351
|
| 270 |
+
},
|
| 271 |
+
{
|
| 272 |
+
"lo": 0.800000011920929,
|
| 273 |
+
"conf": 0.8332225680351257,
|
| 274 |
+
"acc": 0.5105690956115723,
|
| 275 |
+
"n": 1230
|
| 276 |
+
},
|
| 277 |
+
{
|
| 278 |
+
"lo": 0.8666666746139526,
|
| 279 |
+
"conf": 0.9017775058746338,
|
| 280 |
+
"acc": 0.5570417046546936,
|
| 281 |
+
"n": 1271
|
| 282 |
+
},
|
| 283 |
+
{
|
| 284 |
+
"lo": 0.9333333373069763,
|
| 285 |
+
"conf": 0.9865418672561646,
|
| 286 |
+
"acc": 0.6009476780891418,
|
| 287 |
+
"n": 4854
|
| 288 |
+
}
|
| 289 |
+
],
|
| 290 |
+
"acc_by_stage": {
|
| 291 |
+
"<2048": {
|
| 292 |
+
"acc": 0.4209636151790619,
|
| 293 |
+
"n": 18327
|
| 294 |
+
},
|
| 295 |
+
"2048-4096": {
|
| 296 |
+
"acc": 0.521415650844574,
|
| 297 |
+
"n": 4973
|
| 298 |
+
},
|
| 299 |
+
"8192": {
|
| 300 |
+
"acc": 0.6168224215507507,
|
| 301 |
+
"n": 321
|
| 302 |
+
}
|
| 303 |
+
}
|
| 304 |
+
},
|
| 305 |
+
"dageval": {
|
| 306 |
+
"n": 199505,
|
| 307 |
+
"acc": 0.6101701855659485,
|
| 308 |
+
"ece": 0.27116093451079476,
|
| 309 |
+
"classwise_ece": 0.14033046305396282,
|
| 310 |
+
"nll": 3.204610824584961,
|
| 311 |
+
"brier": 0.6528977155685425,
|
| 312 |
+
"mean_conf": 0.8813311457633972,
|
| 313 |
+
"reliability": [
|
| 314 |
+
{
|
| 315 |
+
"lo": 0.20000001788139343,
|
| 316 |
+
"conf": 0.2623162865638733,
|
| 317 |
+
"acc": 0.0,
|
| 318 |
+
"n": 3
|
| 319 |
+
},
|
| 320 |
+
{
|
| 321 |
+
"lo": 0.2666666805744171,
|
| 322 |
+
"conf": 0.31312525272369385,
|
| 323 |
+
"acc": 0.1818181723356247,
|
| 324 |
+
"n": 110
|
| 325 |
+
},
|
| 326 |
+
{
|
| 327 |
+
"lo": 0.3333333432674408,
|
| 328 |
+
"conf": 0.3742755949497223,
|
| 329 |
+
"acc": 0.2713567912578583,
|
| 330 |
+
"n": 1194
|
| 331 |
+
},
|
| 332 |
+
{
|
| 333 |
+
"lo": 0.40000003576278687,
|
| 334 |
+
"conf": 0.4368848204612732,
|
| 335 |
+
"acc": 0.29916611313819885,
|
| 336 |
+
"n": 2878
|
| 337 |
+
},
|
| 338 |
+
{
|
| 339 |
+
"lo": 0.46666669845581055,
|
| 340 |
+
"conf": 0.506822943687439,
|
| 341 |
+
"acc": 0.34820833802223206,
|
| 342 |
+
"n": 7507
|
| 343 |
+
},
|
| 344 |
+
{
|
| 345 |
+
"lo": 0.5333333015441895,
|
| 346 |
+
"conf": 0.5662879943847656,
|
| 347 |
+
"acc": 0.3780185878276825,
|
| 348 |
+
"n": 9690
|
| 349 |
+
},
|
| 350 |
+
{
|
| 351 |
+
"lo": 0.5999999642372131,
|
| 352 |
+
"conf": 0.6332005262374878,
|
| 353 |
+
"acc": 0.4069898724555969,
|
| 354 |
+
"n": 9757
|
| 355 |
+
},
|
| 356 |
+
{
|
| 357 |
+
"lo": 0.6666666269302368,
|
| 358 |
+
"conf": 0.7001890540122986,
|
| 359 |
+
"acc": 0.44430702924728394,
|
| 360 |
+
"n": 9705
|
| 361 |
+
},
|
| 362 |
+
{
|
| 363 |
+
"lo": 0.7333333492279053,
|
| 364 |
+
"conf": 0.7671082615852356,
|
| 365 |
+
"acc": 0.49206504225730896,
|
| 366 |
+
"n": 10271
|
| 367 |
+
},
|
| 368 |
+
{
|
| 369 |
+
"lo": 0.800000011920929,
|
| 370 |
+
"conf": 0.834148108959198,
|
| 371 |
+
"acc": 0.5464140176773071,
|
| 372 |
+
"n": 11559
|
| 373 |
+
},
|
| 374 |
+
{
|
| 375 |
+
"lo": 0.8666666746139526,
|
| 376 |
+
"conf": 0.9023152589797974,
|
| 377 |
+
"acc": 0.5913816690444946,
|
| 378 |
+
"n": 15293
|
| 379 |
+
},
|
| 380 |
+
{
|
| 381 |
+
"lo": 0.9333333373069763,
|
| 382 |
+
"conf": 0.9915003776550293,
|
| 383 |
+
"acc": 0.7039197683334351,
|
| 384 |
+
"n": 121538
|
| 385 |
+
}
|
| 386 |
+
],
|
| 387 |
+
"acc_by_stage": {
|
| 388 |
+
"<2048": {
|
| 389 |
+
"acc": 0.4269912540912628,
|
| 390 |
+
"n": 18895
|
| 391 |
+
},
|
| 392 |
+
"2048-4096": {
|
| 393 |
+
"acc": 0.572346031665802,
|
| 394 |
+
"n": 66956
|
| 395 |
+
},
|
| 396 |
+
"8192": {
|
| 397 |
+
"acc": 0.6439301371574402,
|
| 398 |
+
"n": 58379
|
| 399 |
+
},
|
| 400 |
+
"16384": {
|
| 401 |
+
"acc": 0.6899833083152771,
|
| 402 |
+
"n": 49178
|
| 403 |
+
},
|
| 404 |
+
">=32768": {
|
| 405 |
+
"acc": 0.6262096166610718,
|
| 406 |
+
"n": 6097
|
| 407 |
+
}
|
| 408 |
+
}
|
| 409 |
+
}
|
| 410 |
+
}
|
eval_10k.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"games": 10000, "mean_score": 164394.75, "median_score": 131064.0, "max_score": 835640, "mean_moves": 6475.5087890625, "moves_per_s": 64945.798002414296, "seconds": 997.0635359287262, "sym": false, "ply1": false, "rate_2048": 0.9657999873161316, "rate_4096": 0.8641999959945679, "rate_8192": 0.6326000094413757, "rate_16384": 0.3003999888896942, "rate_32768": 0.03909999877214432, "rate_65536": 0.0, "best_board": [[1, 6, 12, 15], [2, 5, 11, 14], [1, 8, 10, 13], [5, 1, 2, 3]], "t": 1792779363.1609044, "step": 25915, "ckpt": "/root/runs/final/reflex.pt"}
|
eval_ply1.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"games": 128, "mean_score": 4189.34375, "median_score": 3164.0, "max_score": 15412, "mean_moves": 300.59375, "moves_per_s": 2112.761483349328, "seconds": 18.21123695373535, "sym": false, "ply1": true, "rate_2048": 0.0, "rate_4096": 0.0, "rate_8192": 0.0, "rate_16384": 0.0, "rate_32768": 0.0, "rate_65536": 0.0, "best_board": [[4, 5, 2, 1], [1, 2, 3, 5], [3, 1, 8, 10], [1, 2, 6, 9]], "t": 1792779427.8805845, "step": 25915, "ckpt": "/root/runs/final/reflex.pt"}
|
reflex.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8962a89c708a7fa1e2851527718a837c8885ef90ff90d3083f4c3b6bec247810
|
| 3 |
+
size 58080843
|
reflex/__init__.py
ADDED
|
File without changes
|
reflex/__pycache__/__init__.cpython-314.pyc
ADDED
|
Binary file (132 Bytes). View file
|
|
|
reflex/__pycache__/model.cpython-314.pyc
ADDED
|
Binary file (9.49 kB). View file
|
|
|
reflex/env.py
ADDED
|
@@ -0,0 +1,183 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Batched 2048 on torch tensors. Boards are (B,4,4) int64 tile exponents (0 = empty).
|
| 2 |
+
|
| 3 |
+
Action order everywhere in this repo: 0=UP 1=DOWN 2=LEFT 3=RIGHT.
|
| 4 |
+
Every move is reduced to "slide the rows left" on a transformed view and one
|
| 5 |
+
row lookup table (18^4 rows) does the work, so thousands of games step in parallel.
|
| 6 |
+
"""
|
| 7 |
+
import torch
|
| 8 |
+
|
| 9 |
+
ACTIONS = ("UP", "DOWN", "LEFT", "RIGHT")
|
| 10 |
+
V = 18 # exponents 0..17 (131072 is the largest possible tile)
|
| 11 |
+
POW = torch.tensor([V**3, V**2, V, 1])
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def _build_row_table():
|
| 15 |
+
n = V**4
|
| 16 |
+
new = torch.zeros(n, 4, dtype=torch.int64)
|
| 17 |
+
rew = torch.zeros(n, dtype=torch.int64)
|
| 18 |
+
for code in range(n):
|
| 19 |
+
row = [(code // V**(3 - i)) % V for i in range(4)]
|
| 20 |
+
tiles = [t for t in row if t]
|
| 21 |
+
out, r, i = [], 0, 0
|
| 22 |
+
while i < len(tiles):
|
| 23 |
+
if i + 1 < len(tiles) and tiles[i] == tiles[i + 1] and tiles[i] < V - 1:
|
| 24 |
+
out.append(tiles[i] + 1); r += 1 << (tiles[i] + 1); i += 2
|
| 25 |
+
else:
|
| 26 |
+
out.append(tiles[i]); i += 1
|
| 27 |
+
new[code] = torch.tensor(out + [0] * (4 - len(out)))
|
| 28 |
+
rew[code] = r
|
| 29 |
+
return new, rew
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
_TABLES = {}
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def tables(device):
|
| 36 |
+
if device not in _TABLES:
|
| 37 |
+
if "cpu" not in _TABLES:
|
| 38 |
+
_TABLES["cpu"] = _build_row_table()
|
| 39 |
+
_TABLES[device] = tuple(t.to(device) for t in _TABLES["cpu"])
|
| 40 |
+
return _TABLES[device]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def _views(b): # (B,4,4) -> (B,4,4,4): each action as a left-slide view
|
| 44 |
+
t = b.transpose(-1, -2)
|
| 45 |
+
return torch.stack([t, t.flip(-1), b, b.flip(-1)], 1)
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def _unview(v): # inverse of _views, v: (B,4,4,4)
|
| 49 |
+
return torch.stack([v[:, 0].transpose(-1, -2), v[:, 1].flip(-1).transpose(-1, -2),
|
| 50 |
+
v[:, 2], v[:, 3].flip(-1)], 1)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def all_afterstates(b):
|
| 54 |
+
"""b (B,4,4) -> after (B,4,4,4), reward (B,4), legal (B,4) bool."""
|
| 55 |
+
new_t, rew_t = tables(str(b.device))
|
| 56 |
+
idx = (_views(b) * POW.to(b.device)).sum(-1) # (B,4,4)
|
| 57 |
+
after = _unview(new_t[idx])
|
| 58 |
+
reward = rew_t[idx].sum(-1)
|
| 59 |
+
legal = (after != b[:, None]).flatten(2).any(-1)
|
| 60 |
+
return after, reward, legal
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def spawn(b, gen=None):
|
| 64 |
+
"""Put a 2 (p=.9) or 4 at a random empty cell of every board that has one."""
|
| 65 |
+
flat = b.flatten(1).clone()
|
| 66 |
+
empty = flat == 0
|
| 67 |
+
r = torch.rand(flat.shape, device=b.device, generator=gen).masked_fill(~empty, -1)
|
| 68 |
+
pos = r.argmax(1)
|
| 69 |
+
val = 1 + (torch.rand(len(flat), device=b.device, generator=gen) < 0.1).long()
|
| 70 |
+
has = empty.any(1)
|
| 71 |
+
flat[has, pos[has]] = val[has]
|
| 72 |
+
return flat.view_as(b)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def reset(n, device="cpu", gen=None):
|
| 76 |
+
return spawn(spawn(torch.zeros(n, 4, 4, dtype=torch.int64, device=device), gen), gen)
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
# ---------------------------------------------------------------- features
|
| 80 |
+
def board_stats(b):
|
| 81 |
+
"""Hand-made geometric features of boards (...,4,4) -> (...,8), roughly in [0,1]."""
|
| 82 |
+
f = b.float()
|
| 83 |
+
empties = (b == 0).flatten(-2).sum(-1).float() / 16
|
| 84 |
+
mx = f.flatten(-2).max(-1).values
|
| 85 |
+
corners = torch.stack([f[..., 0, 0], f[..., 0, 3], f[..., 3, 0], f[..., 3, 3]], -1)
|
| 86 |
+
max_in_corner = (corners.max(-1).values == mx).float()
|
| 87 |
+
dr, dc = f[..., :, 1:] - f[..., :, :-1], f[..., 1:, :] - f[..., :-1, :]
|
| 88 |
+
# monotonicity: per line, how much of the variation goes one way
|
| 89 |
+
def mono(d):
|
| 90 |
+
inc, dec = d.clamp(min=0).sum(-1), (-d).clamp(min=0).sum(-1)
|
| 91 |
+
tot = inc + dec
|
| 92 |
+
return torch.where(tot > 0, torch.maximum(inc, dec) / tot.clamp(min=1e-6), torch.ones_like(tot)).mean(-1)
|
| 93 |
+
mono_r, mono_c = mono(dr), mono(dc.transpose(-1, -2))
|
| 94 |
+
nz_r, nz_c = (b[..., :, 1:] > 0) & (b[..., :, :-1] > 0), (b[..., 1:, :] > 0) & (b[..., :-1, :] > 0)
|
| 95 |
+
smooth = -((dr.abs() * nz_r).sum((-1, -2)) + (dc.abs() * nz_c).sum((-1, -2))) / (24 * 17)
|
| 96 |
+
merges = (((dr == 0) & nz_r).sum((-1, -2)) + ((dc == 0) & nz_c).sum((-1, -2))).float() / 24
|
| 97 |
+
return torch.stack([empties, mx / 17, max_in_corner, mono_r, mono_c, smooth, merges,
|
| 98 |
+
f.flatten(-2).sum(-1) / (16 * 17)], -1)
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
N_ACT_FEATS = 2 + 8 + 4 # legal, reward, afterstate stats, "can merge again" per next action
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def action_features(b):
|
| 105 |
+
"""The 'enriched state': for each of the 4 moves, what the afterstate looks like.
|
| 106 |
+
Returns feats (B,4,N_ACT_FEATS), legal (B,4), after (B,4,4,4), reward (B,4)."""
|
| 107 |
+
after, reward, legal = all_afterstates(b)
|
| 108 |
+
B = b.shape[0]
|
| 109 |
+
stats = board_stats(after) # (B,4,8)
|
| 110 |
+
_, r2, l2 = all_afterstates(after.reshape(B * 4, 4, 4)) # one more ply, no spawn
|
| 111 |
+
look = (r2.float().log1p() / 17 * l2).view(B, 4, 4)
|
| 112 |
+
feats = torch.cat([legal.float()[..., None], (reward.float().log1p() / 17)[..., None], stats, look], -1)
|
| 113 |
+
return feats * legal[..., None], legal, after, reward
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
# ---------------------------------------------------------------- symmetry (D4)
|
| 117 |
+
# Each transform maps a board to a board; PERM[k][a] = the action that a becomes.
|
| 118 |
+
def sym(b, k):
|
| 119 |
+
if k >= 4:
|
| 120 |
+
b = b.flip(-1)
|
| 121 |
+
return torch.rot90(b, k % 4, (-2, -1))
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def _action_perm():
|
| 125 |
+
perms = []
|
| 126 |
+
probe = torch.arange(16).view(1, 4, 4)
|
| 127 |
+
for k in range(8):
|
| 128 |
+
# find where each action maps by checking the transformed "direction" of a gradient board
|
| 129 |
+
p = []
|
| 130 |
+
for a in range(4):
|
| 131 |
+
g = torch.zeros(1, 4, 4, dtype=torch.int64)
|
| 132 |
+
# board whose only legal move is direction a: a lone tile at the far side
|
| 133 |
+
if a == 0: g[0, 3, 1] = 1
|
| 134 |
+
if a == 1: g[0, 0, 1] = 1
|
| 135 |
+
if a == 2: g[0, 1, 3] = 1
|
| 136 |
+
if a == 3: g[0, 1, 0] = 1
|
| 137 |
+
after, _, _ = all_afterstates(g)
|
| 138 |
+
target = sym(after[:, a], k)
|
| 139 |
+
sb = sym(g, k)
|
| 140 |
+
after2, _, legal2 = all_afterstates(sb)
|
| 141 |
+
hits = [c for c in range(4) if legal2[0, c] and torch.equal(after2[0, c], target[0])]
|
| 142 |
+
p.append(hits[0])
|
| 143 |
+
perms.append(p)
|
| 144 |
+
del probe
|
| 145 |
+
return torch.tensor(perms)
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
SYM_PERM = None
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def sym_perm():
|
| 152 |
+
global SYM_PERM
|
| 153 |
+
if SYM_PERM is None:
|
| 154 |
+
SYM_PERM = _action_perm()
|
| 155 |
+
return SYM_PERM
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def score_of(b):
|
| 159 |
+
"""Lower bound score implied by a board (assumes every spawn was a 2)."""
|
| 160 |
+
f = b.float()
|
| 161 |
+
return torch.where(b > 1, (f - 1) * torch.exp2(f), torch.zeros_like(f)).flatten(1).sum(1)
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
if __name__ == "__main__":
|
| 165 |
+
b = torch.tensor([[[1, 1, 2, 2], [0, 0, 0, 0], [3, 0, 3, 0], [1, 2, 1, 2]]])
|
| 166 |
+
after, rew, legal = all_afterstates(b)
|
| 167 |
+
assert after[0, 2].tolist() == [[2, 3, 0, 0], [0, 0, 0, 0], [4, 0, 0, 0], [1, 2, 1, 2]], after[0, 2]
|
| 168 |
+
assert after[0, 3].tolist() == [[0, 0, 2, 3], [0, 0, 0, 0], [0, 0, 0, 4], [1, 2, 1, 2]]
|
| 169 |
+
assert rew[0, 2] == 4 + 8 + 16 and rew[0, 3] == 4 + 8 + 16
|
| 170 |
+
assert after[0, 0, :, 0].tolist() == [1, 3, 1, 0] and legal.all()
|
| 171 |
+
P = sym_perm()
|
| 172 |
+
assert all(sorted(p.tolist()) == [0, 1, 2, 3] for p in P)
|
| 173 |
+
# symmetry consistency on random boards
|
| 174 |
+
g = torch.Generator().manual_seed(0)
|
| 175 |
+
bs = torch.randint(0, 6, (64, 4, 4), generator=g)
|
| 176 |
+
a0, r0, _ = all_afterstates(bs)
|
| 177 |
+
for k in range(8):
|
| 178 |
+
ak, rk, _ = all_afterstates(sym(bs, k))
|
| 179 |
+
for a in range(4):
|
| 180 |
+
assert torch.equal(ak[:, P[k, a]], sym(a0[:, a], k)) and torch.equal(rk[:, P[k, a]], r0[:, a])
|
| 181 |
+
f, l, _, _ = action_features(bs)
|
| 182 |
+
assert f.shape == (64, 4, N_ACT_FEATS) and torch.isfinite(f).all()
|
| 183 |
+
print("env ok", P.tolist())
|
reflex/evaluate.py
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Play many full games in parallel on GPU with the pure policy (one forward pass
|
| 2 |
+
per move, no search). Watches a checkpoint and re-evaluates whenever it changes.
|
| 3 |
+
|
| 4 |
+
python -m reflex.evaluate --ckpt runs/reflex/latest.pt --games 1024 [--watch] [--sym]
|
| 5 |
+
"""
|
| 6 |
+
import argparse, json, os, time
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn.functional as F
|
| 9 |
+
|
| 10 |
+
from .env import action_features, reset, score_of, spawn, sym, sym_perm
|
| 11 |
+
from .model import load
|
| 12 |
+
|
| 13 |
+
TILES = [2048, 4096, 8192, 16384, 32768, 65536]
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@torch.no_grad()
|
| 17 |
+
def policy(model, b, use_sym=False):
|
| 18 |
+
feats, legal, after, reward = action_features(b)
|
| 19 |
+
with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
|
| 20 |
+
logits, _ = model(b, feats, legal)
|
| 21 |
+
p = F.softmax(logits, -1)
|
| 22 |
+
if use_sym: # average the distribution over the 8 symmetric views (still one batched pass)
|
| 23 |
+
P = sym_perm().to(b.device)
|
| 24 |
+
bs = torch.cat([sym(b, k) for k in range(1, 8)])
|
| 25 |
+
f2, l2, _, _ = action_features(bs)
|
| 26 |
+
with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
|
| 27 |
+
lg, _ = model(bs, f2, l2)
|
| 28 |
+
p2 = F.softmax(lg, -1).view(7, len(b), 4)
|
| 29 |
+
for k in range(1, 8):
|
| 30 |
+
p = p + p2[k - 1][:, P[k]]
|
| 31 |
+
p = p / 8
|
| 32 |
+
return p.nan_to_num(0), legal, after, reward
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
@torch.no_grad()
|
| 36 |
+
def lookahead(model, b, chunk=65536):
|
| 37 |
+
"""1-ply expectimax on the value head: Q(s,a) = r_a + E[V(s')] with the expectation exact
|
| 38 |
+
over every spawn (a 2 w.p. .9 or a 4 w.p. .1 in each empty cell of the afterstate).
|
| 39 |
+
Not search-free: about 8x(empty cells) forward passes per move. Returns q (n,4), -inf if illegal."""
|
| 40 |
+
_, legal, after, reward = action_features(b)
|
| 41 |
+
flat = after.flatten(2) # (n,4,16)
|
| 42 |
+
empty = flat == 0
|
| 43 |
+
eye = torch.eye(16, dtype=flat.dtype, device=b.device)
|
| 44 |
+
child = flat[:, :, None, None] + eye[:, None] * torch.tensor([1, 2], device=b.device)[:, None] # (n,4,16,2,16)
|
| 45 |
+
w = empty[..., None] * torch.tensor([.9, .1], device=b.device) / empty.sum(-1, keepdim=True)[..., None].clamp(min=1)
|
| 46 |
+
valid = (w > 0) & legal[:, :, None, None]
|
| 47 |
+
cb = child[valid].view(-1, 4, 4)
|
| 48 |
+
vals = []
|
| 49 |
+
for i in range(0, len(cb), chunk):
|
| 50 |
+
f, l, _, _ = action_features(cb[i:i + chunk])
|
| 51 |
+
with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
|
| 52 |
+
_, v = model(cb[i:i + chunk], f, l)
|
| 53 |
+
vals.append(torch.expm1(10 * v.float()).clamp(min=0)) # value head predicts log1p(return)/10
|
| 54 |
+
ev = torch.zeros(valid.shape, device=b.device)
|
| 55 |
+
ev[valid] = torch.cat(vals) if vals else ev[valid]
|
| 56 |
+
q = reward.float() + (ev * w).sum((2, 3))
|
| 57 |
+
return q.masked_fill(~legal, float("-inf")), legal, after, reward
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
@torch.no_grad()
|
| 61 |
+
def play(model, n, device="cuda", use_sym=False, live=None, seed=0, ply1=False):
|
| 62 |
+
gen = torch.Generator(device=device).manual_seed(seed)
|
| 63 |
+
b = reset(n, device, gen)
|
| 64 |
+
score = torch.zeros(n, dtype=torch.int64, device=device)
|
| 65 |
+
moves = torch.zeros(n, dtype=torch.int64, device=device)
|
| 66 |
+
alive = torch.ones(n, dtype=torch.bool, device=device)
|
| 67 |
+
t0, steps = time.time(), 0
|
| 68 |
+
while alive.any():
|
| 69 |
+
idx = alive.nonzero().squeeze(1)
|
| 70 |
+
if ply1:
|
| 71 |
+
p, legal, after, reward = lookahead(model, b[idx])
|
| 72 |
+
else:
|
| 73 |
+
p, legal, after, reward = policy(model, b[idx], use_sym)
|
| 74 |
+
dead = ~legal.any(1)
|
| 75 |
+
a = p.argmax(1)
|
| 76 |
+
nb = after[torch.arange(len(idx)), a]
|
| 77 |
+
r = reward[torch.arange(len(idx)), a]
|
| 78 |
+
live_idx = idx[~dead]
|
| 79 |
+
b[live_idx] = spawn(nb[~dead], gen)
|
| 80 |
+
score[live_idx] += r[~dead]
|
| 81 |
+
moves[live_idx] += 1
|
| 82 |
+
alive[idx[dead]] = False
|
| 83 |
+
steps += 1
|
| 84 |
+
if live and steps % 25 == 0:
|
| 85 |
+
j = int(score.argmax())
|
| 86 |
+
json.dump(dict(board=b[j].tolist(), score=int(score[j]), moves=int(moves[j]),
|
| 87 |
+
alive=int(alive.sum()), games=n, step=steps), open(live + ".tmp", "w"))
|
| 88 |
+
os.replace(live + ".tmp", live)
|
| 89 |
+
dt = time.time() - t0
|
| 90 |
+
mx = b.flatten(1).max(1).values
|
| 91 |
+
res = dict(games=n, mean_score=score.float().mean().item(), median_score=score.float().median().item(),
|
| 92 |
+
max_score=int(score.max()), mean_moves=moves.float().mean().item(),
|
| 93 |
+
moves_per_s=moves.sum().item() / dt, seconds=dt, sym=use_sym, ply1=ply1)
|
| 94 |
+
for t in TILES:
|
| 95 |
+
res[f"rate_{t}"] = (mx >= t.bit_length() - 1).float().mean().item()
|
| 96 |
+
best = int(score.argmax())
|
| 97 |
+
res["best_board"] = b[best].tolist()
|
| 98 |
+
return res
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def main():
|
| 102 |
+
ap = argparse.ArgumentParser()
|
| 103 |
+
ap.add_argument("--ckpt", default="/root/runs/reflex/latest.pt")
|
| 104 |
+
ap.add_argument("--games", type=int, default=1024)
|
| 105 |
+
ap.add_argument("--sym", action="store_true")
|
| 106 |
+
ap.add_argument("--ply1", action="store_true", help="1-ply expectimax on the value head (not search-free)")
|
| 107 |
+
ap.add_argument("--watch", action="store_true")
|
| 108 |
+
ap.add_argument("--out", default="/root/runs/eval.jsonl")
|
| 109 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 110 |
+
ap.add_argument("--min-interval", type=float, default=0, help="watch mode: seconds between evaluations "
|
| 111 |
+
"(evaluation shares the GPU with training)")
|
| 112 |
+
ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
|
| 113 |
+
args = ap.parse_args()
|
| 114 |
+
last, best = None, -1
|
| 115 |
+
while True:
|
| 116 |
+
if os.path.exists(args.ckpt) and os.path.getmtime(args.ckpt) != last:
|
| 117 |
+
last, last_run = os.path.getmtime(args.ckpt), time.time()
|
| 118 |
+
model = load(args.ckpt, args.device)
|
| 119 |
+
step = torch.load(args.ckpt, weights_only=False, map_location="cpu").get("step", 0)
|
| 120 |
+
res = play(model, args.games, device=args.device, use_sym=args.sym, seed=args.seed, ply1=args.ply1,
|
| 121 |
+
live=os.path.join(os.path.dirname(args.out), "live.json"))
|
| 122 |
+
res.update(t=time.time(), step=step, ckpt=args.ckpt)
|
| 123 |
+
if args.watch and res["mean_score"] > best: # keep the best checkpoint seen so far
|
| 124 |
+
best = res["mean_score"]
|
| 125 |
+
torch.save(dict(cfg=model.cfg, model=model.state_dict(), step=step, eval=res),
|
| 126 |
+
os.path.join(os.path.dirname(args.ckpt), "best.pt"))
|
| 127 |
+
with open(args.out, "a") as f:
|
| 128 |
+
f.write(json.dumps(res) + "\n")
|
| 129 |
+
print(json.dumps({k: v for k, v in res.items() if k != "best_board"}), flush=True)
|
| 130 |
+
if not args.watch or os.path.exists("/root/runs/WRAP_UP"):
|
| 131 |
+
break
|
| 132 |
+
time.sleep(max(10, args.min_interval - (time.time() - last_run)) if last else 10)
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
if __name__ == "__main__":
|
| 136 |
+
main()
|
reflex/model.py
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Reflex: a non-autoregressive transformer that maps a 4x4 board to a calibrated
|
| 2 |
+
distribution over the 4 moves in one forward pass.
|
| 3 |
+
|
| 4 |
+
Tokens: 16 cell tokens (tile embedding + row/col embeddings) + 4 action tokens
|
| 5 |
+
(the enriched afterstate features of each move) + 1 value token. A bidirectional
|
| 6 |
+
pre-norm encoder (fused qkv + scaled_dot_product_attention) mixes them; each action
|
| 7 |
+
token emits one logit, illegal moves are masked to -inf, so the output is always a
|
| 8 |
+
valid distribution over legal moves only.
|
| 9 |
+
"""
|
| 10 |
+
import torch
|
| 11 |
+
import torch.nn as nn
|
| 12 |
+
import torch.nn.functional as F
|
| 13 |
+
|
| 14 |
+
from .env import ACTIONS, N_ACT_FEATS, V, action_features
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class Block(nn.Module):
|
| 18 |
+
def __init__(self, d, heads, ff):
|
| 19 |
+
super().__init__()
|
| 20 |
+
self.h = heads
|
| 21 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 22 |
+
self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
|
| 23 |
+
self.ff = nn.Sequential(nn.Linear(d, ff * d), nn.GELU(), nn.Linear(ff * d, d))
|
| 24 |
+
|
| 25 |
+
def forward(self, x):
|
| 26 |
+
B, T, D = x.shape
|
| 27 |
+
q, k, v = self.qkv(self.n1(x)).view(B, T, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
|
| 28 |
+
x = x + self.proj(F.scaled_dot_product_attention(q, k, v).transpose(1, 2).reshape(B, T, D))
|
| 29 |
+
return x + self.ff(self.n2(x))
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
class Reflex(nn.Module):
|
| 33 |
+
def __init__(self, d=256, layers=8, heads=8, ff=4):
|
| 34 |
+
super().__init__()
|
| 35 |
+
self.cfg = dict(d=d, layers=layers, heads=heads, ff=ff)
|
| 36 |
+
self.tile = nn.Embedding(V, d)
|
| 37 |
+
self.row = nn.Parameter(torch.randn(4, 1, d) * 0.02)
|
| 38 |
+
self.col = nn.Parameter(torch.randn(1, 4, d) * 0.02)
|
| 39 |
+
self.act_in = nn.Sequential(nn.Linear(N_ACT_FEATS, d), nn.GELU(), nn.Linear(d, d))
|
| 40 |
+
self.act_id = nn.Parameter(torch.randn(4, d) * 0.02)
|
| 41 |
+
self.val_tok = nn.Parameter(torch.randn(1, d) * 0.02)
|
| 42 |
+
self.blocks = nn.ModuleList(Block(d, heads, ff) for _ in range(layers))
|
| 43 |
+
self.norm = nn.LayerNorm(d)
|
| 44 |
+
self.pi = nn.Linear(d, 1)
|
| 45 |
+
self.v = nn.Sequential(nn.Linear(d, d), nn.GELU(), nn.Linear(d, 1))
|
| 46 |
+
|
| 47 |
+
def forward(self, b, feats=None, legal=None):
|
| 48 |
+
"""b (B,4,4) exponents. Returns logits (B,4) with illegal = -inf, value (B,)."""
|
| 49 |
+
if feats is None:
|
| 50 |
+
feats, legal, _, _ = action_features(b)
|
| 51 |
+
B = b.shape[0]
|
| 52 |
+
cells = (self.tile(b) + self.row + self.col).flatten(1, 2) # (B,16,d)
|
| 53 |
+
acts = self.act_in(feats) + self.act_id # (B,4,d)
|
| 54 |
+
x = torch.cat([cells, acts, self.val_tok.expand(B, 1, -1).to(cells.dtype)], 1)
|
| 55 |
+
for blk in self.blocks:
|
| 56 |
+
x = blk(x)
|
| 57 |
+
x = self.norm(x)
|
| 58 |
+
logits = self.pi(x[:, 16:20]).squeeze(-1).float().masked_fill(~legal, float("-inf"))
|
| 59 |
+
return logits, self.v(x[:, 20]).squeeze(-1).float()
|
| 60 |
+
|
| 61 |
+
@torch.no_grad()
|
| 62 |
+
def decide(self, board):
|
| 63 |
+
"""Type-safe single-board API: {'UP': p, 'DOWN': p, 'LEFT': p, 'RIGHT': p}."""
|
| 64 |
+
b = torch.as_tensor(board, dtype=torch.int64).view(1, 4, 4)
|
| 65 |
+
logits, _ = self(b.to(next(self.parameters()).device))
|
| 66 |
+
p = F.softmax(logits, -1)[0].nan_to_num(0).tolist()
|
| 67 |
+
return dict(zip(ACTIONS, p))
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def load(path, device="cpu"):
|
| 71 |
+
ck = torch.load(path, map_location=device, weights_only=False)
|
| 72 |
+
m = Reflex(**ck["cfg"]).to(device)
|
| 73 |
+
m.load_state_dict(ck["model"])
|
| 74 |
+
return m.eval()
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
if __name__ == "__main__":
|
| 78 |
+
import time
|
| 79 |
+
m = Reflex().eval()
|
| 80 |
+
print(f"params {sum(p.numel() for p in m.parameters())/1e6:.2f}M")
|
| 81 |
+
b = torch.tensor([[1, 1, 2, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 3]])
|
| 82 |
+
d = m.decide(b)
|
| 83 |
+
assert abs(sum(d.values()) - 1) < 1e-5 and d["UP"] > 0
|
| 84 |
+
t = time.perf_counter()
|
| 85 |
+
for _ in range(50):
|
| 86 |
+
m.decide(b)
|
| 87 |
+
print("decision", d, f"{(time.perf_counter()-t)/50*1000:.2f} ms/forward on cpu")
|
reflex/play.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Watch Reflex play one game in the terminal on a laptop CPU, one forward pass per move.
|
| 2 |
+
|
| 3 |
+
python -m reflex.play --ckpt reflex.pt [--seed 0] [--every 500]
|
| 4 |
+
"""
|
| 5 |
+
import argparse, time
|
| 6 |
+
import torch
|
| 7 |
+
|
| 8 |
+
from .env import ACTIONS, reset, spawn
|
| 9 |
+
from .evaluate import policy
|
| 10 |
+
from .model import load
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def show(b, score, moves, p):
|
| 14 |
+
tiles = [[(1 << int(x)) if x else 0 for x in row] for row in b.tolist()]
|
| 15 |
+
print(f"\nmove {moves} score {score} " + " ".join(f"{a} {q:.2f}" for a, q in zip(ACTIONS, p.tolist())))
|
| 16 |
+
for row in tiles:
|
| 17 |
+
print(" ".join(f"{t:>6}" if t else " ." for t in row))
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def main():
|
| 21 |
+
ap = argparse.ArgumentParser()
|
| 22 |
+
ap.add_argument("--ckpt", default="reflex.pt")
|
| 23 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 24 |
+
ap.add_argument("--every", type=int, default=500, help="print the board every N moves")
|
| 25 |
+
args = ap.parse_args()
|
| 26 |
+
torch.set_num_threads(max(1, torch.get_num_threads()))
|
| 27 |
+
model = load(args.ckpt, "cpu")
|
| 28 |
+
gen = torch.Generator().manual_seed(args.seed)
|
| 29 |
+
b, score, moves, t = reset(1, "cpu", gen), 0, 0, 0.0
|
| 30 |
+
policy(model, b) # warm-up: builds the move lookup table once
|
| 31 |
+
while True:
|
| 32 |
+
s = time.perf_counter()
|
| 33 |
+
p, legal, after, reward = policy(model, b)
|
| 34 |
+
t += time.perf_counter() - s
|
| 35 |
+
if not legal.any():
|
| 36 |
+
break
|
| 37 |
+
a = int(p.argmax(1))
|
| 38 |
+
if moves % args.every == 0:
|
| 39 |
+
show(b[0], score, moves, p[0])
|
| 40 |
+
score += int(reward[0, a]); moves += 1
|
| 41 |
+
b = spawn(after[:, a], gen)
|
| 42 |
+
show(b[0], score, moves, p[0])
|
| 43 |
+
print(f"\ngame over: score {score}, {moves} moves, max tile {1 << int(b.max())}, "
|
| 44 |
+
f"{t / max(moves, 1) * 1000:.2f} ms per move on cpu (features + forward)")
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
if __name__ == "__main__":
|
| 48 |
+
main()
|