reflex-2048 v0.1: 164k mean, no search, 14.5M params
Browse files
README.md
CHANGED
|
@@ -30,15 +30,13 @@ model-index:
|
|
| 30 |
|
| 31 |
# reflex-2048
|
| 32 |
|
| 33 |
-
|
| 34 |
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
This is not a record. It's a small, cheap data point.
|
| 38 |
|
| 39 |

|
| 40 |
|
| 41 |
-
##
|
| 42 |
|
| 43 |

|
| 44 |
|
|
@@ -48,19 +46,19 @@ This is not a record. It's a small, cheap data point.
|
|
| 48 |
|---|---|---|---|---|---|---|---|---|
|
| 49 |
| 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
| agent | params | search | mean | 32768 |
|
| 54 |
|---|---|---|---|---|
|
| 55 |
| macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
|
| 56 |
| 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
|
| 57 |
| Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
|
| 58 |
-
|
|
| 59 |
| reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
|
| 60 |
|
| 61 |
-
|
| 62 |
|
| 63 |
-
|
| 64 |
|
| 65 |
| boards | agreement | mean confidence | ECE |
|
| 66 |
|---|---|---|---|
|
|
@@ -68,13 +66,15 @@ Calibration against teacher moves (`calibration_report.json`):
|
|
| 68 |
| held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
|
| 69 |
| boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
|
| 70 |
|
| 71 |
-
##
|
|
|
|
|
|
|
| 72 |
|
| 73 |
-
|
| 74 |
-
- **The value head can't drive search.** It's an auxiliary head trained on log return-to-go. It tracks the return across a game (0.88 log-space correlation on held-out teacher boards) but can't rank the four moves of one turn. 1-ply search on it scores 4,189.
|
| 75 |
-
- **Overconfident off-distribution.** On boards it reaches itself, the model reports 88% confidence and agrees with the teacher 61% of the time.
|
| 76 |
|
| 77 |
-
|
|
|
|
|
|
|
| 78 |
|
| 79 |
```bash
|
| 80 |
pip install torch numpy huggingface_hub
|
|
@@ -93,30 +93,28 @@ m.decide([[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]])
|
|
| 93 |
# {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
|
| 94 |
```
|
| 95 |
|
| 96 |
-
|
| 97 |
|
| 98 |
-
|
| 99 |
|
| 100 |
-
##
|
| 101 |
|
| 102 |

|
| 103 |
|
| 104 |

|
| 105 |
|
| 106 |
-
|
| 107 |
-
- **Network:** 8 pre-norm bidirectional transformer blocks, d=384, 12 heads, MLP ratio 4. One logit per action token, with illegal moves masked.
|
| 108 |
-
- **Action order:** 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
|
| 109 |
|
| 110 |
-
##
|
| 111 |
|
| 112 |

|
| 113 |
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
|
| 118 |
-
|
| 119 |
|
| 120 |
-
##
|
| 121 |
|
| 122 |
-
|
|
|
|
| 30 |
|
| 31 |
# reflex-2048
|
| 32 |
|
| 33 |
+
a 14.5M-parameter transformer that plays 2048 with one forward pass per move and no search. it's distilled from the [macroxue/2048-ai](https://github.com/macroxue/2048-ai) expectimax + lookup-table teacher on one A100, for $8.72 total. it averages 164k, about a quarter of its teacher, so treat it as a small, cheap data point rather than a record.
|
| 34 |
|
| 35 |
+
code, training pipeline and paper: [github.com/a3ro-dev/reflex-2048](https://github.com/a3ro-dev/reflex-2048).
|
|
|
|
|
|
|
| 36 |
|
| 37 |

|
| 38 |
|
| 39 |
+
## results
|
| 40 |
|
| 41 |

|
| 42 |
|
|
|
|
| 46 |
|---|---|---|---|---|---|---|---|---|
|
| 47 |
| 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
|
| 48 |
|
| 49 |
+
a second evaluation on 2,048 games from other seeds gave 166,767 mean. per-game scores weren't logged, so there are no standard errors.
|
| 50 |
|
| 51 |
| agent | params | search | mean | 32768 |
|
| 52 |
|---|---|---|---|---|
|
| 53 |
| macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
|
| 54 |
| 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
|
| 55 |
| Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
|
| 56 |
+
| reflex-2048 | 14.5M | none | 164,395 | 3.9% |
|
| 57 |
| reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
|
| 58 |
|
| 59 |
+
a move takes 9.1 ms median on a laptop CPU (Intel, 8 threads, batch 1, features included).
|
| 60 |
|
| 61 |
+
calibration against teacher moves (`calibration_report.json`):
|
| 62 |
|
| 63 |
| boards | agreement | mean confidence | ECE |
|
| 64 |
|---|---|---|---|
|
|
|
|
| 66 |
| held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
|
| 67 |
| boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
|
| 68 |
|
| 69 |
+
## known failures
|
| 70 |
+
|
| 71 |
+
DAgger collapsed play. mixing teacher relabels of the student's own states into 30% of each batch dropped greedy play from 56,652 to 4,188 mean, while held-out agreement stayed at ~87%. the labels were verified correct and the cause is not isolated. this checkpoint is trained without DAgger data.
|
| 72 |
|
| 73 |
+
the value head can't drive search. it's an auxiliary head trained on log return-to-go, and it tracks the return across a game (0.88 log-space correlation on held-out teacher boards), but it can't rank the four moves of one turn. 1-ply search on it scores 4,189.
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
it's overconfident off its teacher's distribution. on boards it reaches by itself, it reports 88% confidence and agrees with the teacher 61% of the time.
|
| 76 |
+
|
| 77 |
+
## use
|
| 78 |
|
| 79 |
```bash
|
| 80 |
pip install torch numpy huggingface_hub
|
|
|
|
| 93 |
# {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
|
| 94 |
```
|
| 95 |
|
| 96 |
+
to watch a full game in the terminal, run `python -m reflex.play --ckpt reflex.pt` from the snapshot folder.
|
| 97 |
|
| 98 |
+
the checkpoint is a plain `torch.save` dict (`cfg`, `model` state dict, `step`, `samples`). loading it needs `weights_only=False`, so only load files you trust.
|
| 99 |
|
| 100 |
+
## model
|
| 101 |
|
| 102 |

|
| 103 |
|
| 104 |

|
| 105 |
|
| 106 |
+
21 tokens: 16 tiles (tile-exponent + row + column embeddings), 4 moves built from each move's afterstate (legality, reward, empties, max tile and corner, monotonicity, smoothness, merges, follow-up rewards), and 1 value token. they go through 8 pre-norm bidirectional transformer blocks (d=384, 12 heads, MLP ratio 4). each move token gives one logit, with illegal moves masked. action order is 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
|
|
|
|
|
|
|
| 107 |
|
| 108 |
+
## training
|
| 109 |
|
| 110 |

|
| 111 |
|
| 112 |
+
the loss is cross-entropy to the teacher's move plus 0.25× a Huber loss on log return-to-go, with a random D4 symmetry per sample. the data is 360M depth-3 and 100M depth-4 teacher self-play decisions.
|
| 113 |
+
|
| 114 |
+
the model starts from a first run's step-6000 checkpoint, taken before any DAgger data reached the trainer. it then trains 212M more samples on depth-3/depth-4 data at 50/50 (261M in total) with AdamW, peak lr 5e-4 and a 1.9 h cosine schedule, in bf16 on one A100.
|
| 115 |
|
| 116 |
+
the teacher is GPL-3.0 code, so the repo ships a patch script for it instead of the code.
|
| 117 |
|
| 118 |
+
## thanks
|
| 119 |
|
| 120 |
+
all the compute was paid for with RunPod credits from [Pranav Ramakrishnan](https://www.linkedin.com/in/pranavram98/). thank you, Pranav.
|