a3ro-dev commited on
Commit
222e6bf
·
verified ·
1 Parent(s): 0b70b58

reflex-2048 v0.1: 164k mean, no search, 14.5M params

Browse files
Files changed (1) hide show
  1. README.md +25 -27
README.md CHANGED
@@ -30,15 +30,13 @@ model-index:
30
 
31
  # reflex-2048
32
 
33
- A 14.5M-parameter transformer that plays 2048 with **one forward pass per move and no search**. It's distilled from the [macroxue/2048-ai](https://github.com/macroxue/2048-ai) expectimax + lookup-table teacher on one A100, for $8.72 total.
34
 
35
- Code, training pipeline and paper: [github.com/a3ro-dev/reflex-2048](https://github.com/a3ro-dev/reflex-2048).
36
-
37
- This is not a record. It's a small, cheap data point.
38
 
39
  ![one look, one move: a real board from move 1,422 becomes 21 tokens, goes through 8 transformer blocks, and comes out as move probabilities, down 0.77 and left 0.23](assets/how.png)
40
 
41
- ## Results
42
 
43
  ![how far it gets: share of 10,000 games that build each tile, 2048 96.6%, 4096 86.4%, 8192 63.3%, 16384 30.0%, 32768 3.9%, 65536 0%; mean score 164,395 against 412,785 to 625,898 for other agents](assets/results.png)
44
 
@@ -48,19 +46,19 @@ This is not a record. It's a small, cheap data point.
48
  |---|---|---|---|---|---|---|---|---|
49
  | 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
50
 
51
- A second evaluation on 2,048 games from other seeds gave 166,767 mean. Per-game scores weren't logged, so there are no standard errors.
52
 
53
  | agent | params | search | mean | 32768 |
54
  |---|---|---|---|---|
55
  | macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
56
  | 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
57
  | Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
58
- | **reflex-2048** | **14.5M** | **none** | **164,395** | **3.9%** |
59
  | reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
60
 
61
- Speed: 9.1 ms per move median on a laptop CPU (Intel, 8 threads, batch 1, features included).
62
 
63
- Calibration against teacher moves (`calibration_report.json`):
64
 
65
  | boards | agreement | mean confidence | ECE |
66
  |---|---|---|---|
@@ -68,13 +66,15 @@ Calibration against teacher moves (`calibration_report.json`):
68
  | held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
69
  | boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
70
 
71
- ## Known failures
 
 
72
 
73
- - **DAgger collapsed play.** Mixing teacher relabels of the student's own states into 30% of each batch dropped greedy play from 56,652 to 4,188 mean, while held-out agreement stayed at ~87%. The labels were verified correct, and the cause is not isolated. This checkpoint is trained without DAgger data.
74
- - **The value head can't drive search.** It's an auxiliary head trained on log return-to-go. It tracks the return across a game (0.88 log-space correlation on held-out teacher boards) but can't rank the four moves of one turn. 1-ply search on it scores 4,189.
75
- - **Overconfident off-distribution.** On boards it reaches itself, the model reports 88% confidence and agrees with the teacher 61% of the time.
76
 
77
- ## Use
 
 
78
 
79
  ```bash
80
  pip install torch numpy huggingface_hub
@@ -93,30 +93,28 @@ m.decide([[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]])
93
  # {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
94
  ```
95
 
96
- To watch a full game in the terminal, run `python -m reflex.play --ckpt reflex.pt` from the snapshot folder.
97
 
98
- The checkpoint is a plain `torch.save` dict (`cfg`, `model` state dict, `step`, `samples`). Loading it needs `weights_only=False`, so only load files you trust.
99
 
100
- ## Model
101
 
102
  ![inside the model: 16 tile tokens, 4 move tokens and 1 value token go through 8 pre-norm blocks of 12-head attention and a 384-1536-384 MLP, then a policy head with one logit per move token and a value head used only in training](assets/arch.png)
103
 
104
  ![what a move token knows: the four afterstates of a real board; up and down both merge the 512s for 1,028 points, only down keeps the 2048 in its corner, and the model puts 0.77 on down](assets/moves.png)
105
 
106
- - **Tokens:** 16 tile tokens (tile-exponent + row + column embeddings), 4 action tokens built from each move's afterstate (legality, reward, empties, max tile and corner, monotonicity, smoothness, merges, follow-up rewards), and 1 value token.
107
- - **Network:** 8 pre-norm bidirectional transformer blocks, d=384, 12 heads, MLP ratio 4. One logit per action token, with illegal moves masked.
108
- - **Action order:** 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
109
 
110
- ## Training
111
 
112
  ![how it learned: the teacher plays, 460M of its moves, copy the teacher, reflex.pt; a DAgger detour dropped play from 56,652 to 4,188](assets/train.png)
113
 
114
- - **Loss:** cross-entropy to the teacher's move plus a 0.25× Huber loss on log return-to-go. Random D4 symmetry per sample.
115
- - **Data:** 360M depth-3 and 100M depth-4 teacher self-play decisions.
116
- - **Schedule:** 261M samples total. The model starts from a first run's step-6000 checkpoint, taken before any DAgger data reached the trainer. It then trains 212M samples on depth-3/depth-4 data at 50/50 with AdamW, peak lr 5e-4 and a 1.9 h cosine schedule, in bf16 on one A100.
117
 
118
- The teacher is GPL-3.0 code. The repo ships only a patch script for it, not the code itself.
119
 
120
- ## Thanks
121
 
122
- The RunPod credits that paid for all the compute came from [Pranav Ramakrishnan](https://www.linkedin.com/in/pranavram98/). Thank you.
 
30
 
31
  # reflex-2048
32
 
33
+ a 14.5M-parameter transformer that plays 2048 with one forward pass per move and no search. it's distilled from the [macroxue/2048-ai](https://github.com/macroxue/2048-ai) expectimax + lookup-table teacher on one A100, for $8.72 total. it averages 164k, about a quarter of its teacher, so treat it as a small, cheap data point rather than a record.
34
 
35
+ code, training pipeline and paper: [github.com/a3ro-dev/reflex-2048](https://github.com/a3ro-dev/reflex-2048).
 
 
36
 
37
  ![one look, one move: a real board from move 1,422 becomes 21 tokens, goes through 8 transformer blocks, and comes out as move probabilities, down 0.77 and left 0.23](assets/how.png)
38
 
39
+ ## results
40
 
41
  ![how far it gets: share of 10,000 games that build each tile, 2048 96.6%, 4096 86.4%, 8192 63.3%, 16384 30.0%, 32768 3.9%, 65536 0%; mean score 164,395 against 412,785 to 625,898 for other agents](assets/results.png)
42
 
 
46
  |---|---|---|---|---|---|---|---|---|
47
  | 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
48
 
49
+ a second evaluation on 2,048 games from other seeds gave 166,767 mean. per-game scores weren't logged, so there are no standard errors.
50
 
51
  | agent | params | search | mean | 32768 |
52
  |---|---|---|---|---|
53
  | macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
54
  | 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
55
  | Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
56
+ | reflex-2048 | 14.5M | none | 164,395 | 3.9% |
57
  | reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
58
 
59
+ a move takes 9.1 ms median on a laptop CPU (Intel, 8 threads, batch 1, features included).
60
 
61
+ calibration against teacher moves (`calibration_report.json`):
62
 
63
  | boards | agreement | mean confidence | ECE |
64
  |---|---|---|---|
 
66
  | held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
67
  | boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
68
 
69
+ ## known failures
70
+
71
+ DAgger collapsed play. mixing teacher relabels of the student's own states into 30% of each batch dropped greedy play from 56,652 to 4,188 mean, while held-out agreement stayed at ~87%. the labels were verified correct and the cause is not isolated. this checkpoint is trained without DAgger data.
72
 
73
+ the value head can't drive search. it's an auxiliary head trained on log return-to-go, and it tracks the return across a game (0.88 log-space correlation on held-out teacher boards), but it can't rank the four moves of one turn. 1-ply search on it scores 4,189.
 
 
74
 
75
+ it's overconfident off its teacher's distribution. on boards it reaches by itself, it reports 88% confidence and agrees with the teacher 61% of the time.
76
+
77
+ ## use
78
 
79
  ```bash
80
  pip install torch numpy huggingface_hub
 
93
  # {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
94
  ```
95
 
96
+ to watch a full game in the terminal, run `python -m reflex.play --ckpt reflex.pt` from the snapshot folder.
97
 
98
+ the checkpoint is a plain `torch.save` dict (`cfg`, `model` state dict, `step`, `samples`). loading it needs `weights_only=False`, so only load files you trust.
99
 
100
+ ## model
101
 
102
  ![inside the model: 16 tile tokens, 4 move tokens and 1 value token go through 8 pre-norm blocks of 12-head attention and a 384-1536-384 MLP, then a policy head with one logit per move token and a value head used only in training](assets/arch.png)
103
 
104
  ![what a move token knows: the four afterstates of a real board; up and down both merge the 512s for 1,028 points, only down keeps the 2048 in its corner, and the model puts 0.77 on down](assets/moves.png)
105
 
106
+ 21 tokens: 16 tiles (tile-exponent + row + column embeddings), 4 moves built from each move's afterstate (legality, reward, empties, max tile and corner, monotonicity, smoothness, merges, follow-up rewards), and 1 value token. they go through 8 pre-norm bidirectional transformer blocks (d=384, 12 heads, MLP ratio 4). each move token gives one logit, with illegal moves masked. action order is 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
 
 
107
 
108
+ ## training
109
 
110
  ![how it learned: the teacher plays, 460M of its moves, copy the teacher, reflex.pt; a DAgger detour dropped play from 56,652 to 4,188](assets/train.png)
111
 
112
+ the loss is cross-entropy to the teacher's move plus 0.25× a Huber loss on log return-to-go, with a random D4 symmetry per sample. the data is 360M depth-3 and 100M depth-4 teacher self-play decisions.
113
+
114
+ the model starts from a first run's step-6000 checkpoint, taken before any DAgger data reached the trainer. it then trains 212M more samples on depth-3/depth-4 data at 50/50 (261M in total) with AdamW, peak lr 5e-4 and a 1.9 h cosine schedule, in bf16 on one A100.
115
 
116
+ the teacher is GPL-3.0 code, so the repo ships a patch script for it instead of the code.
117
 
118
+ ## thanks
119
 
120
+ all the compute was paid for with RunPod credits from [Pranav Ramakrishnan](https://www.linkedin.com/in/pranavram98/). thank you, Pranav.