a3ro-dev commited on
Commit
b77d5ed
·
verified ·
1 Parent(s): 57520c7

reflex-2048 v0.1: 164k mean, no search, 14.5M params

Browse files
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ tags:
4
+ - "2048"
5
+ - game
6
+ - imitation-learning
7
+ - distillation
8
+ - transformer
9
+ pipeline_tag: reinforcement-learning
10
+ model-index:
11
+ - name: reflex-2048
12
+ results:
13
+ - task:
14
+ type: reinforcement-learning
15
+ name: 2048 (standard, 90% twos, no undo)
16
+ metrics:
17
+ - type: mean_score
18
+ value: 164395
19
+ name: mean score, 10,000 games, greedy, no search
20
+ - type: median_score
21
+ value: 131064
22
+ name: median score
23
+ - type: rate_32768
24
+ value: 0.039
25
+ name: share of games reaching 32768
26
+ ---
27
+
28
+ # reflex-2048
29
+
30
+ A 14.5M-parameter transformer that plays 2048 with **one forward pass per move and no search**. It's distilled from the [macroxue/2048-ai](https://github.com/macroxue/2048-ai) expectimax + lookup-table teacher on one A100, for $8.72 total.
31
+
32
+ Code, training pipeline and paper: [github.com/a3ro-dev/reflex-2048](https://github.com/a3ro-dev/reflex-2048).
33
+
34
+ This is not a record. It's a small, cheap data point.
35
+
36
+ ## Results
37
+
38
+ 10,000 fresh games, greedy argmax policy, end-of-schedule checkpoint (not selected by eval score):
39
+
40
+ | mean | median | max | 2048 | 4096 | 8192 | 16384 | 32768 | 65536 |
41
+ |---|---|---|---|---|---|---|---|---|
42
+ | 164,395 | 131,064 | 835,640 | 96.6% | 86.4% | 63.3% | 30.0% | 3.9% | 0% |
43
+
44
+ A second evaluation on 2,048 games from other seeds gave 166,767 mean. Per-game scores weren't logged, so there are no standard errors.
45
+
46
+ | agent | params | search | mean | 32768 |
47
+ |---|---|---|---|---|
48
+ | macroxue expectimax, large tables (the teacher) | tables | depth 4 | 625,898 | 70.7% |
49
+ | 2048former v0.1 (self-reported) | 53.2M | none | 605,491 | 66.1% |
50
+ | Guei et al. 2021, n-tuple | 268.4M | none | 412,785 | 30.2% |
51
+ | **reflex-2048** | **14.5M** | **none** | **164,395** | **3.9%** |
52
+ | reflex-2048 + 1-ply over its value head (128 games) | 14.5M | 1-ply | 4,189 | 0% |
53
+
54
+ Speed: 9.1 ms per move median on a laptop CPU (Intel, 8 threads, batch 1, features included).
55
+
56
+ Calibration against teacher moves (`calibration_report.json`):
57
+
58
+ | boards | agreement | mean confidence | ECE |
59
+ |---|---|---|---|
60
+ | held-out depth-4 teacher games | 93.6% | 0.931 | 0.006 |
61
+ | held-out depth-3 teacher games | 91.1% | 0.921 | 0.010 |
62
+ | boards the model reaches itself (fresh seeds, never trained on) | 61.0% | 0.881 | 0.271 |
63
+
64
+ ## Known failures
65
+
66
+ - **DAgger collapsed play.** Mixing teacher relabels of the student's own states into 30% of each batch dropped greedy play from 56,652 to 4,188 mean, while held-out agreement stayed at ~87%. The labels were verified correct, and the cause is not isolated. This checkpoint is trained without DAgger data.
67
+ - **The value head can't drive search.** It's an auxiliary head trained on log return-to-go. It tracks the return across a game (0.88 log-space correlation on held-out teacher boards) but can't rank the four moves of one turn. 1-ply search on it scores 4,189.
68
+ - **Overconfident off-distribution.** On boards it reaches itself, the model reports 88% confidence and agrees with the teacher 61% of the time.
69
+
70
+ ## Use
71
+
72
+ ```bash
73
+ pip install torch numpy huggingface_hub
74
+ ```
75
+
76
+ ```python
77
+ from huggingface_hub import snapshot_download
78
+ import sys
79
+ path = snapshot_download("a3ro-dev/reflex-2048")
80
+ sys.path.insert(0, path)
81
+ from reflex.model import load
82
+
83
+ m = load(f"{path}/reflex.pt") # cpu by default
84
+ # board as tile exponents (0 = empty, k = tile 2^k), row-major
85
+ m.decide([[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]])
86
+ # {'UP': ..., 'DOWN': ..., 'LEFT': ..., 'RIGHT': ...}
87
+ ```
88
+
89
+ To watch a full game in the terminal, run `python -m reflex.play --ckpt reflex.pt` from the snapshot folder.
90
+
91
+ The checkpoint is a plain `torch.save` dict (`cfg`, `model` state dict, `step`, `samples`). Loading it needs `weights_only=False`, so only load files you trust.
92
+
93
+ ## Model
94
+
95
+ - **Tokens:** 16 tile tokens (tile-exponent + row + column embeddings), 4 action tokens built from each move's afterstate (legality, reward, empties, max tile and corner, monotonicity, smoothness, merges, follow-up rewards), and 1 value token.
96
+ - **Network:** 8 pre-norm bidirectional transformer blocks, d=384, 12 heads, MLP ratio 4. One logit per action token, with illegal moves masked.
97
+ - **Action order:** 0=UP, 1=DOWN, 2=LEFT, 3=RIGHT.
98
+
99
+ ## Training
100
+
101
+ - **Loss:** cross-entropy to the teacher's move plus a 0.25× Huber loss on log return-to-go. Random D4 symmetry per sample.
102
+ - **Data:** 360M depth-3 and 100M depth-4 teacher self-play decisions.
103
+ - **Schedule:** 261M samples total. The model starts from a first run's step-6000 checkpoint, taken before any DAgger data reached the trainer. It then trains 212M samples on depth-3/depth-4 data at 50/50 with AdamW, peak lr 5e-4 and a 1.9 h cosine schedule, in bf16 on one A100.
104
+
105
+ The teacher is GPL-3.0 code. The repo ships only a patch script for it, not the code itself.
calibration_report.json ADDED
@@ -0,0 +1,410 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "ckpt": "/root/runs/final/reflex.pt",
3
+ "step": 25915,
4
+ "d3": {
5
+ "n": 200000,
6
+ "acc": 0.9109599590301514,
7
+ "ece": 0.010334860303325798,
8
+ "classwise_ece": 0.005813934361734425,
9
+ "nll": 0.20451036095619202,
10
+ "brier": 0.12394937127828598,
11
+ "mean_conf": 0.9212714433670044,
12
+ "reliability": [
13
+ {
14
+ "lo": 0.20000001788139343,
15
+ "conf": 0.2616492807865143,
16
+ "acc": 0.0,
17
+ "n": 1
18
+ },
19
+ {
20
+ "lo": 0.2666666805744171,
21
+ "conf": 0.3161606788635254,
22
+ "acc": 0.1666666716337204,
23
+ "n": 12
24
+ },
25
+ {
26
+ "lo": 0.3333333432674408,
27
+ "conf": 0.3777371644973755,
28
+ "acc": 0.38291141390800476,
29
+ "n": 316
30
+ },
31
+ {
32
+ "lo": 0.40000003576278687,
33
+ "conf": 0.43759623169898987,
34
+ "acc": 0.43310657143592834,
35
+ "n": 882
36
+ },
37
+ {
38
+ "lo": 0.46666669845581055,
39
+ "conf": 0.5100187063217163,
40
+ "acc": 0.5065382122993469,
41
+ "n": 4359
42
+ },
43
+ {
44
+ "lo": 0.5333333015441895,
45
+ "conf": 0.5668291449546814,
46
+ "acc": 0.566933810710907,
47
+ "n": 6738
48
+ },
49
+ {
50
+ "lo": 0.5999999642372131,
51
+ "conf": 0.6327977180480957,
52
+ "acc": 0.6009564995765686,
53
+ "n": 6691
54
+ },
55
+ {
56
+ "lo": 0.6666666269302368,
57
+ "conf": 0.7004483938217163,
58
+ "acc": 0.6584302186965942,
59
+ "n": 6880
60
+ },
61
+ {
62
+ "lo": 0.7333333492279053,
63
+ "conf": 0.7675895094871521,
64
+ "acc": 0.7328316569328308,
65
+ "n": 7441
66
+ },
67
+ {
68
+ "lo": 0.800000011920929,
69
+ "conf": 0.8342754244804382,
70
+ "acc": 0.7924944758415222,
71
+ "n": 9060
72
+ },
73
+ {
74
+ "lo": 0.8666666746139526,
75
+ "conf": 0.9024046659469604,
76
+ "acc": 0.8668531179428101,
77
+ "n": 12873
78
+ },
79
+ {
80
+ "lo": 0.9333333373069763,
81
+ "conf": 0.9931983351707458,
82
+ "acc": 0.9901137351989746,
83
+ "n": 144747
84
+ }
85
+ ],
86
+ "acc_by_stage": {
87
+ "<2048": {
88
+ "acc": 0.9084683060646057,
89
+ "n": 11974
90
+ },
91
+ "2048-4096": {
92
+ "acc": 0.9051666259765625,
93
+ "n": 32826
94
+ },
95
+ "8192": {
96
+ "acc": 0.9059008359909058,
97
+ "n": 39554
98
+ },
99
+ "16384": {
100
+ "acc": 0.9102802276611328,
101
+ "n": 60689
102
+ },
103
+ ">=32768": {
104
+ "acc": 0.9193551540374756,
105
+ "n": 54957
106
+ }
107
+ }
108
+ },
109
+ "d4": {
110
+ "n": 200000,
111
+ "acc": 0.9362399578094482,
112
+ "ece": 0.005877783937850665,
113
+ "classwise_ece": 0.0037371709357283678,
114
+ "nll": 0.15541012585163116,
115
+ "brier": 0.09056096524000168,
116
+ "mean_conf": 0.930762767791748,
117
+ "reliability": [
118
+ {
119
+ "lo": 0.2666666805744171,
120
+ "conf": 0.3104710280895233,
121
+ "acc": 0.3333333432674408,
122
+ "n": 9
123
+ },
124
+ {
125
+ "lo": 0.3333333432674408,
126
+ "conf": 0.3777979910373688,
127
+ "acc": 0.32258063554763794,
128
+ "n": 248
129
+ },
130
+ {
131
+ "lo": 0.40000003576278687,
132
+ "conf": 0.43695154786109924,
133
+ "acc": 0.4291091561317444,
134
+ "n": 797
135
+ },
136
+ {
137
+ "lo": 0.46666669845581055,
138
+ "conf": 0.510391354560852,
139
+ "acc": 0.509703516960144,
140
+ "n": 3710
141
+ },
142
+ {
143
+ "lo": 0.5333333015441895,
144
+ "conf": 0.56622713804245,
145
+ "acc": 0.5708888173103333,
146
+ "n": 5558
147
+ },
148
+ {
149
+ "lo": 0.5999999642372131,
150
+ "conf": 0.6333951950073242,
151
+ "acc": 0.6303596496582031,
152
+ "n": 5784
153
+ },
154
+ {
155
+ "lo": 0.6666666269302368,
156
+ "conf": 0.7004299759864807,
157
+ "acc": 0.7268487811088562,
158
+ "n": 6004
159
+ },
160
+ {
161
+ "lo": 0.7333333492279053,
162
+ "conf": 0.7674108147621155,
163
+ "acc": 0.7979346513748169,
164
+ "n": 6488
165
+ },
166
+ {
167
+ "lo": 0.800000011920929,
168
+ "conf": 0.8349699974060059,
169
+ "acc": 0.864363968372345,
170
+ "n": 8309
171
+ },
172
+ {
173
+ "lo": 0.8666666746139526,
174
+ "conf": 0.903027355670929,
175
+ "acc": 0.9281561374664307,
176
+ "n": 12555
177
+ },
178
+ {
179
+ "lo": 0.9333333373069763,
180
+ "conf": 0.9933969974517822,
181
+ "acc": 0.9946790933609009,
182
+ "n": 150538
183
+ }
184
+ ],
185
+ "acc_by_stage": {
186
+ "<2048": {
187
+ "acc": 0.8899039030075073,
188
+ "n": 8429
189
+ },
190
+ "2048-4096": {
191
+ "acc": 0.9120897650718689,
192
+ "n": 24525
193
+ },
194
+ "8192": {
195
+ "acc": 0.9290807843208313,
196
+ "n": 31275
197
+ },
198
+ "16384": {
199
+ "acc": 0.9418529868125916,
200
+ "n": 58782
201
+ },
202
+ ">=32768": {
203
+ "acc": 0.9476288557052612,
204
+ "n": 76989
205
+ }
206
+ }
207
+ },
208
+ "dag": {
209
+ "n": 23621,
210
+ "acc": 0.44477373361587524,
211
+ "ece": 0.21562865744482934,
212
+ "classwise_ece": 0.11597429518194752,
213
+ "nll": 2.2459492683410645,
214
+ "brier": 0.7532422542572021,
215
+ "mean_conf": 0.6604024171829224,
216
+ "reliability": [
217
+ {
218
+ "lo": 0.20000001788139343,
219
+ "conf": 0.2614020109176636,
220
+ "acc": 0.25,
221
+ "n": 12
222
+ },
223
+ {
224
+ "lo": 0.2666666805744171,
225
+ "conf": 0.31140047311782837,
226
+ "acc": 0.24229075014591217,
227
+ "n": 681
228
+ },
229
+ {
230
+ "lo": 0.3333333432674408,
231
+ "conf": 0.3702302873134613,
232
+ "acc": 0.31361210346221924,
233
+ "n": 2248
234
+ },
235
+ {
236
+ "lo": 0.40000003576278687,
237
+ "conf": 0.43376410007476807,
238
+ "acc": 0.3314346671104431,
239
+ "n": 2809
240
+ },
241
+ {
242
+ "lo": 0.46666669845581055,
243
+ "conf": 0.5007346272468567,
244
+ "acc": 0.37642714381217957,
245
+ "n": 2978
246
+ },
247
+ {
248
+ "lo": 0.5333333015441895,
249
+ "conf": 0.5658344626426697,
250
+ "acc": 0.39647752046585083,
251
+ "n": 2555
252
+ },
253
+ {
254
+ "lo": 0.5999999642372131,
255
+ "conf": 0.6301718950271606,
256
+ "acc": 0.44249120354652405,
257
+ "n": 1991
258
+ },
259
+ {
260
+ "lo": 0.6666666269302368,
261
+ "conf": 0.6996316313743591,
262
+ "acc": 0.4875076115131378,
263
+ "n": 1641
264
+ },
265
+ {
266
+ "lo": 0.7333333492279053,
267
+ "conf": 0.76542067527771,
268
+ "acc": 0.46928200125694275,
269
+ "n": 1351
270
+ },
271
+ {
272
+ "lo": 0.800000011920929,
273
+ "conf": 0.8332225680351257,
274
+ "acc": 0.5105690956115723,
275
+ "n": 1230
276
+ },
277
+ {
278
+ "lo": 0.8666666746139526,
279
+ "conf": 0.9017775058746338,
280
+ "acc": 0.5570417046546936,
281
+ "n": 1271
282
+ },
283
+ {
284
+ "lo": 0.9333333373069763,
285
+ "conf": 0.9865418672561646,
286
+ "acc": 0.6009476780891418,
287
+ "n": 4854
288
+ }
289
+ ],
290
+ "acc_by_stage": {
291
+ "<2048": {
292
+ "acc": 0.4209636151790619,
293
+ "n": 18327
294
+ },
295
+ "2048-4096": {
296
+ "acc": 0.521415650844574,
297
+ "n": 4973
298
+ },
299
+ "8192": {
300
+ "acc": 0.6168224215507507,
301
+ "n": 321
302
+ }
303
+ }
304
+ },
305
+ "dageval": {
306
+ "n": 199505,
307
+ "acc": 0.6101701855659485,
308
+ "ece": 0.27116093451079476,
309
+ "classwise_ece": 0.14033046305396282,
310
+ "nll": 3.204610824584961,
311
+ "brier": 0.6528977155685425,
312
+ "mean_conf": 0.8813311457633972,
313
+ "reliability": [
314
+ {
315
+ "lo": 0.20000001788139343,
316
+ "conf": 0.2623162865638733,
317
+ "acc": 0.0,
318
+ "n": 3
319
+ },
320
+ {
321
+ "lo": 0.2666666805744171,
322
+ "conf": 0.31312525272369385,
323
+ "acc": 0.1818181723356247,
324
+ "n": 110
325
+ },
326
+ {
327
+ "lo": 0.3333333432674408,
328
+ "conf": 0.3742755949497223,
329
+ "acc": 0.2713567912578583,
330
+ "n": 1194
331
+ },
332
+ {
333
+ "lo": 0.40000003576278687,
334
+ "conf": 0.4368848204612732,
335
+ "acc": 0.29916611313819885,
336
+ "n": 2878
337
+ },
338
+ {
339
+ "lo": 0.46666669845581055,
340
+ "conf": 0.506822943687439,
341
+ "acc": 0.34820833802223206,
342
+ "n": 7507
343
+ },
344
+ {
345
+ "lo": 0.5333333015441895,
346
+ "conf": 0.5662879943847656,
347
+ "acc": 0.3780185878276825,
348
+ "n": 9690
349
+ },
350
+ {
351
+ "lo": 0.5999999642372131,
352
+ "conf": 0.6332005262374878,
353
+ "acc": 0.4069898724555969,
354
+ "n": 9757
355
+ },
356
+ {
357
+ "lo": 0.6666666269302368,
358
+ "conf": 0.7001890540122986,
359
+ "acc": 0.44430702924728394,
360
+ "n": 9705
361
+ },
362
+ {
363
+ "lo": 0.7333333492279053,
364
+ "conf": 0.7671082615852356,
365
+ "acc": 0.49206504225730896,
366
+ "n": 10271
367
+ },
368
+ {
369
+ "lo": 0.800000011920929,
370
+ "conf": 0.834148108959198,
371
+ "acc": 0.5464140176773071,
372
+ "n": 11559
373
+ },
374
+ {
375
+ "lo": 0.8666666746139526,
376
+ "conf": 0.9023152589797974,
377
+ "acc": 0.5913816690444946,
378
+ "n": 15293
379
+ },
380
+ {
381
+ "lo": 0.9333333373069763,
382
+ "conf": 0.9915003776550293,
383
+ "acc": 0.7039197683334351,
384
+ "n": 121538
385
+ }
386
+ ],
387
+ "acc_by_stage": {
388
+ "<2048": {
389
+ "acc": 0.4269912540912628,
390
+ "n": 18895
391
+ },
392
+ "2048-4096": {
393
+ "acc": 0.572346031665802,
394
+ "n": 66956
395
+ },
396
+ "8192": {
397
+ "acc": 0.6439301371574402,
398
+ "n": 58379
399
+ },
400
+ "16384": {
401
+ "acc": 0.6899833083152771,
402
+ "n": 49178
403
+ },
404
+ ">=32768": {
405
+ "acc": 0.6262096166610718,
406
+ "n": 6097
407
+ }
408
+ }
409
+ }
410
+ }
eval_10k.jsonl ADDED
@@ -0,0 +1 @@
 
 
1
+ {"games": 10000, "mean_score": 164394.75, "median_score": 131064.0, "max_score": 835640, "mean_moves": 6475.5087890625, "moves_per_s": 64945.798002414296, "seconds": 997.0635359287262, "sym": false, "ply1": false, "rate_2048": 0.9657999873161316, "rate_4096": 0.8641999959945679, "rate_8192": 0.6326000094413757, "rate_16384": 0.3003999888896942, "rate_32768": 0.03909999877214432, "rate_65536": 0.0, "best_board": [[1, 6, 12, 15], [2, 5, 11, 14], [1, 8, 10, 13], [5, 1, 2, 3]], "t": 1792779363.1609044, "step": 25915, "ckpt": "/root/runs/final/reflex.pt"}
eval_ply1.jsonl ADDED
@@ -0,0 +1 @@
 
 
1
+ {"games": 128, "mean_score": 4189.34375, "median_score": 3164.0, "max_score": 15412, "mean_moves": 300.59375, "moves_per_s": 2112.761483349328, "seconds": 18.21123695373535, "sym": false, "ply1": true, "rate_2048": 0.0, "rate_4096": 0.0, "rate_8192": 0.0, "rate_16384": 0.0, "rate_32768": 0.0, "rate_65536": 0.0, "best_board": [[4, 5, 2, 1], [1, 2, 3, 5], [3, 1, 8, 10], [1, 2, 6, 9]], "t": 1792779427.8805845, "step": 25915, "ckpt": "/root/runs/final/reflex.pt"}
reflex.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8962a89c708a7fa1e2851527718a837c8885ef90ff90d3083f4c3b6bec247810
3
+ size 58080843
reflex/__init__.py ADDED
File without changes
reflex/__pycache__/__init__.cpython-314.pyc ADDED
Binary file (132 Bytes). View file
 
reflex/__pycache__/model.cpython-314.pyc ADDED
Binary file (9.49 kB). View file
 
reflex/env.py ADDED
@@ -0,0 +1,183 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Batched 2048 on torch tensors. Boards are (B,4,4) int64 tile exponents (0 = empty).
2
+
3
+ Action order everywhere in this repo: 0=UP 1=DOWN 2=LEFT 3=RIGHT.
4
+ Every move is reduced to "slide the rows left" on a transformed view and one
5
+ row lookup table (18^4 rows) does the work, so thousands of games step in parallel.
6
+ """
7
+ import torch
8
+
9
+ ACTIONS = ("UP", "DOWN", "LEFT", "RIGHT")
10
+ V = 18 # exponents 0..17 (131072 is the largest possible tile)
11
+ POW = torch.tensor([V**3, V**2, V, 1])
12
+
13
+
14
+ def _build_row_table():
15
+ n = V**4
16
+ new = torch.zeros(n, 4, dtype=torch.int64)
17
+ rew = torch.zeros(n, dtype=torch.int64)
18
+ for code in range(n):
19
+ row = [(code // V**(3 - i)) % V for i in range(4)]
20
+ tiles = [t for t in row if t]
21
+ out, r, i = [], 0, 0
22
+ while i < len(tiles):
23
+ if i + 1 < len(tiles) and tiles[i] == tiles[i + 1] and tiles[i] < V - 1:
24
+ out.append(tiles[i] + 1); r += 1 << (tiles[i] + 1); i += 2
25
+ else:
26
+ out.append(tiles[i]); i += 1
27
+ new[code] = torch.tensor(out + [0] * (4 - len(out)))
28
+ rew[code] = r
29
+ return new, rew
30
+
31
+
32
+ _TABLES = {}
33
+
34
+
35
+ def tables(device):
36
+ if device not in _TABLES:
37
+ if "cpu" not in _TABLES:
38
+ _TABLES["cpu"] = _build_row_table()
39
+ _TABLES[device] = tuple(t.to(device) for t in _TABLES["cpu"])
40
+ return _TABLES[device]
41
+
42
+
43
+ def _views(b): # (B,4,4) -> (B,4,4,4): each action as a left-slide view
44
+ t = b.transpose(-1, -2)
45
+ return torch.stack([t, t.flip(-1), b, b.flip(-1)], 1)
46
+
47
+
48
+ def _unview(v): # inverse of _views, v: (B,4,4,4)
49
+ return torch.stack([v[:, 0].transpose(-1, -2), v[:, 1].flip(-1).transpose(-1, -2),
50
+ v[:, 2], v[:, 3].flip(-1)], 1)
51
+
52
+
53
+ def all_afterstates(b):
54
+ """b (B,4,4) -> after (B,4,4,4), reward (B,4), legal (B,4) bool."""
55
+ new_t, rew_t = tables(str(b.device))
56
+ idx = (_views(b) * POW.to(b.device)).sum(-1) # (B,4,4)
57
+ after = _unview(new_t[idx])
58
+ reward = rew_t[idx].sum(-1)
59
+ legal = (after != b[:, None]).flatten(2).any(-1)
60
+ return after, reward, legal
61
+
62
+
63
+ def spawn(b, gen=None):
64
+ """Put a 2 (p=.9) or 4 at a random empty cell of every board that has one."""
65
+ flat = b.flatten(1).clone()
66
+ empty = flat == 0
67
+ r = torch.rand(flat.shape, device=b.device, generator=gen).masked_fill(~empty, -1)
68
+ pos = r.argmax(1)
69
+ val = 1 + (torch.rand(len(flat), device=b.device, generator=gen) < 0.1).long()
70
+ has = empty.any(1)
71
+ flat[has, pos[has]] = val[has]
72
+ return flat.view_as(b)
73
+
74
+
75
+ def reset(n, device="cpu", gen=None):
76
+ return spawn(spawn(torch.zeros(n, 4, 4, dtype=torch.int64, device=device), gen), gen)
77
+
78
+
79
+ # ---------------------------------------------------------------- features
80
+ def board_stats(b):
81
+ """Hand-made geometric features of boards (...,4,4) -> (...,8), roughly in [0,1]."""
82
+ f = b.float()
83
+ empties = (b == 0).flatten(-2).sum(-1).float() / 16
84
+ mx = f.flatten(-2).max(-1).values
85
+ corners = torch.stack([f[..., 0, 0], f[..., 0, 3], f[..., 3, 0], f[..., 3, 3]], -1)
86
+ max_in_corner = (corners.max(-1).values == mx).float()
87
+ dr, dc = f[..., :, 1:] - f[..., :, :-1], f[..., 1:, :] - f[..., :-1, :]
88
+ # monotonicity: per line, how much of the variation goes one way
89
+ def mono(d):
90
+ inc, dec = d.clamp(min=0).sum(-1), (-d).clamp(min=0).sum(-1)
91
+ tot = inc + dec
92
+ return torch.where(tot > 0, torch.maximum(inc, dec) / tot.clamp(min=1e-6), torch.ones_like(tot)).mean(-1)
93
+ mono_r, mono_c = mono(dr), mono(dc.transpose(-1, -2))
94
+ nz_r, nz_c = (b[..., :, 1:] > 0) & (b[..., :, :-1] > 0), (b[..., 1:, :] > 0) & (b[..., :-1, :] > 0)
95
+ smooth = -((dr.abs() * nz_r).sum((-1, -2)) + (dc.abs() * nz_c).sum((-1, -2))) / (24 * 17)
96
+ merges = (((dr == 0) & nz_r).sum((-1, -2)) + ((dc == 0) & nz_c).sum((-1, -2))).float() / 24
97
+ return torch.stack([empties, mx / 17, max_in_corner, mono_r, mono_c, smooth, merges,
98
+ f.flatten(-2).sum(-1) / (16 * 17)], -1)
99
+
100
+
101
+ N_ACT_FEATS = 2 + 8 + 4 # legal, reward, afterstate stats, "can merge again" per next action
102
+
103
+
104
+ def action_features(b):
105
+ """The 'enriched state': for each of the 4 moves, what the afterstate looks like.
106
+ Returns feats (B,4,N_ACT_FEATS), legal (B,4), after (B,4,4,4), reward (B,4)."""
107
+ after, reward, legal = all_afterstates(b)
108
+ B = b.shape[0]
109
+ stats = board_stats(after) # (B,4,8)
110
+ _, r2, l2 = all_afterstates(after.reshape(B * 4, 4, 4)) # one more ply, no spawn
111
+ look = (r2.float().log1p() / 17 * l2).view(B, 4, 4)
112
+ feats = torch.cat([legal.float()[..., None], (reward.float().log1p() / 17)[..., None], stats, look], -1)
113
+ return feats * legal[..., None], legal, after, reward
114
+
115
+
116
+ # ---------------------------------------------------------------- symmetry (D4)
117
+ # Each transform maps a board to a board; PERM[k][a] = the action that a becomes.
118
+ def sym(b, k):
119
+ if k >= 4:
120
+ b = b.flip(-1)
121
+ return torch.rot90(b, k % 4, (-2, -1))
122
+
123
+
124
+ def _action_perm():
125
+ perms = []
126
+ probe = torch.arange(16).view(1, 4, 4)
127
+ for k in range(8):
128
+ # find where each action maps by checking the transformed "direction" of a gradient board
129
+ p = []
130
+ for a in range(4):
131
+ g = torch.zeros(1, 4, 4, dtype=torch.int64)
132
+ # board whose only legal move is direction a: a lone tile at the far side
133
+ if a == 0: g[0, 3, 1] = 1
134
+ if a == 1: g[0, 0, 1] = 1
135
+ if a == 2: g[0, 1, 3] = 1
136
+ if a == 3: g[0, 1, 0] = 1
137
+ after, _, _ = all_afterstates(g)
138
+ target = sym(after[:, a], k)
139
+ sb = sym(g, k)
140
+ after2, _, legal2 = all_afterstates(sb)
141
+ hits = [c for c in range(4) if legal2[0, c] and torch.equal(after2[0, c], target[0])]
142
+ p.append(hits[0])
143
+ perms.append(p)
144
+ del probe
145
+ return torch.tensor(perms)
146
+
147
+
148
+ SYM_PERM = None
149
+
150
+
151
+ def sym_perm():
152
+ global SYM_PERM
153
+ if SYM_PERM is None:
154
+ SYM_PERM = _action_perm()
155
+ return SYM_PERM
156
+
157
+
158
+ def score_of(b):
159
+ """Lower bound score implied by a board (assumes every spawn was a 2)."""
160
+ f = b.float()
161
+ return torch.where(b > 1, (f - 1) * torch.exp2(f), torch.zeros_like(f)).flatten(1).sum(1)
162
+
163
+
164
+ if __name__ == "__main__":
165
+ b = torch.tensor([[[1, 1, 2, 2], [0, 0, 0, 0], [3, 0, 3, 0], [1, 2, 1, 2]]])
166
+ after, rew, legal = all_afterstates(b)
167
+ assert after[0, 2].tolist() == [[2, 3, 0, 0], [0, 0, 0, 0], [4, 0, 0, 0], [1, 2, 1, 2]], after[0, 2]
168
+ assert after[0, 3].tolist() == [[0, 0, 2, 3], [0, 0, 0, 0], [0, 0, 0, 4], [1, 2, 1, 2]]
169
+ assert rew[0, 2] == 4 + 8 + 16 and rew[0, 3] == 4 + 8 + 16
170
+ assert after[0, 0, :, 0].tolist() == [1, 3, 1, 0] and legal.all()
171
+ P = sym_perm()
172
+ assert all(sorted(p.tolist()) == [0, 1, 2, 3] for p in P)
173
+ # symmetry consistency on random boards
174
+ g = torch.Generator().manual_seed(0)
175
+ bs = torch.randint(0, 6, (64, 4, 4), generator=g)
176
+ a0, r0, _ = all_afterstates(bs)
177
+ for k in range(8):
178
+ ak, rk, _ = all_afterstates(sym(bs, k))
179
+ for a in range(4):
180
+ assert torch.equal(ak[:, P[k, a]], sym(a0[:, a], k)) and torch.equal(rk[:, P[k, a]], r0[:, a])
181
+ f, l, _, _ = action_features(bs)
182
+ assert f.shape == (64, 4, N_ACT_FEATS) and torch.isfinite(f).all()
183
+ print("env ok", P.tolist())
reflex/evaluate.py ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Play many full games in parallel on GPU with the pure policy (one forward pass
2
+ per move, no search). Watches a checkpoint and re-evaluates whenever it changes.
3
+
4
+ python -m reflex.evaluate --ckpt runs/reflex/latest.pt --games 1024 [--watch] [--sym]
5
+ """
6
+ import argparse, json, os, time
7
+ import torch
8
+ import torch.nn.functional as F
9
+
10
+ from .env import action_features, reset, score_of, spawn, sym, sym_perm
11
+ from .model import load
12
+
13
+ TILES = [2048, 4096, 8192, 16384, 32768, 65536]
14
+
15
+
16
+ @torch.no_grad()
17
+ def policy(model, b, use_sym=False):
18
+ feats, legal, after, reward = action_features(b)
19
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
20
+ logits, _ = model(b, feats, legal)
21
+ p = F.softmax(logits, -1)
22
+ if use_sym: # average the distribution over the 8 symmetric views (still one batched pass)
23
+ P = sym_perm().to(b.device)
24
+ bs = torch.cat([sym(b, k) for k in range(1, 8)])
25
+ f2, l2, _, _ = action_features(bs)
26
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
27
+ lg, _ = model(bs, f2, l2)
28
+ p2 = F.softmax(lg, -1).view(7, len(b), 4)
29
+ for k in range(1, 8):
30
+ p = p + p2[k - 1][:, P[k]]
31
+ p = p / 8
32
+ return p.nan_to_num(0), legal, after, reward
33
+
34
+
35
+ @torch.no_grad()
36
+ def lookahead(model, b, chunk=65536):
37
+ """1-ply expectimax on the value head: Q(s,a) = r_a + E[V(s')] with the expectation exact
38
+ over every spawn (a 2 w.p. .9 or a 4 w.p. .1 in each empty cell of the afterstate).
39
+ Not search-free: about 8x(empty cells) forward passes per move. Returns q (n,4), -inf if illegal."""
40
+ _, legal, after, reward = action_features(b)
41
+ flat = after.flatten(2) # (n,4,16)
42
+ empty = flat == 0
43
+ eye = torch.eye(16, dtype=flat.dtype, device=b.device)
44
+ child = flat[:, :, None, None] + eye[:, None] * torch.tensor([1, 2], device=b.device)[:, None] # (n,4,16,2,16)
45
+ w = empty[..., None] * torch.tensor([.9, .1], device=b.device) / empty.sum(-1, keepdim=True)[..., None].clamp(min=1)
46
+ valid = (w > 0) & legal[:, :, None, None]
47
+ cb = child[valid].view(-1, 4, 4)
48
+ vals = []
49
+ for i in range(0, len(cb), chunk):
50
+ f, l, _, _ = action_features(cb[i:i + chunk])
51
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=b.is_cuda):
52
+ _, v = model(cb[i:i + chunk], f, l)
53
+ vals.append(torch.expm1(10 * v.float()).clamp(min=0)) # value head predicts log1p(return)/10
54
+ ev = torch.zeros(valid.shape, device=b.device)
55
+ ev[valid] = torch.cat(vals) if vals else ev[valid]
56
+ q = reward.float() + (ev * w).sum((2, 3))
57
+ return q.masked_fill(~legal, float("-inf")), legal, after, reward
58
+
59
+
60
+ @torch.no_grad()
61
+ def play(model, n, device="cuda", use_sym=False, live=None, seed=0, ply1=False):
62
+ gen = torch.Generator(device=device).manual_seed(seed)
63
+ b = reset(n, device, gen)
64
+ score = torch.zeros(n, dtype=torch.int64, device=device)
65
+ moves = torch.zeros(n, dtype=torch.int64, device=device)
66
+ alive = torch.ones(n, dtype=torch.bool, device=device)
67
+ t0, steps = time.time(), 0
68
+ while alive.any():
69
+ idx = alive.nonzero().squeeze(1)
70
+ if ply1:
71
+ p, legal, after, reward = lookahead(model, b[idx])
72
+ else:
73
+ p, legal, after, reward = policy(model, b[idx], use_sym)
74
+ dead = ~legal.any(1)
75
+ a = p.argmax(1)
76
+ nb = after[torch.arange(len(idx)), a]
77
+ r = reward[torch.arange(len(idx)), a]
78
+ live_idx = idx[~dead]
79
+ b[live_idx] = spawn(nb[~dead], gen)
80
+ score[live_idx] += r[~dead]
81
+ moves[live_idx] += 1
82
+ alive[idx[dead]] = False
83
+ steps += 1
84
+ if live and steps % 25 == 0:
85
+ j = int(score.argmax())
86
+ json.dump(dict(board=b[j].tolist(), score=int(score[j]), moves=int(moves[j]),
87
+ alive=int(alive.sum()), games=n, step=steps), open(live + ".tmp", "w"))
88
+ os.replace(live + ".tmp", live)
89
+ dt = time.time() - t0
90
+ mx = b.flatten(1).max(1).values
91
+ res = dict(games=n, mean_score=score.float().mean().item(), median_score=score.float().median().item(),
92
+ max_score=int(score.max()), mean_moves=moves.float().mean().item(),
93
+ moves_per_s=moves.sum().item() / dt, seconds=dt, sym=use_sym, ply1=ply1)
94
+ for t in TILES:
95
+ res[f"rate_{t}"] = (mx >= t.bit_length() - 1).float().mean().item()
96
+ best = int(score.argmax())
97
+ res["best_board"] = b[best].tolist()
98
+ return res
99
+
100
+
101
+ def main():
102
+ ap = argparse.ArgumentParser()
103
+ ap.add_argument("--ckpt", default="/root/runs/reflex/latest.pt")
104
+ ap.add_argument("--games", type=int, default=1024)
105
+ ap.add_argument("--sym", action="store_true")
106
+ ap.add_argument("--ply1", action="store_true", help="1-ply expectimax on the value head (not search-free)")
107
+ ap.add_argument("--watch", action="store_true")
108
+ ap.add_argument("--out", default="/root/runs/eval.jsonl")
109
+ ap.add_argument("--seed", type=int, default=0)
110
+ ap.add_argument("--min-interval", type=float, default=0, help="watch mode: seconds between evaluations "
111
+ "(evaluation shares the GPU with training)")
112
+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
113
+ args = ap.parse_args()
114
+ last, best = None, -1
115
+ while True:
116
+ if os.path.exists(args.ckpt) and os.path.getmtime(args.ckpt) != last:
117
+ last, last_run = os.path.getmtime(args.ckpt), time.time()
118
+ model = load(args.ckpt, args.device)
119
+ step = torch.load(args.ckpt, weights_only=False, map_location="cpu").get("step", 0)
120
+ res = play(model, args.games, device=args.device, use_sym=args.sym, seed=args.seed, ply1=args.ply1,
121
+ live=os.path.join(os.path.dirname(args.out), "live.json"))
122
+ res.update(t=time.time(), step=step, ckpt=args.ckpt)
123
+ if args.watch and res["mean_score"] > best: # keep the best checkpoint seen so far
124
+ best = res["mean_score"]
125
+ torch.save(dict(cfg=model.cfg, model=model.state_dict(), step=step, eval=res),
126
+ os.path.join(os.path.dirname(args.ckpt), "best.pt"))
127
+ with open(args.out, "a") as f:
128
+ f.write(json.dumps(res) + "\n")
129
+ print(json.dumps({k: v for k, v in res.items() if k != "best_board"}), flush=True)
130
+ if not args.watch or os.path.exists("/root/runs/WRAP_UP"):
131
+ break
132
+ time.sleep(max(10, args.min_interval - (time.time() - last_run)) if last else 10)
133
+
134
+
135
+ if __name__ == "__main__":
136
+ main()
reflex/model.py ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Reflex: a non-autoregressive transformer that maps a 4x4 board to a calibrated
2
+ distribution over the 4 moves in one forward pass.
3
+
4
+ Tokens: 16 cell tokens (tile embedding + row/col embeddings) + 4 action tokens
5
+ (the enriched afterstate features of each move) + 1 value token. A bidirectional
6
+ pre-norm encoder (fused qkv + scaled_dot_product_attention) mixes them; each action
7
+ token emits one logit, illegal moves are masked to -inf, so the output is always a
8
+ valid distribution over legal moves only.
9
+ """
10
+ import torch
11
+ import torch.nn as nn
12
+ import torch.nn.functional as F
13
+
14
+ from .env import ACTIONS, N_ACT_FEATS, V, action_features
15
+
16
+
17
+ class Block(nn.Module):
18
+ def __init__(self, d, heads, ff):
19
+ super().__init__()
20
+ self.h = heads
21
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
22
+ self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
23
+ self.ff = nn.Sequential(nn.Linear(d, ff * d), nn.GELU(), nn.Linear(ff * d, d))
24
+
25
+ def forward(self, x):
26
+ B, T, D = x.shape
27
+ q, k, v = self.qkv(self.n1(x)).view(B, T, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
28
+ x = x + self.proj(F.scaled_dot_product_attention(q, k, v).transpose(1, 2).reshape(B, T, D))
29
+ return x + self.ff(self.n2(x))
30
+
31
+
32
+ class Reflex(nn.Module):
33
+ def __init__(self, d=256, layers=8, heads=8, ff=4):
34
+ super().__init__()
35
+ self.cfg = dict(d=d, layers=layers, heads=heads, ff=ff)
36
+ self.tile = nn.Embedding(V, d)
37
+ self.row = nn.Parameter(torch.randn(4, 1, d) * 0.02)
38
+ self.col = nn.Parameter(torch.randn(1, 4, d) * 0.02)
39
+ self.act_in = nn.Sequential(nn.Linear(N_ACT_FEATS, d), nn.GELU(), nn.Linear(d, d))
40
+ self.act_id = nn.Parameter(torch.randn(4, d) * 0.02)
41
+ self.val_tok = nn.Parameter(torch.randn(1, d) * 0.02)
42
+ self.blocks = nn.ModuleList(Block(d, heads, ff) for _ in range(layers))
43
+ self.norm = nn.LayerNorm(d)
44
+ self.pi = nn.Linear(d, 1)
45
+ self.v = nn.Sequential(nn.Linear(d, d), nn.GELU(), nn.Linear(d, 1))
46
+
47
+ def forward(self, b, feats=None, legal=None):
48
+ """b (B,4,4) exponents. Returns logits (B,4) with illegal = -inf, value (B,)."""
49
+ if feats is None:
50
+ feats, legal, _, _ = action_features(b)
51
+ B = b.shape[0]
52
+ cells = (self.tile(b) + self.row + self.col).flatten(1, 2) # (B,16,d)
53
+ acts = self.act_in(feats) + self.act_id # (B,4,d)
54
+ x = torch.cat([cells, acts, self.val_tok.expand(B, 1, -1).to(cells.dtype)], 1)
55
+ for blk in self.blocks:
56
+ x = blk(x)
57
+ x = self.norm(x)
58
+ logits = self.pi(x[:, 16:20]).squeeze(-1).float().masked_fill(~legal, float("-inf"))
59
+ return logits, self.v(x[:, 20]).squeeze(-1).float()
60
+
61
+ @torch.no_grad()
62
+ def decide(self, board):
63
+ """Type-safe single-board API: {'UP': p, 'DOWN': p, 'LEFT': p, 'RIGHT': p}."""
64
+ b = torch.as_tensor(board, dtype=torch.int64).view(1, 4, 4)
65
+ logits, _ = self(b.to(next(self.parameters()).device))
66
+ p = F.softmax(logits, -1)[0].nan_to_num(0).tolist()
67
+ return dict(zip(ACTIONS, p))
68
+
69
+
70
+ def load(path, device="cpu"):
71
+ ck = torch.load(path, map_location=device, weights_only=False)
72
+ m = Reflex(**ck["cfg"]).to(device)
73
+ m.load_state_dict(ck["model"])
74
+ return m.eval()
75
+
76
+
77
+ if __name__ == "__main__":
78
+ import time
79
+ m = Reflex().eval()
80
+ print(f"params {sum(p.numel() for p in m.parameters())/1e6:.2f}M")
81
+ b = torch.tensor([[1, 1, 2, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 3]])
82
+ d = m.decide(b)
83
+ assert abs(sum(d.values()) - 1) < 1e-5 and d["UP"] > 0
84
+ t = time.perf_counter()
85
+ for _ in range(50):
86
+ m.decide(b)
87
+ print("decision", d, f"{(time.perf_counter()-t)/50*1000:.2f} ms/forward on cpu")
reflex/play.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Watch Reflex play one game in the terminal on a laptop CPU, one forward pass per move.
2
+
3
+ python -m reflex.play --ckpt reflex.pt [--seed 0] [--every 500]
4
+ """
5
+ import argparse, time
6
+ import torch
7
+
8
+ from .env import ACTIONS, reset, spawn
9
+ from .evaluate import policy
10
+ from .model import load
11
+
12
+
13
+ def show(b, score, moves, p):
14
+ tiles = [[(1 << int(x)) if x else 0 for x in row] for row in b.tolist()]
15
+ print(f"\nmove {moves} score {score} " + " ".join(f"{a} {q:.2f}" for a, q in zip(ACTIONS, p.tolist())))
16
+ for row in tiles:
17
+ print(" ".join(f"{t:>6}" if t else " ." for t in row))
18
+
19
+
20
+ def main():
21
+ ap = argparse.ArgumentParser()
22
+ ap.add_argument("--ckpt", default="reflex.pt")
23
+ ap.add_argument("--seed", type=int, default=0)
24
+ ap.add_argument("--every", type=int, default=500, help="print the board every N moves")
25
+ args = ap.parse_args()
26
+ torch.set_num_threads(max(1, torch.get_num_threads()))
27
+ model = load(args.ckpt, "cpu")
28
+ gen = torch.Generator().manual_seed(args.seed)
29
+ b, score, moves, t = reset(1, "cpu", gen), 0, 0, 0.0
30
+ policy(model, b) # warm-up: builds the move lookup table once
31
+ while True:
32
+ s = time.perf_counter()
33
+ p, legal, after, reward = policy(model, b)
34
+ t += time.perf_counter() - s
35
+ if not legal.any():
36
+ break
37
+ a = int(p.argmax(1))
38
+ if moves % args.every == 0:
39
+ show(b[0], score, moves, p[0])
40
+ score += int(reward[0, a]); moves += 1
41
+ b = spawn(after[:, a], gen)
42
+ show(b[0], score, moves, p[0])
43
+ print(f"\ngame over: score {score}, {moves} moves, max tile {1 << int(b.max())}, "
44
+ f"{t / max(moves, 1) * 1000:.2f} ms per move on cpu (features + forward)")
45
+
46
+
47
+ if __name__ == "__main__":
48
+ main()