File size: 2,218 Bytes
1215447 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
license: apache-2.0
tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe]
---
# TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10)
Context-tokenizer ablation on a ~3.6M-param decoder-only patched transformer
trained on a TempoPFN-style synthetic prior ensemble with contiguous patch
masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at
time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only
the CONTEXT tokenizer varies:
| arm | tokenizer | history | ctx tokens |
|-----|-----------|---------|-----------|
| T0 | fixed-32 (control) | 4,096 | 128 |
| T1 | pyramid, iso-context | 4,096 | 44 |
| T2 | pyramid, iso-token | 16,384 | 128 |
| T3 | adaptive equal-surprise | 16,384 | 128 |
Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt`
(rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS +
long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT
the full leaderboard; treat numbers as ablation-internal, not comparable to
published GIFT scores. Generated by the v10 experiment notebook.
## Results
```
gm_crps_mean gm_crps_std gm_crps_15k long_season other probe_r probe_amp n
arm
T0_fixed 0.1949 0.0113 0.1875 0.2570 0.1352 -0.0067 0.2647 3
T1_pyr_ctx 0.2070 0.0089 0.2015 0.2673 0.1473 -0.0020 0.3043 3
T2_pyr_tok 0.2166 0.0143 0.2030 0.2664 0.1647 0.0000 0.2563 3
T3_adaptive 0.2324 0.0145 0.2175 0.2588 0.2021 0.0057 0.2533 3
--- tripwire checks ---
seed noise floor (max arm sigma): 0.0145
T0_fixed vs T1_pyr_ctx: gap=0.0121 -> UNRESOLVED (< 2*sigma) — add seeds/steps
T1_pyr_ctx vs T2_pyr_tok: gap=0.0095 -> UNRESOLVED (< 2*sigma) — add seeds/steps
T2_pyr_tok vs T3_adaptive: gap=0.0158 -> UNRESOLVED (< 2*sigma) — add seeds/steps
rank stable 15k->30k: ['T0_fixed', 'T1_pyr_ctx', 'T2_pyr_tok', 'T3_adaptive']
H1 compression tax (T1 vs T0): +6.23% (FAILS H1 gate — read T2 as gain minus tax)
```
|