| --- |
| license: apache-2.0 |
| tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe] |
| --- |
| # TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10) |
|
|
| VQ-VAE OPTIMIZATION ablation (codebook health + next-code objective vs raw patches) on a ~3.6M-param decoder-only patched transformer |
| trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from |
| automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long |
| pool uses the cheap families only) with contiguous patch |
| masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at |
| time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only |
| the CONTEXT tokenizer varies: |
|
|
| | arm | tokenizer | history | ctx tokens | |
| |-----|-----------|---------|-----------| |
| | T0 | fixed-32 (control) | 4,096 | 128 | |
| | T1 | pyramid, iso-context | 4,096 | 44 | |
| | T2 | pyramid, iso-token | 16,384 | 128 | |
| | T3 | adaptive equal-surprise | 16,384 | 128 | |
|
|
| Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt` |
| (rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS + |
| long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT |
| the full leaderboard; treat numbers as ablation-internal, not comparable to |
| published GIFT scores. Generated by the v10 experiment notebook. |
|
|
| ## Results |
| ``` |
| === transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) === |
| gm@12500 gm@25000 gm@37500 gm@50000 gm@62500 gm@75000 gm@87500 gm@final gm_final_std long_season params_m perp probe_amp n |
| arm |
| Q0_n3_nat100k 0.2333 0.1937 0.1953 0.1961 0.1892 0.1948 0.1850 0.1854 NaN 0.2503 3.878 648.0925 0.231 1 |
| Q1_n3_tpfnfull100k 0.2091 0.2170 0.2040 0.2374 0.2424 0.2183 0.2124 0.2103 NaN 0.2681 3.878 408.1310 0.382 1 |
| |
| --- tripwire checks --- |
| Q0_n3_nat100k vs Q1_n3_tpfnfull100k: final gap=0.0249 -> sigma unknown (single seed) |
| Q0_n3_nat100k: TRANSFER INVERSION — best at ckpt #7 (0.1850) vs final (0.1854); synthetic overfit persists |
| Q1_n3_tpfnfull100k: TRANSFER INVERSION — best at ckpt #3 (0.2040) vs final (0.2103); synthetic overfit persists |
| |
| ``` |
|
|
