| --- |
| license: apache-2.0 |
| tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe] |
| --- |
| # TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10) |
|
|
| VQ-VAE OPTIMIZATION ablation (codebook health + next-code objective vs raw patches) on a ~3.6M-param decoder-only patched transformer |
| trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from |
| automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long |
| pool uses the cheap families only) with contiguous patch |
| masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at |
| time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only |
| the CONTEXT tokenizer varies: |
|
|
| | arm | tokenizer | history | ctx tokens | |
| |-----|-----------|---------|-----------| |
| | T0 | fixed-32 (control) | 4,096 | 128 | |
| | T1 | pyramid, iso-context | 4,096 | 44 | |
| | T2 | pyramid, iso-token | 16,384 | 128 | |
| | T3 | adaptive equal-surprise | 16,384 | 128 | |
|
|
| Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt` |
| (rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS + |
| long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT |
| the full leaderboard; treat numbers as ablation-internal, not comparable to |
| published GIFT scores. Generated by the v10 experiment notebook. |
|
|
| ## Results |
| ``` |
| === transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) === |
| gm_avg2 gm_avg2_std gm@7500 gm@15000 gm@22500 gm@final gm_final_std long_season params_m perp probe_amp n |
| arm |
| R3_k4096_nat 0.1795 NaN 0.2066 0.1889 0.1840 0.1750 NaN 0.2512 4.2743 1597.8119 0.2750 1 |
| R4_softce_nat 0.1817 0.0048 0.2018 0.1960 0.1834 0.1800 0.0099 0.2508 3.8780 558.6021 0.2380 2 |
| N3_vqce_nobonus_nat 0.1835 0.0021 0.1987 0.1881 0.1883 0.1787 0.0034 0.2485 3.8780 555.6374 0.1750 3 |
| R1_w01_nat 0.1836 NaN 0.1908 0.1854 0.1862 0.1809 NaN 0.2538 3.8780 570.1680 0.2180 1 |
| R2_k2048_nat 0.1864 0.0009 0.2180 0.1911 0.1898 0.1829 0.0035 0.2543 4.0101 970.0609 0.2610 3 |
| R5_horizonce_nat 0.1879 0.0015 0.1976 0.1888 0.1901 0.1858 0.0034 0.2527 4.0430 547.4886 0.1997 3 |
| N1_vqce_nat 0.1909 0.0109 0.2121 0.2052 0.1983 0.1835 0.0049 0.2550 3.8780 620.2301 0.2280 3 |
| N5_vqce_w05_nat 0.1959 NaN 0.2092 0.2005 0.1968 0.1949 NaN 0.2593 3.8780 598.1758 0.2410 1 |
| R6_k2048_hz_nat 0.1991 0.0074 0.2010 0.1928 0.2061 0.1921 0.0045 0.2540 4.3072 949.2821 0.2677 3 |
| N0_raw_nat 0.2037 0.0042 0.2011 0.1938 0.2014 0.2061 0.0048 0.2635 3.5802 NaN 0.2253 3 |
| N2_vqce_multi_nat 0.2039 0.0189 0.1913 0.1943 0.2154 0.1925 0.0034 0.2598 4.2743 597.4680 0.2235 2 |
| P0_fullzone_nat 0.2131 NaN NaN NaN NaN 0.2131 NaN 0.2832 3.5802 NaN NaN 1 |
| N4_rawce_nat 0.2404 0.0030 0.2043 0.2293 0.2371 0.2437 0.0028 0.2832 3.7452 461.1100 0.3755 2 |
| |
| --- tripwire checks --- |
| R3_k4096_nat vs R4_softce_nat: final gap=0.0050 -> UNRESOLVED (< 2*sigma) |
| R4_softce_nat vs N3_vqce_nobonus_nat: final gap=-0.0013 -> UNRESOLVED (< 2*sigma) |
| N3_vqce_nobonus_nat vs R1_w01_nat: final gap=0.0022 -> UNRESOLVED (< 2*sigma) |
| R1_w01_nat vs R2_k2048_nat: final gap=0.0020 -> UNRESOLVED (< 2*sigma) |
| R2_k2048_nat vs R5_horizonce_nat: final gap=0.0028 -> UNRESOLVED (< 2*sigma) |
| R5_horizonce_nat vs N1_vqce_nat: final gap=-0.0022 -> UNRESOLVED (< 2*sigma) |
| N1_vqce_nat vs N5_vqce_w05_nat: final gap=0.0114 -> UNRESOLVED (< 2*sigma) |
| N5_vqce_w05_nat vs R6_k2048_hz_nat: final gap=-0.0029 -> UNRESOLVED (< 2*sigma) |
| R6_k2048_hz_nat vs N0_raw_nat: final gap=0.0140 -> UNRESOLVED (< 2*sigma) |
| N0_raw_nat vs N2_vqce_multi_nat: final gap=-0.0136 -> UNRESOLVED (< 2*sigma) |
| N2_vqce_multi_nat vs P0_fullzone_nat: final gap=0.0207 -> RESOLVED |
| P0_fullzone_nat vs N4_rawce_nat: final gap=0.0306 -> RESOLVED |
| N0_raw_nat: TRANSFER INVERSION — best at ckpt #2 (0.1938) vs final (0.2061); synthetic overfit persists |
| N2_vqce_multi_nat: TRANSFER INVERSION — best at ckpt #1 (0.1913) vs final (0.1925); synthetic overfit persists |
| N4_rawce_nat: TRANSFER INVERSION — best at ckpt #1 (0.2043) vs final (0.2437); synthetic overfit persists |
| |
| ``` |
|
|
