license: apache-2.0
library_name: tabicl
tags:
- tabular
- tabular-classification
- in-context-learning
- foundation-model
TabICLv2 replication (classifier only)
This release accompanies work focusing on speeding up stage 1 of TabICLv2 pretraining.
This stage-1 trainer runs at 1.27 s/step vs 3.8 s/step for the official tabiclv2 trainer on single H100, completing the 500K-step stage-1 recipe in ~7 days instead of ~22 H100-days. Prior generation on a separate CPU with a deterministic, replayable stream (bit-exact across restarts) to avoid CPU bottlenecks.
Stages 2 and 3 were then run to completion so the stage-1 result can be judged by a full, evaluable model.
| stage | recipe | wall time (1× H100) | official (same hw, measured/est.) |
|---|---|---|---|
| 1 | 500K steps, 1,024 rows/dataset, LR 8e-4 | ~7 days (1.27 s/step) | ~22 days (3.8 s/step, measured) |
| 2 | 40K steps, 400–10,240 rows log-uniform, LR 1e-4 | ~2 days | — |
| 3 | 10K steps, 400–60,000 rows log-uniform, LR 2e-5 | ~2.3 days | — |
Classification only (max_classes=10); the regressor was not trained.
Quality (same-harness comparisons vs the released tabicl-classifier-v2)
| benchmark | ours | reference (released v2) |
|---|---|---|
| TabArena-Lite | Elo 1537.6 | Elo 1557.7 |
| TALENT (181 clf datasets), mean acc | 0.8400 (W/T/L 50/30/101) | 0.8426 |
| Large-dataset suite (15–48K rows), mean acc | 0.9133 | 0.9204 |
Recipe deviations from the official scripts (documented, not endorsed)
- bf16 autocast w/ fp32 stability islands vs tf32
- plain WD 0.1 vs 0.01
- ssmax/embeddings/norms/heads on aux AdamW vs all-in-one Muon group
- warmup 2% vs 1%
- cosine floor 0 vs 1e-7
- min_features 2 vs 1
- micro-batch 8 vs 4
- stage-2/3 train fraction exactly 0.80 vs 0.79–0.81
Checkpoints
| file | stage | steps | note |
|---|---|---|---|
| stage1/step-{50000..500000}.ckpt | 1 | 50K–500K | 6 snapshots |
| stage2/step-{10000..40000}.ckpt | 2 | 10K–40K | 4 snapshots |
| stage3/step-{2500..10000}.ckpt | 3 | 2.5K–10K | stage3/step-10000.ckpt = final model |
Research release from Nolano AI (Apache-2.0). Training/evaluation code and replication checkpoints by Ayush Kaushal, accompanying a stage-1 training-efficiency study. These are research artifacts, not a supported product and no maintenance or support is implied.