| --- |
| license: apache-2.0 |
| library_name: tabicl |
| tags: |
| - tabular |
| - tabular-classification |
| - in-context-learning |
| - foundation-model |
| --- |
| |
| # TabICLv2 replication (classifier only) |
|
|
| This release accompanies work focusing on **speeding up stage 1 of TabICLv2 pretraining**. |
|
|
| This stage-1 trainer runs at **1.27 s/step vs 3.8 s/step** for the official tabiclv2 trainer on [single H100](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nccadsh100v5-series?tabs=sizebasic#:~:text=Standard_NCC40ads_H100_v5), completing the 500K-step stage-1 recipe in **~7 days instead of ~22 H100-days**. Prior generation on a [separate CPU](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/compute-optimized/faldsv7-series?tabs=sizebasic#:~:text=96-,Standard_F64alds_v7,-64) with a deterministic, replayable stream (bit-exact across restarts) to avoid CPU bottlenecks. |
|
|
| Stages 2 and 3 were then run to completion so the stage-1 result can be judged by a full, evaluable model. |
|
|
| | stage | recipe | wall time (1× H100) | official (same hw, measured/est.) | |
| |---|---|---|---| |
| | **1** | 500K steps, 1,024 rows/dataset, LR 8e-4 | **~7 days (1.27 s/step)** | ~22 days (3.8 s/step, measured) | |
| | 2 | 40K steps, 400–10,240 rows log-uniform, LR 1e-4 | ~2 days | — | |
| | 3 | 10K steps, 400–60,000 rows log-uniform, LR 2e-5 | ~2.3 days | — | |
|
|
| **Classification only** (`max_classes=10`); the regressor was not trained. |
|
|
| ## Quality (same-harness comparisons vs the released tabicl-classifier-v2) |
|
|
| | benchmark | ours | reference (released v2) | |
| |---|---|---| |
| | TabArena-Lite | Elo 1537.6 | Elo 1557.7 | |
| | TALENT (181 clf datasets), mean acc | 0.8400 (W/T/L 50/30/101) | 0.8426 | |
| | Large-dataset suite (15–48K rows), mean acc | 0.9133 | 0.9204 | |
|
|
| ## Recipe deviations from the official scripts (documented, not endorsed) |
| * bf16 autocast w/ fp32 stability islands vs tf32 |
| * plain WD 0.1 vs 0.01 |
| * ssmax/embeddings/norms/heads on aux AdamW vs all-in-one Muon group |
| * warmup 2% vs 1% |
| * cosine floor 0 vs 1e-7 |
| * min_features 2 vs 1 |
| * micro-batch 8 vs 4 |
| * stage-2/3 train fraction exactly 0.80 vs 0.79–0.81 |
| |
| |
| ## Checkpoints |
| | file | stage | steps | note | |
| |---|---|---|---| |
| | stage1/step-{50000..500000}.ckpt | 1 | 50K–500K | 6 snapshots | |
| | stage2/step-{10000..40000}.ckpt | 2 | 10K–40K | 4 snapshots | |
| | stage3/step-{2500..10000}.ckpt | 3 | 2.5K–10K | stage3/step-10000.ckpt = final model | |
| |
| ----- |
| |
| Research release from Nolano AI (Apache-2.0). Training/evaluation code and replication checkpoints by Ayush Kaushal, accompanying a stage-1 training-efficiency study. These are research artifacts, not a supported product and no maintenance or support is implied. |
| |
