README.md

---
license: apache-2.0
library_name: tabicl
tags:
  - tabular
  - tabular-classification
  - in-context-learning
  - foundation-model
---

# TabICLv2 replication (classifier only)

This release accompanies work focusing on **speeding up stage 1 of TabICLv2 pretraining**.

This stage-1 trainer runs at **1.27 s/step vs 3.8 s/step** for the official tabiclv2 trainer on [single H100](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nccadsh100v5-series?tabs=sizebasic#:~:text=Standard_NCC40ads_H100_v5), completing the 500K-step stage-1 recipe in **~7 days instead of ~22 H100-days**. Prior generation on a [separate CPU](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/compute-optimized/faldsv7-series?tabs=sizebasic#:~:text=96-,Standard_F64alds_v7,-64) with a deterministic, replayable stream (bit-exact across restarts) to avoid CPU bottlenecks.

Stages 2 and 3 were then run to completion so the stage-1 result can be judged by a full, evaluable model.

| stage | recipe | wall time (1× H100) | official (same hw, measured/est.) |
|---|---|---|---|
| **1** | 500K steps, 1,024 rows/dataset, LR 8e-4 | **~7 days (1.27 s/step)** | ~22 days (3.8 s/step, measured) |
| 2 | 40K steps, 400–10,240 rows log-uniform, LR 1e-4 | ~2 days | — |
| 3 | 10K steps, 400–60,000 rows log-uniform, LR 2e-5 | ~2.3 days | — |

**Classification only** (`max_classes=10`); the regressor was not trained.

## Quality (same-harness comparisons vs the released tabicl-classifier-v2)

| benchmark | ours | reference (released v2) |
|---|---|---|
| TabArena-Lite | Elo 1537.6 | Elo 1557.7 |
| TALENT (181 clf datasets), mean acc | 0.8400 (W/T/L 50/30/101) | 0.8426 |
| Large-dataset suite (15–48K rows), mean acc | 0.9133 | 0.9204 |

## Recipe deviations from the official scripts (documented, not endorsed)
* bf16 autocast w/ fp32 stability islands vs tf32
* plain WD 0.1 vs 0.01
* ssmax/embeddings/norms/heads on aux AdamW vs all-in-one Muon group
* warmup 2% vs 1%
* cosine floor 0 vs 1e-7
* min_features 2 vs 1
* micro-batch 8 vs 4
* stage-2/3 train fraction exactly 0.80 vs 0.79–0.81


## Checkpoints
| file | stage | steps | note |
|---|---|---|---|
| stage1/step-{50000..500000}.ckpt | 1 | 50K–500K | 6 snapshots |
| stage2/step-{10000..40000}.ckpt  | 2 | 10K–40K | 4 snapshots |
| stage3/step-{2500..10000}.ckpt   | 3 | 2.5K–10K | stage3/step-10000.ckpt = final model |

-----

Research release from Nolano AI (Apache-2.0). Training/evaluation code and replication checkpoints by Ayush Kaushal, accompanying a stage-1 training-efficiency study. These are research artifacts, not a supported product and no maintenance or support is implied.