EulerCAD-Deep-SIGReg
Deep fine-tuned with SIGReg, from LeJEPA, a loss that pushes the embeddings of a batch toward an isotropic Gaussian. Same network as Deep: 14 layers of width 384, 36.9M parameters.
Code, training and analysis: github.com/NicoloLocatelli/EulerCAD.
Fine-tune
- Starts from Deep's weights and runs 20,000 more updates of 160 training drawings.
- Loss:
(1 - w) * next_row + w * SIGReg, withw = 0.018, so SIGReg is about 10% of the loss at the start. - SIGReg reads one vector per drawing, the mean of the hidden states over its walk, at the output of each of the 14 layers and of the final norm. Each of these 15 readouts is centered over the batch and divided by one standard deviation before SIGReg.
- Peak learning rate 3e-4 after 250 warmup updates, flat for the first half and linear to zero in the second. Muon optimizer, as for Deep.
- The best checkpoint by next-row validation loss is the last one, at update 20,000: 0.7191, against 0.7144 for Deep and 0.7154 for the same fine-tune without SIGReg.
Effect
SIGReg lowered the mean SIGReg on the training batches from 1.60 to 0.94 and raised the effective rank of every layer. On the validation split, retrieval of the other views of each 3D object is best at layer 7 for both models:
| Model | mAP | recall@1 | recall@10 |
|---|---|---|---|
| Deep | 0.250 | 0.538 | 0.760 |
| Deep-SIGReg | 0.268 | 0.505 | 0.734 |
So SIGReg ranks the other views higher overall, but gets the first answer right less often.
Files
model.safetensors: the weights.config.json: the tokenizer, the model configuration, and the update of the fine-tune the weights come from.
Use
git clone https://github.com/NicoloLocatelli/EulerCAD.git
cd EulerCAD
uv sync
uv run eulercad model download deep-sigreg # into runs/deep-sigreg/checkpoints/
uv run eulercad analyze runs/deep-sigreg/checkpoints/model.safetensors --split val
To repeat the fine-tune, download Deep and the data, then:
uv run eulercad model download
uv run eulercad data download
uv run eulercad train --preset deep-sigreg
License
The weights and the code are under the MIT License. The training data derive from ABC, whose CAD models belong to their creators under the Onshape Terms of Use, section 1.g.ii.
- Downloads last month
- 30
Model tree for NicoloLocatelli/EulerCAD-Deep-SIGReg
Base model
NicoloLocatelli/EulerCAD-Deep