EulerCAD-Deep-SIGReg

Deep fine-tuned with SIGReg, from LeJEPA, a loss that pushes the embeddings of a batch toward an isotropic Gaussian. Same network as Deep: 14 layers of width 384, 36.9M parameters.

Code, training and analysis: github.com/NicoloLocatelli/EulerCAD.

Fine-tune

  • Starts from Deep's weights and runs 20,000 more updates of 160 training drawings.
  • Loss: (1 - w) * next_row + w * SIGReg, with w = 0.018, so SIGReg is about 10% of the loss at the start.
  • SIGReg reads one vector per drawing, the mean of the hidden states over its walk, at the output of each of the 14 layers and of the final norm. Each of these 15 readouts is centered over the batch and divided by one standard deviation before SIGReg.
  • Peak learning rate 3e-4 after 250 warmup updates, flat for the first half and linear to zero in the second. Muon optimizer, as for Deep.
  • The best checkpoint by next-row validation loss is the last one, at update 20,000: 0.7191, against 0.7144 for Deep and 0.7154 for the same fine-tune without SIGReg.

Effect

SIGReg lowered the mean SIGReg on the training batches from 1.60 to 0.94 and raised the effective rank of every layer. On the validation split, retrieval of the other views of each 3D object is best at layer 7 for both models:

Model mAP recall@1 recall@10
Deep 0.250 0.538 0.760
Deep-SIGReg 0.268 0.505 0.734

So SIGReg ranks the other views higher overall, but gets the first answer right less often.

Files

  • model.safetensors: the weights.
  • config.json: the tokenizer, the model configuration, and the update of the fine-tune the weights come from.

Use

git clone https://github.com/NicoloLocatelli/EulerCAD.git
cd EulerCAD
uv sync
uv run eulercad model download deep-sigreg        # into runs/deep-sigreg/checkpoints/
uv run eulercad analyze runs/deep-sigreg/checkpoints/model.safetensors --split val

To repeat the fine-tune, download Deep and the data, then:

uv run eulercad model download
uv run eulercad data download
uv run eulercad train --preset deep-sigreg

License

The weights and the code are under the MIT License. The training data derive from ABC, whose CAD models belong to their creators under the Onshape Terms of Use, section 1.g.ii.

Downloads last month
30
Safetensors
Model size
36.9M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NicoloLocatelli/EulerCAD-Deep-SIGReg

Finetuned
(1)
this model

Dataset used to train NicoloLocatelli/EulerCAD-Deep-SIGReg

Paper for NicoloLocatelli/EulerCAD-Deep-SIGReg