File size: 2,337 Bytes
5859526
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
---
license: mit
tags:
  - cad
  - graphs
  - transformer
  - self-supervised
datasets:
  - NicoloLocatelli/EulerCAD-ABC-Preprocessed
---

# EulerCAD-Deep

Deep is a decoder-only transformer pretrained on 2D CAD drawings. Each drawing is a graph of points and
straight segments, and the model reads it as a random Euler walk: an ordered sequence of rows that crosses
every segment once. It learns to predict the next row. A drawing's embedding is the mean of one layer's
hidden states over the walk.

Code, training and analysis: [github.com/NicoloLocatelli/EulerCAD](https://github.com/NicoloLocatelli/EulerCAD).

## Model

- 14 layers of width 384, 6 attention heads, 36.9M parameters.
- Gated attention, QK normalization, value residual.
- Vocabulary of 512 node identities and 256 bins for coordinates and distances. Drawings with more than
  512 points are dropped.

## Training

- Data: the 246,108 training drawings of
  [EulerCAD-ABC-Preprocessed](https://huggingface.co/datasets/NicoloLocatelli/EulerCAD-ABC-Preprocessed),
  2D views of 3D objects from the [ABC dataset](https://deep-geometry.github.io/abc-dataset/).
- 208,200 updates of 160 drawings, with the Muon optimizer and a warmup-stable-decay schedule, on one
  RTX 4060 Ti (16 GB).
- These weights are the checkpoint at update 208,000, which has the best validation loss: 0.7144.

## Files

- `model.safetensors`: the weights.
- `config.json`: the tokenizer, the model configuration, and the update the weights come from.

## Use

```bash
git clone https://github.com/NicoloLocatelli/EulerCAD.git
cd EulerCAD
uv sync
uv run eulercad model download          # into runs/deep/checkpoints/
uv run eulercad analyze runs/deep/checkpoints/model.safetensors --split val
```

`analyze` measures every layer and scores retrieval of the other views of each 3D object. On the
validation split, retrieval is best at layer 7: mean average precision 0.250, recall@1 0.538.

In Python:

```python
from pathlib import Path
from eulercad.training.artifacts import load_model

model = load_model(Path("runs/deep/checkpoints/model.safetensors"))
```

## License

The weights and the code are under the MIT License. The training data derive from ABC, whose CAD models
belong to their creators under the [Onshape Terms of Use](https://www.onshape.com/legal/terms-of-use),
section 1.g.ii.