File size: 3,013 Bytes
de2ce3f c912c26 de2ce3f c912c26 de2ce3f c912c26 de2ce3f c912c26 de2ce3f c912c26 de2ce3f c912c26 de2ce3f c912c26 18e7382 c912c26 18e7382 c912c26 de2ce3f c912c26 dc58d19 de2ce3f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
license: mit
tags:
- svg
- vector-graphics
- generative
library_name: pytorch
---
# Grimoire — VSQ + ART checkpoints
Final checkpoints for **Grimoire**, a two-stage conditional SVG generative model.
- **VSQ** (stage 1): a vector-quantized SVG tokenizer/autoencoder (ResNet encoder →
FSQ codebook → differentiable vector decoder rendered with diffvg).
- **ART** (stage 2): an autoregressive transformer over VSQ tokens, conditioned on a
frozen BERT text embedding (class `VQ_SVG_Stage2`).
## Layout
```
<dataset>/<stage>/last.ckpt # PyTorch-Lightning checkpoint (weights under "state_dict")
<dataset>/<stage>/config.yaml # config to rebuild the model before loading
```
| dataset | VSQ (stage 1) | ART (stage 2) |
|--------------|:-:|:-:|
| figr8 | ✅ | ✅ |
| mnist_bw | ✅ | ✅ |
| mnist_color | ✅ | ✅ |
| fonts | ✅ | — |
| emoji | ✅ | — |
> Matching datasets: [grimoire-figr8](https://huggingface.co/datasets/Potpov/grimoire-figr8),
> [grimoire-mnist](https://huggingface.co/datasets/Potpov/grimoire-mnist),
> [grimoire-emoji](https://huggingface.co/datasets/Potpov/grimoire-emoji).
## Loading & inference (with the code checked out)
**Stage 1 — VSQ reconstruction** (figr8 / fonts, single-layer):
```bash
python scripts/hf_inference_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir figr8/vsq \
--dataset_csv_path <csv with a file_path column> --outpath out/
```
**Stage 2 — ART text→SVG generation** (loads VSQ + trained ART from HF, samples an SVG):
```bash
python scripts/hf_generate_demo.py --hf_repo Potpov/grimoire-checkpoints \
--art_subdir figr8/art --vsq_subdir figr8/vsq --prompt "a star" --outfile out/gen.svg
```
**Emoji — layered/colored VSQ reconstruction** (the `VSQ_layers` + `hydra` path is different):
```bash
python scripts/hf_inference_emoji_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir emoji/vsq \
--emoji_dir <dir with preprocessed_v2/> --outfile out/emoji.svg
```
All three are verified working (torch 2.0.1 / cu118 / diffvg-from-source).
### Notes
- Build the VSQ from its **own** `vsq/config.yaml`. `mnist_color/*` configs are the repo configs.
- `emoji/vsq` is `emoji_VSQ_hydra_small_noout` (`hydra` decoder, `single_code_representation=true`,
`num_segments=16`, `pred_color=true`) — the properly-trained emoji VSQ; it reconstructs emojis
faithfully. (An earlier push used the older `VSQ_EMOJI_COLOR`, which under-reconstructed.)
- **Loading the ART (stage-2) checkpoints:** strip only the *leading* `model.` prefix (a blanket
`replace("model.","")` also deletes the inner `.model.` in `transformer.model.layers.*` → the
transformer loads as random weights) and remap `.ff.3.`→`.ff.2.` (x-transformers version shift).
`scripts/hf_generate_demo.py:_remap_key` does this. VSQ checkpoints load with a plain leading-prefix strip.
## Requirements
torch 2.0.1 (cu118) · diffvg (from source) · see the code repo's `install.sh` / `requirements.txt`.
|