---
license: mit
tags:
  - svg
  - vector-graphics
  - generative
library_name: pytorch
---

# Grimoire — VSQ + ART checkpoints

Final checkpoints for **Grimoire**, a two-stage conditional SVG generative model.

- **VSQ** (stage 1): a vector-quantized SVG tokenizer/autoencoder (ResNet encoder →
  FSQ codebook → differentiable vector decoder rendered with diffvg).
- **ART** (stage 2): an autoregressive transformer over VSQ tokens, conditioned on a
  frozen BERT text embedding (class `VQ_SVG_Stage2`).

## Layout

```
<dataset>/<stage>/last.ckpt      # PyTorch-Lightning checkpoint (weights under "state_dict")
<dataset>/<stage>/config.yaml    # config to rebuild the model before loading
```

| dataset      | VSQ (stage 1) | ART (stage 2) |
|--------------|:-:|:-:|
| figr8        | ✅ | ✅ |
| mnist_bw     | ✅ | ✅ |
| mnist_color  | ✅ | ✅ |
| fonts        | ✅ | — |
| emoji        | ✅ | — |

> Matching datasets: [grimoire-figr8](https://huggingface.co/datasets/Potpov/grimoire-figr8),
> [grimoire-mnist](https://huggingface.co/datasets/Potpov/grimoire-mnist),
> [grimoire-emoji](https://huggingface.co/datasets/Potpov/grimoire-emoji).

## Loading & inference (with the code checked out)

**Stage 1 — VSQ reconstruction** (figr8 / fonts, single-layer):
```bash
python scripts/hf_inference_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir figr8/vsq \
    --dataset_csv_path <csv with a file_path column> --outpath out/
```

**Stage 2 — ART text→SVG generation** (loads VSQ + trained ART from HF, samples an SVG):
```bash
python scripts/hf_generate_demo.py --hf_repo Potpov/grimoire-checkpoints \
    --art_subdir figr8/art --vsq_subdir figr8/vsq --prompt "a star" --outfile out/gen.svg
```

**Emoji — layered/colored VSQ reconstruction** (the `VSQ_layers` + `hydra` path is different):
```bash
python scripts/hf_inference_emoji_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir emoji/vsq \
    --emoji_dir <dir with preprocessed_v2/> --outfile out/emoji.svg
```

All three are verified working (torch 2.0.1 / cu118 / diffvg-from-source).

### Notes
- Build the VSQ from its **own** `vsq/config.yaml`. `mnist_color/*` configs are the repo configs.
- `emoji/vsq` is `emoji_VSQ_hydra_small_noout` (`hydra` decoder, `single_code_representation=true`,
  `num_segments=16`, `pred_color=true`) — the properly-trained emoji VSQ; it reconstructs emojis
  faithfully. (An earlier push used the older `VSQ_EMOJI_COLOR`, which under-reconstructed.)
- **Loading the ART (stage-2) checkpoints:** strip only the *leading* `model.` prefix (a blanket
  `replace("model.","")` also deletes the inner `.model.` in `transformer.model.layers.*` → the
  transformer loads as random weights) and remap `.ff.3.`→`.ff.2.` (x-transformers version shift).
  `scripts/hf_generate_demo.py:_remap_key` does this. VSQ checkpoints load with a plain leading-prefix strip.

## Requirements
torch 2.0.1 (cu118) · diffvg (from source) · see the code repo's `install.sh` / `requirements.txt`.