| --- |
| license: mit |
| tags: |
| - svg |
| - vector-graphics |
| - generative |
| library_name: pytorch |
| --- |
| |
| # Grimoire — VSQ + ART checkpoints |
|
|
| Final checkpoints for **Grimoire**, a two-stage conditional SVG generative model. |
|
|
| - **VSQ** (stage 1): a vector-quantized SVG tokenizer/autoencoder (ResNet encoder → |
| FSQ codebook → differentiable vector decoder rendered with diffvg). |
| - **ART** (stage 2): an autoregressive transformer over VSQ tokens, conditioned on a |
| frozen BERT text embedding (class `VQ_SVG_Stage2`). |
|
|
| ## Layout |
|
|
| ``` |
| <dataset>/<stage>/last.ckpt # PyTorch-Lightning checkpoint (weights under "state_dict") |
| <dataset>/<stage>/config.yaml # config to rebuild the model before loading |
| ``` |
|
|
| | dataset | VSQ (stage 1) | ART (stage 2) | |
| |--------------|:-:|:-:| |
| | figr8 | ✅ | ✅ | |
| | mnist_bw | ✅ | ✅ | |
| | mnist_color | ✅ | ✅ | |
| | fonts | ✅ | — | |
| | emoji | ✅ | — | |
|
|
| > Matching datasets: [grimoire-figr8](https://huggingface.co/datasets/Potpov/grimoire-figr8), |
| > [grimoire-mnist](https://huggingface.co/datasets/Potpov/grimoire-mnist), |
| > [grimoire-emoji](https://huggingface.co/datasets/Potpov/grimoire-emoji). |
|
|
| ## Loading & inference (with the code checked out) |
|
|
| **Stage 1 — VSQ reconstruction** (figr8 / fonts, single-layer): |
| ```bash |
| python scripts/hf_inference_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir figr8/vsq \ |
| --dataset_csv_path <csv with a file_path column> --outpath out/ |
| ``` |
|
|
| **Stage 2 — ART text→SVG generation** (loads VSQ + trained ART from HF, samples an SVG): |
| ```bash |
| python scripts/hf_generate_demo.py --hf_repo Potpov/grimoire-checkpoints \ |
| --art_subdir figr8/art --vsq_subdir figr8/vsq --prompt "a star" --outfile out/gen.svg |
| ``` |
|
|
| **Emoji — layered/colored VSQ reconstruction** (the `VSQ_layers` + `hydra` path is different): |
| ```bash |
| python scripts/hf_inference_emoji_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir emoji/vsq \ |
| --emoji_dir <dir with preprocessed_v2/> --outfile out/emoji.svg |
| ``` |
|
|
| All three are verified working (torch 2.0.1 / cu118 / diffvg-from-source). |
|
|
| ### Notes |
| - Build the VSQ from its **own** `vsq/config.yaml`. `mnist_color/*` configs are the repo configs. |
| - `emoji/vsq` is `emoji_VSQ_hydra_small_noout` (`hydra` decoder, `single_code_representation=true`, |
| `num_segments=16`, `pred_color=true`) — the properly-trained emoji VSQ; it reconstructs emojis |
| faithfully. (An earlier push used the older `VSQ_EMOJI_COLOR`, which under-reconstructed.) |
| - **Loading the ART (stage-2) checkpoints:** strip only the *leading* `model.` prefix (a blanket |
| `replace("model.","")` also deletes the inner `.model.` in `transformer.model.layers.*` → the |
| transformer loads as random weights) and remap `.ff.3.`→`.ff.2.` (x-transformers version shift). |
| `scripts/hf_generate_demo.py:_remap_key` does this. VSQ checkpoints load with a plain leading-prefix strip. |
|
|
| ## Requirements |
| torch 2.0.1 (cu118) · diffvg (from source) · see the code repo's `install.sh` / `requirements.txt`. |
|
|
