| --- |
| library_name: pytorch |
| license: mit |
| model_name: mtg-art-cnn |
| pipeline_tag: image-classification |
| tags: |
| - magic-the-gathering |
| - cnn |
| - from-scratch |
| - multi-task |
| - safetensors |
| - gradio |
| --- |
| |
| # MTG art CNN: colour identity + card type from the artwork alone |
|
|
| A **small convolutional network trained from random initialisation in plain PyTorch** (no pretrained weights, |
| no `torchvision` models) that looks at the *art crop* of a Magic: The Gathering card and predicts two labels |
| with two softmax heads: |
|
|
| * **Colour identity** (7): White, Blue, Black, Red, Green, Colorless, Multicolor |
| * **Card type** (7): Creature, Planeswalker, Land, Artifact, Enchantment, Instant, Sorcery |
|
|
| It is the final project for the Skillspire AI & Machine Learning course (Nathan King). The |
| live demo is the Space [BuildITSystems/mtg-art-demo](https://huggingface.co/spaces/BuildITSystems/mtg-art-demo). |
|
|
| ## Architecture |
|
|
| `TwoHeadCNN(width=32, blocks=[2, 2, 2, 2], dropout=0.2)`: a 5x5 stride-2 stem, four pre-activation |
| residual stages (widths 32/64/128/256, stride 2 each, 8 BasicBlocks), |
| global average pooling and two linear heads; **2,800,366 parameters**, 5.6 MB in bf16. Input: the art crop |
| resized to 160x224 (HxW), normalised with the training mean/std stored in `config.json`. |
| The complete definition is `model.py` in the demo Space |
| ([BuildITSystems/mtg-art-demo](https://huggingface.co/spaces/BuildITSystems/mtg-art-demo)), byte-identical to the training code. |
|
|
| ## Training |
|
|
| 36,325 training artworks (one per `illustration_id`), Scryfall art crops of English paper |
| printings; validation grouped by card, test = whole held-out sets (unseen cards *and* unseen sets). |
| Loss = 1.0 x L_colour + 1.0 x L_type with `ce` |
| (label smoothing 0.1); AdamW lr 0.002, |
| weight decay 0.05 (none on BN/bias), 5% linear warmup then cosine to 1%, batch 256, |
| bf16 autocast, grad-norm clip 1.0, early stopping on val mean macro-F1 (patience 6); |
| best epoch ? of 30 planned (val colour macro-F1 |
| nan, type macro-F1 nan). |
| Why these choices: colour classes are balanced but type is heavily skewed (half of all cards are creatures, |
| ~1% planeswalkers), so macro-F1 drives model selection and the loss is the main experiment dial. |
|
|
| ## Evaluation |
|
|
| Unseen test split (3,401 art crops from held-out sets), checkpoint `best`: |
|
|
| | head | accuracy | top-2 | macro-F1 | ECE | |
| |---|---:|---:|---:|---:| |
| | Colour identity | 0.3684 | 0.5698 | 0.3450 | 0.0419 | |
| | Card type | 0.5293 | 0.6745 | 0.2857 | 0.0538 | |
|
|
| Per-class F1 (type head): Creature 0.689 (n=1606), Planeswalker 0.217 (n=37), Land 0.597 (n=347), Artifact 0.211 (n=314), Enchantment 0.103 (n=418), Instant 0.116 (n=393), Sorcery 0.067 (n=286) |
|
|
| Per-class F1 (colour head): White 0.296 (n=520), Blue 0.430 (n=517), Black 0.334 (n=521), Red 0.492 (n=511), Green 0.486 (n=517), Colorless 0.194 (n=257), Multicolor 0.183 (n=558) |
|
|
| Single-image latency (100 images, batch 1, resize + normalise + forward + softmax on a decoded image, JPEG decode excluded, `timeit.default_timer`; instructor benchmark 0.074 s): |
|
|
| | device | mean | p50 | p95 | vs benchmark | |
| |---|---:|---:|---:|---:| |
| | GPU bf16 (NVIDIA GeForce RTX 3090 Ti) | 3.73 ms | 3.68 ms | 4.24 ms | 0.050x | |
| | CPU fp32 (16 threads) | 5.20 ms | 5.05 ms | 6.01 ms | 0.070x | |
|
|
| ## How to use |
|
|
| ```python |
| import json, torch |
| from PIL import Image |
| from safetensors.torch import load_file |
| from model import TwoHeadCNN # model.py from this repo |
| |
| cfg = json.load(open("config.json")) |
| model = TwoHeadCNN.from_config(cfg["model"]).eval() |
| model.load_state_dict(load_file("model.safetensors")) |
| H, W = cfg["image_size"] |
| img = Image.open("art_crop.jpg").convert("RGB").resize((W, H), Image.BILINEAR) # bilinear, as in training/eval |
| x = torch.tensor(list(img.getdata()), dtype=torch.float32).view(H, W, 3).permute(2, 0, 1) / 255 |
| x = (x - torch.tensor(cfg["normalization"]["mean"]).view(3, 1, 1)) / torch.tensor(cfg["normalization"]["std"]).view(3, 1, 1) |
| with torch.no_grad(): |
| color_logits, type_logits = model(x[None]) |
| print(cfg["labels"]["color"][color_logits.argmax().item()], cfg["labels"]["type"][type_logits.argmax().item()]) |
| ``` |
|
|
| ## Limitations |
|
|
| * Trained on official card art only; photos of physical cards, sleeves, glare and perspective are out of |
| distribution (the Space crops a photo with a fixed template, which is approximate). |
| * The type head reads *style* cues (a creature portrait vs a landscape), not the type line; lands and |
| sorceries with similar art are confused, planeswalkers are rare. |
| * Colour is the colour identity of the card the art belongs to, which the art does not always show. |
|
|
| ## Legal |
|
|
| This project is unofficial Fan Content permitted under the Wizards of the Coast Fan Content Policy. Not approved/endorsed by Wizards. Portions of the materials used are property of Wizards of the Coast. (c) Wizards of the Coast LLC. |
|
|
| Card metadata and art crops were obtained from [Scryfall](https://scryfall.com) under its API terms; Scryfall is not affiliated with this project. The training image set is not redistributed; the Space ships six test-split art crops as examples under the Fan Content Policy, and only the trained weights are published here. |
|
|
