| --- |
| language: en |
| library_name: onnx |
| tags: |
| - image-classification |
| - multi-label-classification |
| - danbooru |
| - anime |
| - onnx |
| pipeline_tag: image-classification |
| license: other |
| license_name: dinov3-license |
| license_link: https://ai.meta.com/resources/models-and-libraries/dinov3-license/ |
| datasets: |
| - trojblue/danbooru2025-metadata |
| base_model: |
| - facebook/dinov3-vitb16-pretrain-lvd1689m |
| - facebook/dinov3-vitl16-pretrain-lvd1689m |
| --- |
| |
| # DanbooruTagQuery |
|
|
| Lightweight multi-label anime image tagger using **DINOv3 ViT backbone** and a **cross-attention tag query head**, trained on Danbooru images. |
|
|
| - 11,424 tags (general + character + copyright) |
| - 448×448 input, ONNX-exported |
|
|
| [GitHub repo](https://github.com/realphongha/DanbooruTagQuery) |
| You can try our models here: |
| [realphongha/DanbooruTagQuery](https://huggingface.co/spaces/realphongha/DanbooruTagQuery) |
|
|
| ## Model description |
|
|
| Image → ViT backbone (DINOv3) → patch tokens → cross-attention → per-tag logits. |
|
|
| Each tag is a **single learnable embedding vector** (a "query"). An `nn.MultiheadAttention` layer computes attention between all tag queries and all ViT patch tokens; each query's attended feature is projected through a scalar linear classifier. No MLP, no positional encoding — the query itself encodes "what to look for". This design is modular: the head can be swapped without touching the dataset, training loop, or metrics. |
|
|
| ```text |
| Input image (448×448) |
| │ |
| ▼ |
| ┌─────────────────────┐ |
| │ DINOv3 │ pretrained ViT backbone |
| └─────────┬───────────┘ |
| │ tokens: (B, N_patches+5, D) |
| ▼ |
| ┌─────────────────────┐ |
| │ Tag Query Head │ |
| │ │ |
| │ tag_queries: │ learned (num_tags, D) |
| │ (num_tags, D) ─────┼─→ cross-attention ──→ tag features (B, num_tags, D) |
| │ │ queries attend to ViT patch tokens |
| │ classifier: │ |
| │ Linear(D→1) ───────┘ → logits (B, num_tags) |
| └─────────────────────┘ |
| │ |
| ▼ |
| sigmoid(logits) → per-tag probabilities |
| ``` |
|
|
| ## Evaluation |
|
|
| Evaluated against popular community taggers on the **intersection evaluation subset** (3,383 tags present in every model's vocabulary), drawn from a held-out danbooru2025 test set of images with post ID > 7220105 (avoids contamination with WD-SwinV2 training data). Each model ran at its native resolution; thresholds searched over `[0.10, 0.15, …, 0.95]`. |
|
|
| | Model | Params | Input | Latency | mAP | Macro F1 | Micro F1 | Best threshold | |
| |---|---|---|---|---|---|---|---| |
| | **Ours (L/16) - trained from scratch** | 319.0M | 448×448 | 36.6ms | **0.5352** | **0.4775** | **0.6884** | 0.20 | |
| | WD-eva02-large-tagger-v3 | 315.2M | 448×448 | 50.3ms | 0.4822 | 0.4344 | 0.6684 | 0.30 | |
| | **Ours (B/16) - trained from scratch** | 96.8M | 448×448 | 24.9ms | 0.4693 | 0.4195 | 0.6684 | 0.20 | |
| | WD-SwinV2-tagger-v3 | 98.0M | 448×448 | 35.8ms | 0.4603 | 0.4140 | 0.6474 | 0.15 | |
| | ML-Danbooru | 68.9M | 448×448 | 34.0ms | 0.4023 | 0.3490 | 0.5952 | 0.60 | |
| | JoyTag | 91.5M | 448×448 | 20.2ms | 0.3783 | 0.3429 | 0.6179 | 0.35 | |
| | DeepDanbooru (CNN) | 161.0M | 512×512 | 33.6ms | 0.2100 | 0.1920 | 0.4692 | 0.15 | |
|
|
| Key results: |
|
|
| - **Ours (L/16)** leads all baselines by **+0.053 mAP** over WD-eva02 (the strongest comparable model) with 33% faster inference (36.6ms vs 50.3ms). |
| - **Ours (B/16)** outperforms all models in the <100M class (WD-SwinV2 +0.009 mAP, JoyTag +0.091 mAP) at 24.9ms latency. |
|
|
| *Latency measured on a single NVIDIA RTX 5090 (PyTorch eager for local runs; ONNX Runtime for ONNX-exported models).* |
|
|
| ## Intended uses |
|
|
| - Tagging anime / illustration images (general, character, and copyright tags) |
| - Generating caption / metadata text for image collections |
| - Danbooru-style workflows, e.g. injecting a tag preset into image-generation UIs |
| - Batch CLI tagging of local image folders |
|
|
| **Note: this model tags adult (NSFW) content.** Danbooru contains explicit imagery and its tag vocabulary reflects that. Do not use in contexts where such content is unacceptable. |
|
|
| ## Training data |
| - Metadata source for Danbooru posts/images: [`trojblue/danbooru2025-metadata`](https://huggingface.co/datasets/trojblue/danbooru2025-metadata) |
| - Filters: deleted/banned/flagged/pending images removed, score > 10, file size 50 KB–20 MB, dimensions ≥ 224×224 |
| - Tags: `tag_string_general` + `tag_string_character` + `tag_string_copyright` unified; tags with frequency ≥ 100; images with ≥ 2 tags |
| - Vocabulary: **11,424 tags**; ~1M training images; last 100K images held out as validation |
| - Noise tags removed via `data/ignored_tags.txt` |
|
|
| ## Future plan |
| - [ ] Care more about the data (de-duplication, remove noise, etc.). Current data cleaning pipeline is too basic? |
| - [ ] Try KD + Transfer Learning from L16 => B16/S16 models. |
| - [ ] Another self-attention block for the head => more semantic for the tags. |
|
|
| ## Acknowledgements |
|
|
| - [DINOv3](https://github.com/facebookresearch/dinov3) (Meta AI) — pretrained ViT backbones (licensed under the DINOv3 License) |
| - [`trojblue/danbooru2025-metadata`](https://huggingface.co/datasets/trojblue/danbooru2025-metadata) — training data |
| - Baseline reimplementations live in [`src/compared_models/`](https://github.com/realphongha/DanbooruTagQuery/tree/master/src/compared_models): DeepDanbooru, WD-tagger, ML-Danbooru, JoyTag |
| - [timm](https://github.com/huggingface/pytorch-image-models) for backbone loading and [ONNX Runtime](https://onnxruntime.ai/) for deployment |
