File size: 4,617 Bytes
2e2eb08 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 | ---
license: apache-2.0
library_name: transformers
pipeline_tag: image-classification
base_model: facebook/dinov3-vitl16-pretrain-lvd1689m
tags:
- image-classification
- food-recognition
- dinov3
- vision-transformer
- pytorch
- tsotsa-img
datasets:
- TSOTSA-Img
metrics:
- accuracy
- f1
---
# DINOv3-food
DINOv3-food is a food image recognition model fine-tuned from
[`facebook/dinov3-vitl16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m)
on TSOTSA-Img, a merged food image dataset built from AFD, FruitVeg-81,
Food-101, and UECFood256.
The model predicts 389 food categories and uses a DINOv3 ViT-L/16 backbone
with a lightweight linear classification head.
## Dataset
TSOTSA-Img is the merged dataset used for food recognition in this work. It is
split into training and test subsets:
- Training split: used to fine-tune the model.
- Test split: used for final evaluation.
The merged dataset combines food images and labels from:
- AFD
- FruitVeg-81
- Food-101
- UECFood256
## Training
The selected checkpoint was fine-tuned for 8 epochs.
| Setting | Value |
|---|---:|
| Base model | `facebook/dinov3-vitl16-pretrain-lvd1689m` |
| Backbone | DINOv3 ViT-L/16 |
| Number of labels | 389 |
| Epochs | 8 |
| Batch size | 16 |
| Learning rate | `2e-5` |
| Weight decay | `0.01` |
| Warmup ratio | `0.05` |
| Validation selection | best validation behavior, with emphasis on validation loss |
Validation metrics for the selected run:
| Metric | Value |
|---|---:|
| Validation loss | 0.1100 |
| Accuracy | 0.9731 |
| Macro-F1 | 0.9727 |
| Top-5 accuracy | 0.9968 |
## Evaluation
Final evaluation was performed on the individual source datasets and on the
merged TSOTSA-Img test split.
| Dataset | Accuracy |
|---|---:|
| FruitVeg-81 | 0.9976 |
| AFD | 0.9997 |
| Food-101 | 0.9551 |
| UECFood256 | 0.8215 |
| TSOTSA-Img test | 0.9062 |
For the TSOTSA-Img test split:
| Metric | Value |
|---|---:|
| Accuracy | 0.9062 |
| Macro-F1 | 0.9072 |
## Model format
This repository stores a custom backbone-plus-classifier model:
- `backbone/`: DINOv3 backbone saved with `transformers`.
- `classifier.pt`: linear classification head.
- `classifier_config.json`: label mappings and classifier metadata.
- `preprocessor_config.json`: image preprocessing configuration.
Because this model uses a custom wrapper around the DINOv3 backbone, loading it
with `AutoModelForImageClassification.from_pretrained(...)` is not sufficient.
Use the project loader or reconstruct the wrapper before inference.
## Usage
Example with the project inference class:
```python
from inference.food_classifier import FoodClassifier
model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
classifier = FoodClassifier(model_dir)
prediction = classifier.predict("path/to/food_image.jpg")
print(prediction)
```
Manual loading:
```python
import json
import torch
from transformers import AutoImageProcessor, AutoModel
from finetuning.train_classifier import BackboneImageClassifier
model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
with open(f"{model_dir}/classifier_config.json", "r", encoding="utf-8") as f:
classifier_config = json.load(f)
id2label = {
int(label_id): label
for label_id, label in classifier_config["id2label"].items()
}
label2id = {
label: int(label_id)
for label, label_id in classifier_config["label2id"].items()
}
backbone = AutoModel.from_pretrained(f"{model_dir}/backbone")
model = BackboneImageClassifier(
backbone=backbone,
num_labels=int(classifier_config["num_labels"]),
id2label=id2label,
label2id=label2id,
)
classifier_state = torch.load(f"{model_dir}/classifier.pt", map_location="cpu")
model.classifier.load_state_dict(classifier_state)
model.eval()
processor = AutoImageProcessor.from_pretrained(model_dir)
```
## Intended use
This model is intended for food image recognition over the TSOTSA-Img label
space. It can be used for research experiments, dataset benchmarking, and food
recognition pipelines where the target labels overlap with the 389 supported
categories.
## Limitations
- The model is restricted to the 389 labels in `classifier_config.json`.
- Performance may degrade on food categories outside the TSOTSA-Img label
space.
- Predictions may be sensitive to ambiguous images, mixed dishes, heavy
occlusion, or visually similar food categories.
- The model card reports accuracy on the available benchmark splits and should
not be interpreted as performance on all possible food domains.
## Citation
|