README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-classification
base_model: facebook/dinov3-vitl16-pretrain-lvd1689m
tags:
  - image-classification
  - food-recognition
  - dinov3
  - vision-transformer
  - pytorch
  - tsotsa-img
datasets:
  - TSOTSA-Img
metrics:
  - accuracy
  - f1
---

# DINOv3-food

DINOv3-food is a food image recognition model fine-tuned from
[`facebook/dinov3-vitl16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m)
on TSOTSA-Img, a merged food image dataset built from AFD, FruitVeg-81,
Food-101, and UECFood256.

The model predicts 389 food categories and uses a DINOv3 ViT-L/16 backbone
with a lightweight linear classification head.

## Dataset

TSOTSA-Img is the merged dataset used for food recognition in this work. It is
split into training and test subsets:

- Training split: used to fine-tune the model.
- Test split: used for final evaluation.

The merged dataset combines food images and labels from:

- AFD
- FruitVeg-81
- Food-101
- UECFood256

## Training

The selected checkpoint was fine-tuned for 8 epochs.

| Setting | Value |
|---|---:|
| Base model | `facebook/dinov3-vitl16-pretrain-lvd1689m` |
| Backbone | DINOv3 ViT-L/16 |
| Number of labels | 389 |
| Epochs | 8 |
| Batch size | 16 |
| Learning rate | `2e-5` |
| Weight decay | `0.01` |
| Warmup ratio | `0.05` |
| Validation selection | best validation behavior, with emphasis on validation loss |

Validation metrics for the selected run:

| Metric | Value |
|---|---:|
| Validation loss | 0.1100 |
| Accuracy | 0.9731 |
| Macro-F1 | 0.9727 |
| Top-5 accuracy | 0.9968 |

## Evaluation

Final evaluation was performed on the individual source datasets and on the
merged TSOTSA-Img test split.

| Dataset | Accuracy |
|---|---:|
| FruitVeg-81 | 0.9976 |
| AFD | 0.9997 |
| Food-101 | 0.9551 |
| UECFood256 | 0.8215 |
| TSOTSA-Img test | 0.9062 |

For the TSOTSA-Img test split:

| Metric | Value |
|---|---:|
| Accuracy | 0.9062 |
| Macro-F1 | 0.9072 |

## Model format

This repository stores a custom backbone-plus-classifier model:

- `backbone/`: DINOv3 backbone saved with `transformers`.
- `classifier.pt`: linear classification head.
- `classifier_config.json`: label mappings and classifier metadata.
- `preprocessor_config.json`: image preprocessing configuration.

Because this model uses a custom wrapper around the DINOv3 backbone, loading it
with `AutoModelForImageClassification.from_pretrained(...)` is not sufficient.
Use the project loader or reconstruct the wrapper before inference.

## Usage

Example with the project inference class:

```python
from inference.food_classifier import FoodClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
classifier = FoodClassifier(model_dir)

prediction = classifier.predict("path/to/food_image.jpg")
print(prediction)
```

Manual loading:

```python
import json
import torch
from transformers import AutoImageProcessor, AutoModel
from finetuning.train_classifier import BackboneImageClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"

with open(f"{model_dir}/classifier_config.json", "r", encoding="utf-8") as f:
    classifier_config = json.load(f)

id2label = {
    int(label_id): label
    for label_id, label in classifier_config["id2label"].items()
}
label2id = {
    label: int(label_id)
    for label, label_id in classifier_config["label2id"].items()
}

backbone = AutoModel.from_pretrained(f"{model_dir}/backbone")
model = BackboneImageClassifier(
    backbone=backbone,
    num_labels=int(classifier_config["num_labels"]),
    id2label=id2label,
    label2id=label2id,
)

classifier_state = torch.load(f"{model_dir}/classifier.pt", map_location="cpu")
model.classifier.load_state_dict(classifier_state)
model.eval()

processor = AutoImageProcessor.from_pretrained(model_dir)
```

## Intended use

This model is intended for food image recognition over the TSOTSA-Img label
space. It can be used for research experiments, dataset benchmarking, and food
recognition pipelines where the target labels overlap with the 389 supported
categories.

## Limitations

- The model is restricted to the 389 labels in `classifier_config.json`.
- Performance may degrade on food categories outside the TSOTSA-Img label
  space.
- Predictions may be sensitive to ambiguous images, mixed dishes, heavy
  occlusion, or visually similar food categories.
- The model card reports accuracy on the available benchmark splits and should
  not be interpreted as performance on all possible food domains.

## Citation