Upload dinov3_custom food classifier from run 3: eps_8

README.md

@@ -0,0 +1,171 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: image-classification
+base_model: facebook/dinov3-vitl16-pretrain-lvd1689m
+tags:
+  - image-classification
+  - food-recognition
+  - dinov3
+  - vision-transformer
+  - pytorch
+  - tsotsa-img
+datasets:
+  - TSOTSA-Img
+metrics:
+  - accuracy
+  - f1
+---
+
+# DINOv3-food
+
+DINOv3-food is a food image recognition model fine-tuned from
+[`facebook/dinov3-vitl16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m)
+on TSOTSA-Img, a merged food image dataset built from AFD, FruitVeg-81,
+Food-101, and UECFood256.
+
+The model predicts 389 food categories and uses a DINOv3 ViT-L/16 backbone
+with a lightweight linear classification head.
+
+## Dataset
+
+TSOTSA-Img is the merged dataset used for food recognition in this work. It is
+split into training and test subsets:
+
+- Training split: used to fine-tune the model.
+- Test split: used for final evaluation.
+
+The merged dataset combines food images and labels from:
+
+- AFD
+- FruitVeg-81
+- Food-101
+- UECFood256
+
+## Training
+
+The selected checkpoint was fine-tuned for 8 epochs.
+
+| Setting | Value |
+|---|---:|
+| Base model | `facebook/dinov3-vitl16-pretrain-lvd1689m` |
+| Backbone | DINOv3 ViT-L/16 |
+| Number of labels | 389 |
+| Epochs | 8 |
+| Batch size | 16 |
+| Learning rate | `2e-5` |
+| Weight decay | `0.01` |
+| Warmup ratio | `0.05` |
+| Validation selection | best validation behavior, with emphasis on validation loss |
+
+Validation metrics for the selected run:
+
+| Metric | Value |
+|---|---:|
+| Validation loss | 0.1100 |
+| Accuracy | 0.9731 |
+| Macro-F1 | 0.9727 |
+| Top-5 accuracy | 0.9968 |
+
+## Evaluation
+
+Final evaluation was performed on the individual source datasets and on the
+merged TSOTSA-Img test split.
+
+| Dataset | Accuracy |
+|---|---:|
+| FruitVeg-81 | 0.9976 |
+| AFD | 0.9997 |
+| Food-101 | 0.9551 |
+| UECFood256 | 0.8215 |
+| TSOTSA-Img test | 0.9062 |
+
+For the TSOTSA-Img test split:
+
+| Metric | Value |
+|---|---:|
+| Accuracy | 0.9062 |
+| Macro-F1 | 0.9072 |
+
+## Model format
+
+This repository stores a custom backbone-plus-classifier model:
+
+- `backbone/`: DINOv3 backbone saved with `transformers`.
+- `classifier.pt`: linear classification head.
+- `classifier_config.json`: label mappings and classifier metadata.
+- `preprocessor_config.json`: image preprocessing configuration.
+
+Because this model uses a custom wrapper around the DINOv3 backbone, loading it
+with `AutoModelForImageClassification.from_pretrained(...)` is not sufficient.
+Use the project loader or reconstruct the wrapper before inference.
+
+## Usage
+
+Example with the project inference class:
+
+```python
+from inference.food_classifier import FoodClassifier
+
+model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
+classifier = FoodClassifier(model_dir)
+
+prediction = classifier.predict("path/to/food_image.jpg")
+print(prediction)
+```
+
+Manual loading:
+
+```python
+import json
+import torch
+from transformers import AutoImageProcessor, AutoModel
+from finetuning.train_classifier import BackboneImageClassifier
+
+model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
+
+with open(f"{model_dir}/classifier_config.json", "r", encoding="utf-8") as f:
+    classifier_config = json.load(f)
+
+id2label = {
+    int(label_id): label
+    for label_id, label in classifier_config["id2label"].items()
+}
+label2id = {
+    label: int(label_id)
+    for label, label_id in classifier_config["label2id"].items()
+}
+
+backbone = AutoModel.from_pretrained(f"{model_dir}/backbone")
+model = BackboneImageClassifier(
+    backbone=backbone,
+    num_labels=int(classifier_config["num_labels"]),
+    id2label=id2label,
+    label2id=label2id,
+)
+
+classifier_state = torch.load(f"{model_dir}/classifier.pt", map_location="cpu")
+model.classifier.load_state_dict(classifier_state)
+model.eval()
+
+processor = AutoImageProcessor.from_pretrained(model_dir)
+```
+
+## Intended use
+
+This model is intended for food image recognition over the TSOTSA-Img label
+space. It can be used for research experiments, dataset benchmarking, and food
+recognition pipelines where the target labels overlap with the 389 supported
+categories.
+
+## Limitations
+
+- The model is restricted to the 389 labels in `classifier_config.json`.
+- Performance may degrade on food categories outside the TSOTSA-Img label
+  space.
+- Predictions may be sensitive to ambiguous images, mixed dishes, heavy
+  occlusion, or visually similar food categories.
+- The model card reports accuracy on the available benchmark splits and should
+  not be interpreted as performance on all possible food domains.
+
+## Citation

all_results.json

@@ -1,21 +1,21 @@
 {
     "epoch": 8.0,
-    "eval_accuracy": 0.9731962296486718,
-    "eval_f1_macro": 0.97272463600211,
-    "eval_loss": 0.11423125863075256,
-    "eval_runtime": 215.0703,
-    "eval_samples_per_second": 135.653,
-    "eval_steps_per_second": 11.308,
-    "eval_top5_accuracy": 0.9967780634104542,
-    "run_wall_time_hours": 12.102496446039941,
-    "run_wall_time_minutes": 726.1497867623965,
-    "run_wall_time_seconds": 43568.98720574379,
+    "eval_accuracy": 0.9730591259640102,
+    "eval_f1_macro": 0.9727023139698624,
+    "eval_loss": 0.11002838611602783,
+    "eval_runtime": 216.4171,
+    "eval_samples_per_second": 134.809,
+    "eval_steps_per_second": 8.428,
+    "eval_top5_accuracy": 0.996846615252785,
+    "run_wall_time_hours": 10.413995842536291,
+    "run_wall_time_minutes": 624.8397505521774,
+    "run_wall_time_seconds": 37490.385033130646,
     "total_flos": 0.0,
-    "train_loss": 0.446697575658745,
-    "train_runtime": 43197.8762,
-    "train_samples_per_second": 48.627,
-    "train_steps_per_second": 4.052,
-    "train_wall_time_hours": 11.99947108036942,
-    "train_wall_time_minutes": 719.9682648221652,
-    "train_wall_time_seconds": 43198.09588932991
+    "train_loss": 0.37596295774437,
+    "train_runtime": 37253.6996,
+    "train_samples_per_second": 56.386,
+    "train_steps_per_second": 3.524,
+    "train_wall_time_hours": 10.348306588000721,
+    "train_wall_time_minutes": 620.8983952800432,
+    "train_wall_time_seconds": 37253.9037168026
 }

backbone/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:860eb2b858e1dcc139302a2c875fce78df92f77d933e8500b98444954fb5fe6c
+oid sha256:4623464dfa91314a072783b772355546571b4ce0d54bc17287e7c464c84e26e7
 size 1212559808

classifier.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:be0cee77682c3acbca96fc00edb13843acf273371a5600c06659bcab99e535d7
+oid sha256:e5941cc94cb6e61f79a00830562beb2356b174bda58772243d6b8fef29407777
 size 1596861

eval_results.json

@@ -1,10 +1,10 @@
 {
     "epoch": 8.0,
-    "eval_accuracy": 0.9731962296486718,
-    "eval_f1_macro": 0.97272463600211,
-    "eval_loss": 0.11423125863075256,
-    "eval_runtime": 215.0703,
-    "eval_samples_per_second": 135.653,
-    "eval_steps_per_second": 11.308,
-    "eval_top5_accuracy": 0.9967780634104542
+    "eval_accuracy": 0.9730591259640102,
+    "eval_f1_macro": 0.9727023139698624,
+    "eval_loss": 0.11002838611602783,
+    "eval_runtime": 216.4171,
+    "eval_samples_per_second": 134.809,
+    "eval_steps_per_second": 8.428,
+    "eval_top5_accuracy": 0.996846615252785
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7228be9fd21ce92856b879364aa3b773ae88f325a050411162237b2c748b6e37
+oid sha256:a8b6e793374a9998ffd81330b87bc3e5ea7d675652abed91c317a66d6a8a71e1
 size 1214158628

run_summary.json

@@ -6,37 +6,37 @@
   "output_dir": "./model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8",
   "logging_dir": "./finetuning/logs/runs/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8",
   "cache_dir": "./cached_dataset/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m",
-  "started_at": "2026-06-14T07:33:22.181034+00:00",
-  "train_finished_at": "2026-06-14T19:33:20.276955+00:00",
-  "finished_at": "2026-06-14T19:39:31.168271+00:00",
-  "train_duration_seconds": 43198.09588932991,
-  "train_duration_minutes": 719.9682648221652,
-  "train_duration_hours": 11.99947108036942,
-  "run_duration_seconds": 43568.98720574379,
-  "run_duration_minutes": 726.1497867623965,
-  "run_duration_hours": 12.102496446039941,
+  "started_at": "2026-06-16T11:27:32.527592+00:00",
+  "train_finished_at": "2026-06-16T21:48:26.431336+00:00",
+  "finished_at": "2026-06-16T21:52:22.912657+00:00",
+  "train_duration_seconds": 37253.9037168026,
+  "train_duration_minutes": 620.8983952800432,
+  "train_duration_hours": 10.348306588000721,
+  "run_duration_seconds": 37490.385033130646,
+  "run_duration_minutes": 624.8397505521774,
+  "run_duration_hours": 10.413995842536291,
   "train_metrics": {
-    "train_runtime": 43197.8762,
-    "train_samples_per_second": 48.627,
-    "train_steps_per_second": 4.052,
+    "train_runtime": 37253.6996,
+    "train_samples_per_second": 56.386,
+    "train_steps_per_second": 3.524,
     "total_flos": 0.0,
-    "train_loss": 0.446697575658745,
+    "train_loss": 0.37596295774437,
     "epoch": 8.0,
-    "train_wall_time_seconds": 43198.09588932991,
-    "train_wall_time_minutes": 719.9682648221652,
-    "train_wall_time_hours": 11.99947108036942,
-    "run_wall_time_seconds": 43568.98720574379,
-    "run_wall_time_minutes": 726.1497867623965,
-    "run_wall_time_hours": 12.102496446039941
+    "train_wall_time_seconds": 37253.9037168026,
+    "train_wall_time_minutes": 620.8983952800432,
+    "train_wall_time_hours": 10.348306588000721,
+    "run_wall_time_seconds": 37490.385033130646,
+    "run_wall_time_minutes": 624.8397505521774,
+    "run_wall_time_hours": 10.413995842536291
   },
   "eval_metrics": {
-    "eval_loss": 0.11423125863075256,
-    "eval_accuracy": 0.9731962296486718,
-    "eval_f1_macro": 0.97272463600211,
-    "eval_top5_accuracy": 0.9967780634104542,
-    "eval_runtime": 215.0703,
-    "eval_samples_per_second": 135.653,
-    "eval_steps_per_second": 11.308,
+    "eval_loss": 0.11002838611602783,
+    "eval_accuracy": 0.9730591259640102,
+    "eval_f1_macro": 0.9727023139698624,
+    "eval_top5_accuracy": 0.996846615252785,
+    "eval_runtime": 216.4171,
+    "eval_samples_per_second": 134.809,
+    "eval_steps_per_second": 8.428,
     "epoch": 8.0
   }
 }

train_results.json

@@ -1,14 +1,14 @@
 {
     "epoch": 8.0,
-    "run_wall_time_hours": 12.102496446039941,
-    "run_wall_time_minutes": 726.1497867623965,
-    "run_wall_time_seconds": 43568.98720574379,
+    "run_wall_time_hours": 10.413995842536291,
+    "run_wall_time_minutes": 624.8397505521774,
+    "run_wall_time_seconds": 37490.385033130646,
     "total_flos": 0.0,
-    "train_loss": 0.446697575658745,
-    "train_runtime": 43197.8762,
-    "train_samples_per_second": 48.627,
-    "train_steps_per_second": 4.052,
-    "train_wall_time_hours": 11.99947108036942,
-    "train_wall_time_minutes": 719.9682648221652,
-    "train_wall_time_seconds": 43198.09588932991
+    "train_loss": 0.37596295774437,
+    "train_runtime": 37253.6996,
+    "train_samples_per_second": 56.386,
+    "train_steps_per_second": 3.524,
+    "train_wall_time_hours": 10.348306588000721,
+    "train_wall_time_minutes": 620.8983952800432,
+    "train_wall_time_seconds": 37253.9037168026
 }