DOC.md

Smart Plate — Model Development Documentation

Technical documentation of the image-classification model that powers Smart Plate (Indian Food Classification and Calorie Estimation). It covers the complete model development workflow as implemented in notebook/indian-food-image-detection-vit.ipynb.

---

Project Overview

Objective. Build an image-classification model that identifies an Indian dish from a single food photograph, so that the dish can be mapped to its calorie / nutritional profile in the Smart Plate application.

Problem being solved. Manually identifying a dish and looking up its nutritional value is slow and error-prone. This project automates the first half of that task — recognising the dish from a photo — across 80 popular Indian food categories. The predicted label is later joined with a calorie database in the application layer to estimate nutrition.

The model is built by fine-tuning a Vision Transformer (ViT) image classifier.

---

Dataset Information

Property	Value
Source	Kaggle — Indian Food Images Dataset (indian-food-images-dataset)
Path used	Indian Food Images/Indian Food Images/*/*.jpg
Total samples	4,000 images
Classes	80 Indian food categories
Samples per class	50 (balanced)
Image format	RGB JPEG, variable resolution
Train / test split	60% / 40%, stratified by label, shuffled
Train samples	2,400
Test samples	1,600 (20 per class)

Classes (80): adhirasam, aloo_gobi, aloo_matar, aloo_methi, aloo_shimla_mirch, aloo_tikki, anarsa, ariselu, bandar_laddu, basundi, bhatura, bhindi_masala, biryani, boondi, butter_chicken, chak_hao_kheer, cham_cham, chana_masala, chapati, chhena_kheeri, chicken_razala, chicken_tikka, chicken_tikka_masala, chikki, daal_baati_churma, daal_puri, dal_makhani, dal_tadka, dharwad_pedha, doodhpak, double_ka_meetha, dum_aloo, gajar_ka_halwa, gavvalu, ghevar, gulab_jamun, imarti, jalebi, kachori, kadai_paneer, kadhi_pakoda, kajjikaya, kakinada_khaja, kalakand, karela_bharta, kofta, kuzhi_paniyaram, lassi, ledikeni, litti_chokha, lyangcha, maach_jhol, makki_di_roti_sarson_da_saag, malapua, misi_roti, misti_doi, modak, mysore_pak, naan, navrattan_korma, palak_paneer, paneer_butter_masala, phirni, pithe, poha, poornalu, pootharekulu, qubani_ka_meetha, rabri, ras_malai, rasgulla, sandesh, shankarpali, sheer_korma, sheera, shrikhand, sohan_halwa, sohan_papdi, sutar_feni, unni_appam.

Preprocessing performed at load time:

• ImageFile.LOAD_TRUNCATED_IMAGES = True to tolerate truncated/corrupted JPEGs.
• File paths and parent-folder names collected into a Pandas DataFrame (image, label).
• Random oversampling (imblearn.RandomOverSampler, random_state=83) applied for class balancing. The dataset was already balanced (50/class), so the count remained 4,000.
• Conversion to a HuggingFace Dataset with the image column cast to Image and the label column cast to a ClassLabel (string ↔ integer id mapping).

---

Tech Stack

Language: Python 3

Core libraries & frameworks:

• PyTorch + torchvision — deep-learning backend and image transforms
• HuggingFace Transformers — ViTForImageClassification, ViTImageProcessor, Trainer, TrainingArguments, pipeline
• HuggingFace Datasets — dataset construction, ClassLabel, stratified split, on-the-fly transforms
• HuggingFace Evaluate — accuracy metric
• Accelerate — training backend
• scikit-learn — accuracy, F1, confusion matrix, classification report
• imbalanced-learn (imblearn) — RandomOverSampler
• pandas / numpy — data handling
• matplotlib — confusion-matrix visualisation
• Pillow (PIL) / tqdm — image I/O and progress bars
• huggingface_hub — model publishing

Development environment: Kaggle Notebook with a single CUDA GPU (device=0).

---

Methodology

The workflow, in the order performed in the notebook:

1. Data Loading

All */*.jpg files were globbed from the dataset directory; the parent folder name of each file was used as its class label. Paths and labels were assembled into a DataFrame (4000 rows × 2 columns).

2. Data Exploration & Visualization

• Inspected the DataFrame head and the list of 80 unique labels.
• Rendered a sample image to confirm correct loading.
• Built and printed id2label / label2id mappings.
• After training, a confusion matrix (80×80) and a per-class classification report were generated for analysis.

3. Data Cleaning & Preprocessing

• Enabled truncated-image loading.
• Oversampled to enforce class balance.
• Cast columns to Image and ClassLabel; encoded string labels to integer ids.

4. Data Augmentation

Augmentation was applied on-the-fly to the training split only, via dataset.set_transform:

Transform	Train	Validation/Test
Resize((224, 224))	✅	✅
RandomRotation(90)	✅	—
RandomAdjustSharpness(2)	✅	—
RandomHorizontalFlip(0.5)	✅	—
ToTensor()	✅	✅
Normalize(mean, std)	✅	✅

mean/std are taken from the pretrained ViTImageProcessor.

5. Feature Engineering

Not applicable in the classical sense — the Vision Transformer learns visual feature representations directly from pixels. "Feature engineering" here is limited to the resize/normalise/augment pipeline above.

6. Model Selection

A Vision Transformer (ViT) was selected for image classification. The underlying base architecture is google/vit-base-patch16-224-in21k. In the notebook, training starts from the task-specific checkpoint aishrica/indian_food_prediction, a ViT-Base classifier already aligned to this exact 80-class Indian-food task.

7. Model Architecture

ViTForImageClassification — a ViT-Base encoder (patch size 16, 224×224 input) with a linear classification head re-initialised for num_labels = 80.

8. Training Procedure

• A custom collate_fn stacks pixel_values and labels into batches.
• The HuggingFace Trainer orchestrates training with compute_metrics (accuracy).
• The model was evaluated before training (baseline) and after every epoch; the best checkpoint (by accuracy) was reloaded at the end.
• Total optimisation: 760 steps over 20 epochs, ~31 minutes runtime on GPU.

9. Hyperparameter Configuration

Hyperparameter	Value
Epochs	20
Learning rate	1e-6
Train batch size	64
Eval batch size	32
Weight decay	0.02
Warmup steps	50
Evaluation strategy	per epoch
Save strategy	per epoch (save_total_limit=1)
load_best_model_at_end	True (metric = accuracy)
remove_unused_columns	False

10. Validation Strategy

A single stratified 60/40 split was used. The 40% split serves as the eval_dataset during training (per-epoch evaluation) and as the final test set. Best-model selection is driven by validation accuracy. (No separate, fully held-out test set was reserved — see Limitations.)

11. Evaluation Metrics

Accuracy (HuggingFace evaluate) during training; accuracy, macro-F1, confusion matrix, and a full per-class precision/recall/F1 report (scikit-learn) at the end.

12. Testing & Inference

trainer.predict produced final test metrics. A transformers.pipeline (image-classification) was then used to run single-image inference, validated against the ground-truth label.

---

Model Details

Attribute	Value
Base model	google/vit-base-patch16-224-in21k
Fine-tuning checkpoint	indian_food_prediction (published as aishrica/indian_food_prediction)
Architecture	Vision Transformer — ViT-Base, patch 16, 224×224 (ViTForImageClassification)
Trainable parameters	≈ 85.86 M
Weights file	model.safetensors (~343 MB)
Input	RGB image, resized to 224×224, normalised with ViT mean/std → tensor [3, 224, 224]
Output	Logits over 80 classes → softmax probabilities; argmax = predicted dish

---

Results

Baseline (before fine-tuning):

• eval_loss: 3.1995 · eval_accuracy: 0.7494

After fine-tuning (20 epochs):

• train_loss: 3.1225 (final)
• eval_loss: 3.1539 · eval_accuracy: 0.7519

Final test evaluation:

Metric	Value
Test accuracy	0.7519
Macro F1	0.7352
Test loss	3.1539

Observations & findings:

• The task-specific checkpoint was already fitted to this dataset, and the learning rate was very small (1e-6); consequently fine-tuning produced only a marginal gain (74.94% → 75.19%).
• Performance is highly class-dependent. Strong classes (F1 ≈ 0.90+) include bhindi_masala, boondi, chak_hao_kheer, jalebi, imarti, makki_di_roti_sarson_da_saag. Weak classes include chhena_kheeri (F1 = 0.00), doodhpak, karela_bharta, ledikeni — typically dishes that are visually similar to others (e.g. curries and milk-based sweets).
• Predicted probabilities are poorly calibrated / very flat. In the inference example the correct top-1 prediction carried a confidence of only 0.04, which is consistent with the high cross-entropy loss (3.15) despite ~75% top-1 accuracy. Top-1 ranking is reliable; the absolute confidence scores should not be read as probabilities.

---

Inference Pipeline

How predictions are generated. A transformers.pipeline('image-classification') loads the saved model and label mappings. An input image is resized/normalised by the ViT processor, passed through the encoder + classification head, and softmax scores are returned ranked by probability.

Input format: a single image (PIL Image, file path, or URL).

Output format: a list of { "label": <class>, "score": <float> } objects sorted by descending score. Example:

[
  { "label": "butter_chicken",        "score": 0.0434 },
  { "label": "chicken_tikka_masala",  "score": 0.0335 },
  { "label": "chana_masala",          "score": 0.0264 },
  { "label": "chicken_tikka",         "score": 0.0256 },
  { "label": "paneer_butter_masala",  "score": 0.0226 }
]

In the Smart Plate application the top label (raw_label) is joined with calorie_db.json to return the dish category and calorie estimate.

---

Project Structure

indian-food-calories/
├── notebook/
│   └── indian-food-image-detection-vit.ipynb   ← model development (this document)
├── models/
│   └── aishrica_food_predictor/                ← bundled ViT weights served by the app
├── app.py                                       ← FastAPI inference backend (/predict, /foods)
├── calorie_db.json                              ← calorie data for the 80 classes
├── static/                                      ← Smart Plate web UI (pages/css/js)
├── requirements.txt
├── README.md
└── DOC.md                                       ← this file

Note: the notebook trains/publishes indian_food_prediction; the application serves a bundled equivalent ViT model from models/aishrica_food_predictor/. Both are ViT-Base classifiers over the same 80 Indian-food classes.

---

Reproducibility

Requirements. Python 3, a CUDA GPU (recommended), and:

pip install -U evaluate "transformers" "datasets>=2.14.5" "accelerate>=0.27" \
    torch torchvision scikit-learn imbalanced-learn pandas numpy matplotlib pillow tqdm

Steps to run the notebook:

1. Download the Indian Food Images Dataset from Kaggle and update the dataset glob path in the data-loading cell if you are not running on Kaggle.
2. Open notebook/indian-food-image-detection-vit.ipynb.
3. Run the cells in order: install → imports → data loading → preprocessing/oversampling → split → transforms → model load → baseline eval → trainer.train() → evaluation/report → trainer.save_model().
4. (Optional) Run the final cells to log in to the HuggingFace Hub and push the model.

The trained model is saved to the indian_food_prediction/ output directory and can be loaded directly for inference.

---

Limitations

• Marginal fine-tuning effect. With a near-converged task-specific checkpoint and lr = 1e-6, training added only ~0.25% accuracy — the run is closer to verification than substantive fine-tuning.
• Poorly calibrated confidence. Softmax scores are very flat (top scores ≈ 0.04), so they cannot be presented to users as reliable probabilities without temperature scaling.
• No held-out test set. The same 40% split is used for per-epoch validation and final testing, which can optimistically bias the reported metrics.
• Class confusion. Visually similar dishes (curries; milk-based sweets) are frequently confused; at least one class (chhena_kheeri) scores F1 = 0.
• Small dataset. 50 images/class limits generalisation to real-world photos with varied lighting, plating, and backgrounds.
• Single-dish assumption. The model classifies one dish per image; it does not segment or detect multiple items on a plate.

---

Future Improvements

• Reserve a separate test set (train/validation/test) for unbiased evaluation.
• Tune the learning rate / schedule (e.g. 1e-5–5e-5 with discriminative LR) and add early stopping for meaningful fine-tuning.
• Calibrate outputs (temperature scaling or label smoothing) so confidence scores are trustworthy in the UI.
• Expand and diversify the dataset — more images per class and real "in-the-wild" photos; stronger augmentation (color jitter, random erasing, mixup/cutmix).
• Address weak classes with targeted data collection and per-class error analysis.
• Move toward food segmentation / multi-item detection (e.g. instance segmentation) to handle full plates beyond the current single-dish classification setup.
• Add portion/volume estimation from images to improve the downstream calorie estimate.
• Export an optimised model (ONNX / quantisation) to speed up and shrink the served model.

---

Document generated from the notebook’s executed code and outputs. Metrics reflect the recorded run (20 epochs, test accuracy 0.7519, macro-F1 0.7352).