# Smart Plate — Model Development Documentation

Technical documentation of the image-classification model that powers **Smart Plate**
(*Indian Food Classification and Calorie Estimation*). It covers the complete model
development workflow as implemented in
[`notebook/indian-food-image-detection-vit.ipynb`](notebook/indian-food-image-detection-vit.ipynb).

---

## Project Overview

**Objective.** Build an image-classification model that identifies an Indian dish from a
single food photograph, so that the dish can be mapped to its calorie / nutritional
profile in the Smart Plate application.

**Problem being solved.** Manually identifying a dish and looking up its nutritional value
is slow and error-prone. This project automates the first half of that task — *recognising
the dish from a photo* — across 80 popular Indian food categories. The predicted label is
later joined with a calorie database in the application layer to estimate nutrition.

The model is built by **fine-tuning a Vision Transformer (ViT)** image classifier.

---

## Dataset Information

| Property | Value |
|---|---|
| Source | Kaggle — *Indian Food Images Dataset* (`indian-food-images-dataset`) |
| Path used | `Indian Food Images/Indian Food Images/*/*.jpg` |
| Total samples | **4,000 images** |
| Classes | **80** Indian food categories |
| Samples per class | 50 (balanced) |
| Image format | RGB JPEG, variable resolution |
| Train / test split | **60% / 40%**, stratified by label, shuffled |
| Train samples | 2,400 |
| Test samples | 1,600 (20 per class) |

**Classes (80):** adhirasam, aloo_gobi, aloo_matar, aloo_methi, aloo_shimla_mirch,
aloo_tikki, anarsa, ariselu, bandar_laddu, basundi, bhatura, bhindi_masala, biryani,
boondi, butter_chicken, chak_hao_kheer, cham_cham, chana_masala, chapati, chhena_kheeri,
chicken_razala, chicken_tikka, chicken_tikka_masala, chikki, daal_baati_churma, daal_puri,
dal_makhani, dal_tadka, dharwad_pedha, doodhpak, double_ka_meetha, dum_aloo, gajar_ka_halwa,
gavvalu, ghevar, gulab_jamun, imarti, jalebi, kachori, kadai_paneer, kadhi_pakoda, kajjikaya,
kakinada_khaja, kalakand, karela_bharta, kofta, kuzhi_paniyaram, lassi, ledikeni, litti_chokha,
lyangcha, maach_jhol, makki_di_roti_sarson_da_saag, malapua, misi_roti, misti_doi, modak,
mysore_pak, naan, navrattan_korma, palak_paneer, paneer_butter_masala, phirni, pithe, poha,
poornalu, pootharekulu, qubani_ka_meetha, rabri, ras_malai, rasgulla, sandesh, shankarpali,
sheer_korma, sheera, shrikhand, sohan_halwa, sohan_papdi, sutar_feni, unni_appam.

**Preprocessing performed at load time:**
- `ImageFile.LOAD_TRUNCATED_IMAGES = True` to tolerate truncated/corrupted JPEGs.
- File paths and parent-folder names collected into a Pandas DataFrame (`image`, `label`).
- **Random oversampling** (`imblearn.RandomOverSampler`, `random_state=83`) applied for class
  balancing. The dataset was already balanced (50/class), so the count remained 4,000.
- Conversion to a HuggingFace `Dataset` with the `image` column cast to `Image` and the
  `label` column cast to a `ClassLabel` (string ↔ integer id mapping).

---

## Tech Stack

**Language:** Python 3

**Core libraries & frameworks:**
- **PyTorch** + **torchvision** — deep-learning backend and image transforms
- **HuggingFace Transformers** — `ViTForImageClassification`, `ViTImageProcessor`, `Trainer`, `TrainingArguments`, `pipeline`
- **HuggingFace Datasets** — dataset construction, `ClassLabel`, stratified split, on-the-fly transforms
- **HuggingFace Evaluate** — accuracy metric
- **Accelerate** — training backend
- **scikit-learn** — accuracy, F1, confusion matrix, classification report
- **imbalanced-learn (imblearn)** — `RandomOverSampler`
- **pandas / numpy** — data handling
- **matplotlib** — confusion-matrix visualisation
- **Pillow (PIL)** / **tqdm** — image I/O and progress bars
- **huggingface_hub** — model publishing

**Development environment:** Kaggle Notebook with a single CUDA GPU (`device=0`).

---

## Methodology

The workflow, in the order performed in the notebook:

### 1. Data Loading
All `*/*.jpg` files were globbed from the dataset directory; the parent folder name of each
file was used as its class label. Paths and labels were assembled into a DataFrame
(`4000 rows × 2 columns`).

### 2. Data Exploration & Visualization
- Inspected the DataFrame head and the list of 80 unique labels.
- Rendered a sample image to confirm correct loading.
- Built and printed `id2label` / `label2id` mappings.
- After training, a **confusion matrix** (80×80) and a per-class **classification report**
  were generated for analysis.

### 3. Data Cleaning & Preprocessing
- Enabled truncated-image loading.
- Oversampled to enforce class balance.
- Cast columns to `Image` and `ClassLabel`; encoded string labels to integer ids.

### 4. Data Augmentation
Augmentation was applied **on-the-fly to the training split only**, via
`dataset.set_transform`:

| Transform | Train | Validation/Test |
|---|---|---|
| `Resize((224, 224))` | ✅ | ✅ |
| `RandomRotation(90)` | ✅ | — |
| `RandomAdjustSharpness(2)` | ✅ | — |
| `RandomHorizontalFlip(0.5)` | ✅ | — |
| `ToTensor()` | ✅ | ✅ |
| `Normalize(mean, std)` | ✅ | ✅ |

`mean`/`std` are taken from the pretrained `ViTImageProcessor`.

### 5. Feature Engineering
Not applicable in the classical sense — the Vision Transformer learns visual feature
representations directly from pixels. "Feature engineering" here is limited to the
resize/normalise/augment pipeline above.

### 6. Model Selection
A **Vision Transformer (ViT)** was selected for image classification. The underlying base
architecture is `google/vit-base-patch16-224-in21k`. In the notebook, training starts from
the task-specific checkpoint `aishrica/indian_food_prediction`, a ViT-Base classifier
already aligned to this exact 80-class Indian-food task.

### 7. Model Architecture
`ViTForImageClassification` — a ViT-Base encoder (patch size 16, 224×224 input) with a
linear classification head re-initialised for `num_labels = 80`.

### 8. Training Procedure
- A custom `collate_fn` stacks `pixel_values` and `labels` into batches.
- The HuggingFace `Trainer` orchestrates training with `compute_metrics` (accuracy).
- The model was **evaluated before training** (baseline) and **after every epoch**; the best
  checkpoint (by accuracy) was reloaded at the end.
- Total optimisation: **760 steps over 20 epochs**, ~31 minutes runtime on GPU.

### 9. Hyperparameter Configuration

| Hyperparameter | Value |
|---|---|
| Epochs | 20 |
| Learning rate | 1e-6 |
| Train batch size | 64 |
| Eval batch size | 32 |
| Weight decay | 0.02 |
| Warmup steps | 50 |
| Evaluation strategy | per epoch |
| Save strategy | per epoch (`save_total_limit=1`) |
| `load_best_model_at_end` | True (metric = accuracy) |
| `remove_unused_columns` | False |

### 10. Validation Strategy
A single **stratified 60/40 split** was used. The 40% split serves as the `eval_dataset`
during training (per-epoch evaluation) and as the final test set. Best-model selection is
driven by validation accuracy. *(No separate, fully held-out test set was reserved — see
Limitations.)*

### 11. Evaluation Metrics
Accuracy (HuggingFace `evaluate`) during training; **accuracy, macro-F1, confusion matrix,
and a full per-class precision/recall/F1 report** (scikit-learn) at the end.

### 12. Testing & Inference
`trainer.predict` produced final test metrics. A `transformers.pipeline`
(`image-classification`) was then used to run single-image inference, validated against the
ground-truth label.

---

## Model Details

| Attribute | Value |
|---|---|
| Base model | `google/vit-base-patch16-224-in21k` |
| Fine-tuning checkpoint | `indian_food_prediction` (published as `aishrica/indian_food_prediction`) |
| Architecture | Vision Transformer — ViT-Base, patch 16, 224×224 (`ViTForImageClassification`) |
| Trainable parameters | **≈ 85.86 M** |
| Weights file | `model.safetensors` (~343 MB) |
| Input | RGB image, resized to 224×224, normalised with ViT mean/std → tensor `[3, 224, 224]` |
| Output | Logits over **80 classes** → softmax probabilities; argmax = predicted dish |

---

## Results

**Baseline (before fine-tuning):**
- eval_loss: 3.1995 · eval_accuracy: **0.7494**

**After fine-tuning (20 epochs):**
- train_loss: 3.1225 (final)
- eval_loss: 3.1539 · eval_accuracy: **0.7519**

**Final test evaluation:**

| Metric | Value |
|---|---|
| Test accuracy | **0.7519** |
| Macro F1 | **0.7352** |
| Test loss | 3.1539 |

**Observations & findings:**
- The task-specific checkpoint was already fitted to this dataset, and the learning rate was very
  small (1e-6); consequently fine-tuning produced only a marginal gain (**74.94% → 75.19%**).
- Performance is **highly class-dependent.** Strong classes (F1 ≈ 0.90+) include
  bhindi_masala, boondi, chak_hao_kheer, jalebi, imarti, makki_di_roti_sarson_da_saag.
  Weak classes include `chhena_kheeri` (F1 = 0.00), doodhpak, karela_bharta, ledikeni —
  typically dishes that are visually similar to others (e.g. curries and milk-based sweets).
- **Predicted probabilities are poorly calibrated / very flat.** In the inference example the
  correct top-1 prediction carried a confidence of only ~0.04, which is consistent with the
  high cross-entropy loss (~3.15) despite ~75% top-1 accuracy. Top-1 *ranking* is reliable;
  the absolute confidence scores should not be read as probabilities.

---

## Inference Pipeline

**How predictions are generated.** A `transformers.pipeline('image-classification')` loads
the saved model and label mappings. An input image is resized/normalised by the ViT
processor, passed through the encoder + classification head, and softmax scores are returned
ranked by probability.

**Input format:** a single image (PIL `Image`, file path, or URL).

**Output format:** a list of `{ "label": <class>, "score": <float> }` objects sorted by
descending score. Example:

```json
[
  { "label": "butter_chicken",        "score": 0.0434 },
  { "label": "chicken_tikka_masala",  "score": 0.0335 },
  { "label": "chana_masala",          "score": 0.0264 },
  { "label": "chicken_tikka",         "score": 0.0256 },
  { "label": "paneer_butter_masala",  "score": 0.0226 }
]
```

In the Smart Plate application the top label (`raw_label`) is joined with `calorie_db.json`
to return the dish category and calorie estimate.

---

## Project Structure

```
indian-food-calories/
├── notebook/
│   └── indian-food-image-detection-vit.ipynb   ← model development (this document)
├── models/
│   └── aishrica_food_predictor/                ← bundled ViT weights served by the app
├── app.py                                       ← FastAPI inference backend (/predict, /foods)
├── calorie_db.json                              ← calorie data for the 80 classes
├── static/                                      ← Smart Plate web UI (pages/css/js)
├── requirements.txt
├── README.md
└── DOC.md                                       ← this file
```

> Note: the notebook trains/publishes `indian_food_prediction`; the application serves
> a bundled equivalent ViT model from `models/aishrica_food_predictor/`. Both are ViT-Base
> classifiers over the same 80 Indian-food classes.

---

## Reproducibility

**Requirements.** Python 3, a CUDA GPU (recommended), and:

```bash
pip install -U evaluate "transformers" "datasets>=2.14.5" "accelerate>=0.27" \
    torch torchvision scikit-learn imbalanced-learn pandas numpy matplotlib pillow tqdm
```

**Steps to run the notebook:**
1. Download the *Indian Food Images Dataset* from Kaggle and update the dataset glob path in
   the data-loading cell if you are not running on Kaggle.
2. Open `notebook/indian-food-image-detection-vit.ipynb`.
3. Run the cells in order: install → imports → data loading → preprocessing/oversampling →
   split → transforms → model load → baseline eval → `trainer.train()` → evaluation/report →
   `trainer.save_model()`.
4. (Optional) Run the final cells to log in to the HuggingFace Hub and push the model.

The trained model is saved to the `indian_food_prediction/` output directory and can be
loaded directly for inference.

---

## Limitations

- **Marginal fine-tuning effect.** With a near-converged task-specific checkpoint and lr = 1e-6,
  training added only ~0.25% accuracy — the run is closer to verification than substantive
  fine-tuning.
- **Poorly calibrated confidence.** Softmax scores are very flat (top scores ≈ 0.04), so they
  cannot be presented to users as reliable probabilities without temperature scaling.
- **No held-out test set.** The same 40% split is used for per-epoch validation and final
  testing, which can optimistically bias the reported metrics.
- **Class confusion.** Visually similar dishes (curries; milk-based sweets) are frequently
  confused; at least one class (`chhena_kheeri`) scores F1 = 0.
- **Small dataset.** 50 images/class limits generalisation to real-world photos with varied
  lighting, plating, and backgrounds.
- **Single-dish assumption.** The model classifies one dish per image; it does not segment or
  detect multiple items on a plate.

---

## Future Improvements

- **Reserve a separate test set** (train/validation/test) for unbiased evaluation.
- **Tune the learning rate / schedule** (e.g. 1e-5–5e-5 with discriminative LR) and add early
  stopping for meaningful fine-tuning.
- **Calibrate outputs** (temperature scaling or label smoothing) so confidence scores are
  trustworthy in the UI.
- **Expand and diversify the dataset** — more images per class and real "in-the-wild" photos;
  stronger augmentation (color jitter, random erasing, mixup/cutmix).
- **Address weak classes** with targeted data collection and per-class error analysis.
- **Move toward food segmentation / multi-item detection** (e.g. instance segmentation)
  to handle full plates beyond the current single-dish classification setup.
- **Add portion/volume estimation** from images to improve the downstream calorie estimate.
- **Export an optimised model** (ONNX / quantisation) to speed up and shrink the served model.

---

*Document generated from the notebook’s executed code and outputs. Metrics reflect the recorded
run (20 epochs, test accuracy 0.7519, macro-F1 0.7352).*