README.md

---
title: Pet Classification Comparison
emoji: 🐾
colorFrom: purple
colorTo: gray
sdk: gradio
sdk_version: 6.12.0
app_file: app.py
pinned: false
short_description: ViT vs CLIP vs OpenAI on 3 custom pet classes
---

# Pet Classification Comparison

This app compares 3 image classification approaches on pet images:

- Fine-tuned ViT model ([vasanthi8134/oxford-pets-3class-vit](https://huggingface.co/vasanthi8134/oxford-pets-3class-vit))
- Zero-shot CLIP model (`openai/clip-vit-base-patch32`)
- OpenAI vision model (LLM image classification)

## Dataset Used For Training

- Hugging Face dataset loader: `load_dataset("pcuenq/oxford-pets")`
- Original dataset source: Oxford-IIIT Pet dataset
- Dataset used in this project: **custom 3-class subset** based on Oxford-IIIT Pet
- Selected classes:
  - `Egyptian Mau`
  - `leonberger`
  - `samoyed`
- Number of classes: **3**
- Total images: **90**

### Custom Split

The custom subset was created by selecting **30 images per class** and splitting them into:

- **Train:** 60 images total (**20 per class**)
- **Validation:** 15 images total (**5 per class**)
- **Test:** 15 images total (**5 per class**)

## Preprocessing Steps

### Training transforms
- Random resized crop
- Random horizontal flip
- Conversion to tensor
- Normalization with ViT image processor values

### Validation / Test transforms
- Resize
- Center crop
- Conversion to tensor
- Normalization with ViT image processor values

## Trained Model

- Base model: `google/vit-base-patch16-224-in21k`
- Approach: **transfer learning / fine-tuning**
- Fine-tuned model link: [https://huggingface.co/vasanthi8134/oxford-pets-3class-vit](https://huggingface.co/vasanthi8134/oxford-pets-3class-vit)

## Training Performance

### Training Setup

| Parameter | Value |
|---|---:|
| Epochs | 5 |
| Learning rate | 5e-5 |
| Batch size | 8 |

### Final Evaluation

| Metric | Value |
|---|---:|
| Validation accuracy | 1.0 |
| Test accuracy | 1.0 |

Because this project uses a small and simplified custom subset with only 3 classes, the fine-tuned model performs very well on this task.

## Evaluation Method

The final model was evaluated on:
- a **validation split** during training
- a separate **test split** after training

The model with the best validation performance was used as the final selected model.

## Example Image Results

The table below reports example predictions from all 3 approaches.

| Image | True Class | ViT Prediction | CLIP Prediction | OpenAI Prediction |
|---|---|---|---|---|
| `leonberger.jpg` | leonberger | leonberger (0.4457) | leonberger (1.0) | leonberger (0.95) |
| `Egyptian_Mau.jpg` | Egyptian Mau | Egyptian Mau (0.4171) | Egyptian Mau (1.0) | Egyptian Mau (0.95) |

## Model Comparison

This application compares:

1. **My fine-tuned ViT model**
2. **CLIP zero-shot classification**
3. **OpenAI vision classification**

### Short comparison

- **My fine-tuned ViT model** is specialized for the selected 3 classes because it was trained on the custom subset.
- **CLIP** works in a zero-shot setting and still performs well on clear images without task-specific fine-tuning.
- **OpenAI vision** also performs well and returns a label, confidence score, and short reasoning.

## Hugging Face Links

### Model
[https://huggingface.co/vasanthi8134/oxford-pets-3class-vit](https://huggingface.co/vasanthi8134/oxford-pets-3class-vit)

### App
[https://huggingface.co/spaces/vasanthi8134/pet-classification-comparison](https://huggingface.co/spaces/vasanthi8134/pet-classification-comparison)

## Application Features

The Hugging Face Space includes:
- image upload
- prediction from the fine-tuned ViT model
- prediction from the zero-shot CLIP model
- prediction from the OpenAI vision model
- example images for quick testing
- JSON output for direct comparison

## Final Selected Model

The final selected model for the custom classification task is:

- **ViT fine-tuned on the custom 3-class Oxford-IIIT Pet subset**

It was selected because it is the project-specific transfer learning model required by the assignment and achieved perfect accuracy on the simplified validation and test splits.

## Notes

This is a simplified educational computer vision project created to demonstrate:

- transfer learning on custom data
- Hugging Face model deployment
- Hugging Face Space deployment
- comparison between open-source and closed-source image classification approaches