ci: deploy README.md
Browse files
README.md
CHANGED
|
@@ -1,14 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
sdk_version: 6.10.0
|
| 8 |
-
app_file: app.py
|
| 9 |
-
pinned: false
|
| 10 |
---
|
| 11 |
|
| 12 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
-
|
|
|
|
| 1 |
+
# 🖼️ Image Captioning — ViT + GPT-2
|
| 2 |
+
|
| 3 |
+
An image captioning pipeline using a **Vision Transformer (ViT)** encoder and **GPT-2**
|
| 4 |
+
decoder. Upload any image and get an AI-generated natural language description with
|
| 5 |
+
adjustable beam search and caption count controls.
|
| 6 |
+
|
| 7 |
---
|
| 8 |
+
|
| 9 |
+
## 🌐 Live Demo
|
| 10 |
+
|
| 11 |
+
👉 **[Try it live on HF Spaces](https://amarshiv86-image-captioning-vit-gpt2.hf.space)**
|
| 12 |
+
|
|
|
|
|
|
|
|
|
|
| 13 |
---
|
| 14 |
|
| 15 |
+
## 🏗️ How It Works
|
| 16 |
+
|
| 17 |
+
```
|
| 18 |
+
Image Input
|
| 19 |
+
↓
|
| 20 |
+
ViT — encodes image into patch embeddings
|
| 21 |
+
↓
|
| 22 |
+
GPT-2 — decodes embeddings into natural language tokens
|
| 23 |
+
↓
|
| 24 |
+
Beam Search — selects best caption from multiple candidates
|
| 25 |
+
↓
|
| 26 |
+
Natural Language Caption
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
---
|
| 30 |
+
|
| 31 |
+
## 🛠️ Tech Stack
|
| 32 |
+
|
| 33 |
+
| Layer | Technology |
|
| 34 |
+
|---|---|
|
| 35 |
+
| Vision encoder | ViT (Vision Transformer) |
|
| 36 |
+
| Text decoder | GPT-2 |
|
| 37 |
+
| Framework | HuggingFace Transformers 5.4.0 |
|
| 38 |
+
| UI | Gradio 6.10.0 |
|
| 39 |
+
| CI/CD | GitHub Actions |
|
| 40 |
+
| Hosting | Hugging Face Spaces |
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## 📁 Project Structure
|
| 45 |
+
|
| 46 |
+
```
|
| 47 |
+
image-captioning/
|
| 48 |
+
├── app.py # Gradio app — model loading + inference + UI
|
| 49 |
+
├── requirements.txt # Pinned dependencies
|
| 50 |
+
├── .gitignore
|
| 51 |
+
└── .github/
|
| 52 |
+
└── workflows/
|
| 53 |
+
└── deploy.yml # Auto-deploy to HF Spaces on push
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
---
|
| 57 |
+
|
| 58 |
+
## 🚀 Run Locally
|
| 59 |
+
|
| 60 |
+
```bash
|
| 61 |
+
git clone https://github.com/amarshiv86/image-captioning
|
| 62 |
+
cd image-captioning
|
| 63 |
+
pip install -r requirements.txt
|
| 64 |
+
python app.py
|
| 65 |
+
# → open the gradio.live link printed in terminal
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
---
|
| 69 |
+
|
| 70 |
+
## ⚙️ Features
|
| 71 |
+
|
| 72 |
+
- **Beam search** width 1–8 — higher = more accurate captions
|
| 73 |
+
- **Multiple captions** — generate up to 4 to compare model confidence
|
| 74 |
+
- Auto-detects GPU (CUDA) or falls back to CPU
|
| 75 |
+
- No custom training — uses pre-trained model from HF Hub
|
| 76 |
+
|
| 77 |
+
---
|
| 78 |
+
|
| 79 |
+
## 🤖 Model
|
| 80 |
+
|
| 81 |
+
[`nlpconnect/vit-gpt2-image-captioning`](https://huggingface.co/nlpconnect/vit-gpt2-image-captioning)
|
| 82 |
+
— a pre-trained VisionEncoderDecoderModel combining ViT and GPT-2,
|
| 83 |
+
fine-tuned on COCO image captions (~118k images).
|
| 84 |
+
|
| 85 |
+
---
|
| 86 |
+
|
| 87 |
+
## 🔗 Part of AI Engineer Portfolio
|
| 88 |
+
|
| 89 |
+
| Project | Description |
|
| 90 |
+
|---|---|
|
| 91 |
+
| P1 — [Weather Prediction](https://github.com/amarshiv86/weather-mlops-pipeline) | Classical ML + full MLOps pipeline |
|
| 92 |
+
| P2 — [Sentiment Analysis](https://github.com/amarshiv86/sentiment-analysis-mlops-pipeline) | Fine-tuned distilBERT + drift detection |
|
| 93 |
+
| P3 — Image Captioning (this repo) | Multi-modal ViT+GPT2 inference |
|
| 94 |
+
|
| 95 |
+
---
|
| 96 |
+
|
| 97 |
+
## 📄 License
|
| 98 |
|
| 99 |
+
MIT — free to use, modify, and distribute.
|