amarshiv86 commited on
Commit
3d2e1c2
·
verified ·
1 Parent(s): 38d7e53

ci: deploy README.md

Browse files
Files changed (1) hide show
  1. README.md +95 -10
README.md CHANGED
@@ -1,14 +1,99 @@
 
 
 
 
 
 
1
  ---
2
- title: Image Captioning ViT GPT2
3
- emoji: 🖼️
4
- colorFrom: blue
5
- colorTo: indigo
6
- sdk: gradio
7
- sdk_version: 6.10.0
8
- app_file: app.py
9
- pinned: false
10
  ---
11
 
12
- # 🖼Image Captioning — ViT + GPT-2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
 
14
- Upload any image and get an AI-generated caption using Vision Transformer + GPT-2.
 
1
+ # 🖼️ Image Captioning — ViT + GPT-2
2
+
3
+ An image captioning pipeline using a **Vision Transformer (ViT)** encoder and **GPT-2**
4
+ decoder. Upload any image and get an AI-generated natural language description with
5
+ adjustable beam search and caption count controls.
6
+
7
  ---
8
+
9
+ ## 🌐 Live Demo
10
+
11
+ 👉 **[Try it live on HF Spaces](https://amarshiv86-image-captioning-vit-gpt2.hf.space)**
12
+
 
 
 
13
  ---
14
 
15
+ ## 🏗How It Works
16
+
17
+ ```
18
+ Image Input
19
+
20
+ ViT — encodes image into patch embeddings
21
+
22
+ GPT-2 — decodes embeddings into natural language tokens
23
+
24
+ Beam Search — selects best caption from multiple candidates
25
+
26
+ Natural Language Caption
27
+ ```
28
+
29
+ ---
30
+
31
+ ## 🛠️ Tech Stack
32
+
33
+ | Layer | Technology |
34
+ |---|---|
35
+ | Vision encoder | ViT (Vision Transformer) |
36
+ | Text decoder | GPT-2 |
37
+ | Framework | HuggingFace Transformers 5.4.0 |
38
+ | UI | Gradio 6.10.0 |
39
+ | CI/CD | GitHub Actions |
40
+ | Hosting | Hugging Face Spaces |
41
+
42
+ ---
43
+
44
+ ## 📁 Project Structure
45
+
46
+ ```
47
+ image-captioning/
48
+ ├── app.py # Gradio app — model loading + inference + UI
49
+ ├── requirements.txt # Pinned dependencies
50
+ ├── .gitignore
51
+ └── .github/
52
+ └── workflows/
53
+ └── deploy.yml # Auto-deploy to HF Spaces on push
54
+ ```
55
+
56
+ ---
57
+
58
+ ## 🚀 Run Locally
59
+
60
+ ```bash
61
+ git clone https://github.com/amarshiv86/image-captioning
62
+ cd image-captioning
63
+ pip install -r requirements.txt
64
+ python app.py
65
+ # → open the gradio.live link printed in terminal
66
+ ```
67
+
68
+ ---
69
+
70
+ ## ⚙️ Features
71
+
72
+ - **Beam search** width 1–8 — higher = more accurate captions
73
+ - **Multiple captions** — generate up to 4 to compare model confidence
74
+ - Auto-detects GPU (CUDA) or falls back to CPU
75
+ - No custom training — uses pre-trained model from HF Hub
76
+
77
+ ---
78
+
79
+ ## 🤖 Model
80
+
81
+ [`nlpconnect/vit-gpt2-image-captioning`](https://huggingface.co/nlpconnect/vit-gpt2-image-captioning)
82
+ — a pre-trained VisionEncoderDecoderModel combining ViT and GPT-2,
83
+ fine-tuned on COCO image captions (~118k images).
84
+
85
+ ---
86
+
87
+ ## 🔗 Part of AI Engineer Portfolio
88
+
89
+ | Project | Description |
90
+ |---|---|
91
+ | P1 — [Weather Prediction](https://github.com/amarshiv86/weather-mlops-pipeline) | Classical ML + full MLOps pipeline |
92
+ | P2 — [Sentiment Analysis](https://github.com/amarshiv86/sentiment-analysis-mlops-pipeline) | Fine-tuned distilBERT + drift detection |
93
+ | P3 — Image Captioning (this repo) | Multi-modal ViT+GPT2 inference |
94
+
95
+ ---
96
+
97
+ ## 📄 License
98
 
99
+ MIT free to use, modify, and distribute.