File size: 2,597 Bytes
35c248d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 | ---
title: NanoVLM
emoji: 🤖
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: "5.0.0"
app_file: app.py
pinned: false
license: mit
---
# NanoVLM
NanoVLM is a lightweight Vision-Language Model (VLM) designed for efficient multimodal understanding. It combines image encoding and language generation to answer questions about images while remaining suitable for resource-constrained environments.
## Features
- 🖼️ Image understanding
- 💬 Visual Question Answering (VQA)
- ⚡ Lightweight architecture
- 🤗 Hugging Face compatible
- 📓 Training and inference notebook included
## Model Architecture
The model consists of:
- Vision Encoder
- Projection Layer
- Language Model
- Cross-modal fusion between visual and textual representations
```
Image
│
Vision Encoder
│
Projection Layer
│
Language Model
│
Generated Answer
```
## Installation
Clone the repository
```bash
git clone https://github.com/your-username/nanovlm.git
cd nanovlm
```
Install dependencies
```bash
pip install -r requirements.txt
```
## Usage
### Inference
```python
from PIL import Image
image = Image.open("example.jpg")
question = "What is in this image?"
answer = model.chat(image, question)
print(answer)
```
### Notebook
Open the notebook:
```
NanoVlm_fixed.ipynb
```
and execute all cells.
## Dataset
The model can be trained on datasets such as:
- COCO Captions
- VQAv2
- Flickr30k
- Custom datasets
## Training
Example:
```bash
python train.py
```
Adjust hyperparameters such as:
- Learning rate
- Batch size
- Number of epochs
- Image resolution
## Results
| Metric | Value |
|---------|------:|
| Accuracy | -- |
| BLEU | -- |
| CIDEr | -- |
*(Replace with your experimental results.)*
## Repository Structure
```
.
├── NanoVlm_fixed.ipynb
├── app.py
├── requirements.txt
├── README.md
├── images/
└── model/
```
## Requirements
- Python 3.10+
- PyTorch
- Transformers
- Pillow
- Torchvision
- Gradio (optional)
Install:
```bash
pip install torch torchvision transformers pillow gradio
```
## Citation
If you use this project, please cite:
```bibtex
@misc{nanovlm2026,
title={NanoVLM: A Lightweight Vision-Language Model},
author={Your Name},
year={2026},
publisher={Hugging Face}
}
```
## License
MIT License
## Acknowledgements
This project builds upon the excellent work of:
- Hugging Face Transformers
- PyTorch
- Vision Transformer (ViT)
- Large Language Models research
---
⭐ If you find this project useful, consider giving it a star on GitHub or liking it on Hugging Face. |