README.md

---
title: GPT2VL Stackformer V2
colorFrom: purple
colorTo: indigo
sdk: gradio
sdk_version: 5.9.1
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: A lightweight vision-language model for image captioning
---

# GPT2VL — Stackformer V2

A lightweight vision-language model for image captioning. Combines a frozen **GPT-2 Small** backbone with a **ViT-B/16** vision encoder, a **Perceiver-style** visual resampler, and **gated sparse cross-attention** (spliced before layers 3, 6, and 9). Trained on `Trickxter/COCO2017-captions`.

## Overview

- **Image-to-Text Captioning** — Upload or paste an image to generate descriptive captions.
- **Generation Parameters** — Fine-tune sampling via Temperature, Top-K, Top-P (Nucleus), and Max New Tokens controls.
- **Real-Time Model Metrics** — Evaluates per-step confidence, generation throughput, token counts, and GPU inference latency.
- **Parameter Efficiency** — Only the resampler and gated cross-attention blocks (~few million parameters) were fine-tuned while GPT-2 and ViT-B/16 backbones remain frozen.

## Model Weights

Default model repository: [`gurumurthy3/gpt2vl-stackformer-v2`](https://huggingface.co/gurumurthy3/gpt2vl-stackformer-v2)

Set the `MODEL_REPO_ID` secret or environment variable to point to your own Hugging Face model repository if forked.

## Deployment & Setup Notes

- **Hardware & ZeroGPU:** Runs on Hugging Face ZeroGPU (dynamic GPU allocation for inference calls using `@spaces.GPU`). Model operates with FP32-internal stability math and PyTorch AMP autocast for high performance and numerical stability.
- **Dependencies & Stackformer:** Uses `stackformer` modules integrated directly for cross-attention and resampler execution.
- **Checkpoint Layout:** Expects `config.json` and `model_trainable.safetensors` (adapter weights only) in the Hugging Face Hub repository. Backbone weights for GPT-2 and ViT-B/16 are loaded dynamically from public weights at startup.