README.md

metadata

title: GPT2VL Stackformer V2
colorFrom: purple
colorTo: indigo
sdk: gradio
sdk_version: 5.9.1
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: A lightweight vision-language model for image captioning

GPT2VL — Stackformer V2

A lightweight vision-language model for image captioning. Combines a frozen GPT-2 Small backbone with a ViT-B/16 vision encoder, a Perceiver-style visual resampler, and gated sparse cross-attention (spliced before layers 3, 6, and 9). Trained on Trickxter/COCO2017-captions.

Overview

• Image-to-Text Captioning — Upload or paste an image to generate descriptive captions.
• Generation Parameters — Fine-tune sampling via Temperature, Top-K, Top-P (Nucleus), and Max New Tokens controls.
• Real-Time Model Metrics — Evaluates per-step confidence, generation throughput, token counts, and GPU inference latency.
• Parameter Efficiency — Only the resampler and gated cross-attention blocks (~few million parameters) were fine-tuned while GPT-2 and ViT-B/16 backbones remain frozen.

Model Weights

Default model repository: gurumurthy3/gpt2vl-stackformer-v2

Set the MODEL_REPO_ID secret or environment variable to point to your own Hugging Face model repository if forked.

Deployment & Setup Notes

• Hardware & ZeroGPU: Runs on Hugging Face ZeroGPU (dynamic GPU allocation for inference calls using @spaces.GPU). Model operates with FP32-internal stability math and PyTorch AMP autocast for high performance and numerical stability.
• Dependencies & Stackformer: Uses stackformer modules integrated directly for cross-attention and resampler execution.
• Checkpoint Layout: Expects config.json and model_trainable.safetensors (adapter weights only) in the Hugging Face Hub repository. Backbone weights for GPT-2 and ViT-B/16 are loaded dynamically from public weights at startup.