README.md

metadata

title: GPT-2 Stackformer Vision V2
emoji: 🖼️
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit

GPT-2 + Vision (stackformer) — V2

A GPT-2 backbone rebuilt on the stackformer library, with a frozen ViT-B/16 vision encoder and sparse cross-attention layers, trained on Flickr8k. Supports two modes:

• Text → Text: plain GPT-2-style continuation, no image.
• Image → Text: upload an image to get a caption (optionally steer it with a prompt prefix).

Model weights: gurumurthy3/gpt2-stackformer-vision_V2

Notes

• The GPT-2 backbone was kept frozen during training; only the vision projector, Perceiver resampler, and 3 sparse cross-attention layers were fine-tuned (~16.6M params) on Flickr8k for a few epochs on a single T4 GPU. Expect short, simple captions rather than long fluent prose.
• app.py includes two patches for known bugs in stackformer's attention implementation (inverted causal masking, and attention dropout not respecting .eval()) — both are required for correct inference, not just training, and are applied automatically at startup.
• If you change the model repo, set the MODEL_REPO_ID environment variable in the Space's settings rather than editing app.py.