---
title: GPT-2 Stackformer Vision V2
emoji: 🖼️
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit
---

# GPT-2 + Vision (stackformer) — V2

A GPT-2 backbone rebuilt on the [`stackformer`](https://pypi.org/project/stackformer/)
library, with a frozen ViT-B/16 vision encoder and sparse cross-attention layers, trained
on Flickr8k. Supports two modes:

- **Text → Text**: plain GPT-2-style continuation, no image.
- **Image → Text**: upload an image to get a caption (optionally steer it with a prompt
  prefix).

Model weights: [`gurumurthy3/gpt2-stackformer-vision_V2`](https://huggingface.co/gurumurthy3/gpt2-stackformer-vision_V2)

## Notes

- The GPT-2 backbone was kept frozen during training; only the vision projector, Perceiver
  resampler, and 3 sparse cross-attention layers were fine-tuned (~16.6M params) on
  Flickr8k for a few epochs on a single T4 GPU. Expect short, simple captions rather than
  long fluent prose.
- `app.py` includes two patches for known bugs in `stackformer`'s attention implementation
  (inverted causal masking, and attention dropout not respecting `.eval()`) — both are
  required for correct inference, not just training, and are applied automatically at
  startup.
- If you change the model repo, set the `MODEL_REPO_ID` environment variable in the
  Space's settings rather than editing `app.py`.