| --- |
| title: GPT-2 Stackformer Vision V2 |
| emoji: 🖼️ |
| colorFrom: indigo |
| colorTo: pink |
| sdk: gradio |
| sdk_version: 5.49.1 |
| app_file: app.py |
| pinned: false |
| license: mit |
| --- |
| |
| # GPT-2 + Vision (stackformer) — V2 |
|
|
| A GPT-2 backbone rebuilt on the [`stackformer`](https://pypi.org/project/stackformer/) |
| library, with a frozen ViT-B/16 vision encoder and sparse cross-attention layers, trained |
| on Flickr8k. Supports two modes: |
|
|
| - **Text → Text**: plain GPT-2-style continuation, no image. |
| - **Image → Text**: upload an image to get a caption (optionally steer it with a prompt |
| prefix). |
|
|
| Model weights: [`gurumurthy3/gpt2-stackformer-vision_V2`](https://huggingface.co/gurumurthy3/gpt2-stackformer-vision_V2) |
|
|
| ## Notes |
|
|
| - The GPT-2 backbone was kept frozen during training; only the vision projector, Perceiver |
| resampler, and 3 sparse cross-attention layers were fine-tuned (~16.6M params) on |
| Flickr8k for a few epochs on a single T4 GPU. Expect short, simple captions rather than |
| long fluent prose. |
| - `app.py` includes two patches for known bugs in `stackformer`'s attention implementation |
| (inverted causal masking, and attention dropout not respecting `.eval()`) — both are |
| required for correct inference, not just training, and are applied automatically at |
| startup. |
| - If you change the model repo, set the `MODEL_REPO_ID` environment variable in the |
| Space's settings rather than editing `app.py`. |
