README.md

---
tags:
- image-to-text
- vision-language
- gpt2
- stackformer
---

# GPT2VL (Stackformer V2)

Gated sparse cross-attention Vision-Language model: a frozen GPT-2 small backbone
with a Perceiver-style visual resampler and gated cross-attention blocks
(spliced before layers 3, 6, 9), trained on `Trickxter/COCO2017-captions`.

**Important:** `model_trainable.safetensors` contains ONLY the trained weights
(`resampler.*` and `cross_blocks.*`). It does NOT include the frozen GPT-2
backbone or ViT vision encoder weights. To use this model you need:
1. The base `gpt2` weights (loaded via `GPT2LMHeadModel.from_pretrained("gpt2")`)
2. The `stackformer` package / this repo's `GPT2VL` model definition
3. This checkpoint's `config.json` for architecture hyperparameters
4. `model_trainable.safetensors` loaded with `strict=False` on top of the assembled model