| --- |
| license: mit |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: image-to-text |
| tags: |
| - image-captioning |
| - show-attend-and-tell |
| - visual-attention |
| datasets: |
| - flickr8k |
| metrics: |
| - bleu |
| - cider |
| --- |
| |
| # capit-sat |
|
|
| Show, Attend and Tell image captioner, trained from scratch on Flickr8k (Karpathy split). |
| The glass-box half of [capit](https://github.com/Bukunmi2108/capit) — exposes per-word |
| attention, beam candidates, and word-by-word playback. |
|
|
| ## Test-set scores (pycocoevalcap, Karpathy test = 1000 images) |
|
|
| | beam | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | CIDEr | |
| |-----:|-------:|-------:|-------:|-------:|------:| |
| | 1 | 61.99 | 44.37 | 30.23 | 20.05 | 55.51 | |
| | 3 | 64.77 | 47.34 | 33.68 | 23.45 | 62.20 | |
| | 5 | 65.54 | 47.84 | 34.08 | 23.63 | 62.80 | |
|
|
| ## Training |
|
|
| - Backbone: frozen ResNet-50 (ImageNet). Decoder trained from scratch. |
| - Best val BLEU-4 19.62 at epoch 7 (early-stopped); Colab T4. |
| - Splits: train 6000, val 1000, test 1000. |
|
|
| ## Known limitation |
|
|
| Attention is effectively 7x7: ResNet-50 at 224px is natively 7x7 and the encoder upsamples |
| to 14x14, so heatmaps are coarse (~32px blocks). Captions are grounded; the spots are |
| region-level, not pixel-level. |
|
|
| ## Use |
|
|
| `huggingface_hub.hf_hub_download("Bukunmi2108/capit-sat", "capit-sat.pt")` + `vocab.json`, then |
| `capit.serving.load_artifact(...)`. |
|
|
