README.md

---
language: km
license: mit
tags:
- image-to-text
- image-captioning
- khmer
- pytorch
- show-attend-and-tell
datasets:
- phonsobon/khmer_images_captioning_v2
---

# Khmer Image Captioning (Show, Attend and Tell)

A from-scratch image captioning model for **Khmer**, following the classic *Show, Attend and Tell*
architecture:

- **Encoder**: ResNet-101 pretrained on ImageNet, used only as a **frozen feature extractor**
  (no fine-tuning) producing a 14x14 grid of 2048-dim region features.
- **Decoder**: an LSTM with Bahdanau (additive) attention, embedding table, and gating layers,
  **trained from scratch** on Khmer captions.
- **Tokenization**: word-level, using `khmer-nltk` for Khmer word segmentation.

Trained on [`phonsobon/khmer_images_captioning_v2`](https://huggingface.co/datasets/phonsobon/khmer_images_captioning_v2).

This is **not** a standard `transformers` model — it uses custom PyTorch code (`modeling_khmer_captioning.py`
in this repo) rather than `AutoModel`.

## Files

- `config.json` — architecture hyperparameters
- `decoder.pt` — trained decoder weights (`state_dict`)
- `vocab.json` — word-level vocabulary (`itos` list)
- `modeling_khmer_captioning.py` — model classes + `load_model()` / `caption_image()` helpers

The encoder is **not** included — it's just off-the-shelf frozen ImageNet ResNet-101 weights,
downloaded automatically via `torchvision` when you load the model.

## Usage

```bash
pip install torch torchvision pillow huggingface_hub
```

```python
from huggingface_hub import snapshot_download
import sys

repo_dir = snapshot_download("phonsobon/khmer-images_captioning")
sys.path.insert(0, repo_dir)

from modeling_khmer_captioning import load_model, caption_image

encoder, decoder, itos, stoi, cfg = load_model(repo_dir)
caption = caption_image("your_image.jpg", encoder, decoder, itos, stoi, cfg, beam_size=3)
print(caption)
```

## Limitations

- Trained on a relatively small dataset (~3.7k image-caption pairs), so generalization outside
  that domain will be limited.
- Khmer text has no spaces between words; captions are segmented with `khmer-nltk`, whose
  probabilistic tokenizer won't always agree exactly with human segmentation, which affects
  BLEU-style evaluation more than it affects readability.