phonsobon/Images_captioning_fine_tune_Florence-2-base

Florence-2-base fine-tuned for Khmer Image Captioning

This model is a fine-tuned version of microsoft/Florence-2-base on the phonsobon/khmer_images_captioning_v2 dataset. It generates image captions in the Khmer language.

Model description

Florence-2 is a unified vision-language model from Microsoft that handles a range of vision tasks (captioning, detection, segmentation, OCR, etc.) using task-specific prompts. This model was fine-tuned specifically on the <MORE_DETAILED_CAPTION> task so that, given an image, it produces a detailed descriptive caption written in Khmer.

• Base model: microsoft/Florence-2-base
• Fine-tuning method: LoRA (via peft)
• Task prompt used: <MORE_DETAILED_CAPTION>
• Language: Khmer (km)

Intended uses & limitations

This model is intended for generating Khmer-language descriptions of general-scene photographs (landscapes, people, temples, markets, food, etc.), similar to the training distribution described below. It is not intended for:

• Safety-critical decision making
• Captioning of specialized domains not represented in the training data (e.g. medical imagery, satellite imagery)
• Tasks requiring perfect factual accuracy (like any captioning model, it can hallucinate details)

Since Florence-2's original tokenizer was not heavily pretrained on Khmer script, caption quality and fluency may vary; feedback and further fine-tuning on more data is welcome.

Training data

The model was fine-tuned on phonsobon/khmer_images_captioning_v2:

Split	Rows
train	3,690
val	977
test	978

Each example consists of an image paired with a Khmer-language caption (caption length: 57–384 characters).

Training procedure

• Base model: microsoft/Florence-2-base
• Fine-tuning method: LoRA (rank 8, alpha 16, dropout 0.05) applied to attention projection layers
• Task prompt: <MORE_DETAILED_CAPTION>
• Epochs: 3
• Effective batch size: 16 (batch size 4 × gradient accumulation 4)
• Learning rate: 2e-4 (LoRA)
• LR schedule: linear warmup (3%) + linear decay
• Max target length: 1024 tokens
• Precision: bf16 autocast
• Hardware: Kaggle GPU (T4 / P100)

How to use

import torch
from transformers import AutoProcessor, AutoModelForCausalLM
from peft import PeftModel
from PIL import Image

base_model_id = "microsoft/Florence-2-base"
adapter_id = "phonsobon/Images_captioning_fine_tune_Florence-2-base"

processor = AutoProcessor.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True, torch_dtype=torch.float32)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

image = Image.open("your_image.jpg").convert("RGB")
task_prompt = "<MORE_DETAILED_CAPTION>"

inputs = processor(text=task_prompt, images=image, return_tensors="pt")
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    num_beams=3,
    do_sample=False,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
caption = processor.post_process_generation(
    generated_text, task=task_prompt, image_size=(image.width, image.height)
)[task_prompt]

print(caption)

Evaluation

Citation

If you use this model, please cite the base Florence-2 model and this dataset:

@article{xiao2023florence2,
  title={Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}