Gilfoyle-alised/BLIP-finetune-on-COCO

BLIP-LoRA Image Captioning Model

Fine-tuned BLIP for Image Captioning on COCO 2014

This repository contains a LoRA fine-tuned adapter for Salesforce/blip-image-captioning-base.

git hub repo: github

---

Model Overview

This model is based on BLIP Image Captioning Base and fine-tuned using LoRA.

Instead of storing the full BLIP model weights, this repository stores only the lightweight LoRA adapter weights. During inference, the base BLIP model is loaded first, and then this LoRA adapter is attached.

Item	Description
Base model	Salesforce/blip-image-captioning-base
Fine-tuning method	LoRA
Framework	PyTorch, Transformers, PEFT
Task	Image captioning
Dataset	COCO 2014 captions
Output language	English
Adapter file	adapter_model.safetensors

---

Repository Files

The repository should contain these files:

.
├── adapter_config.json
├── adapter_model.safetensors
├── preprocessor_config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── vocab.txt
├── training_state.pt
├── blip_lora_coco2014.ipynb
└── README.md

Important: make sure the files are named exactly like this:

adapter_config.json
adapter_model.safetensors
preprocessor_config.json
tokenizer.json
tokenizer_config.json
special_tokens_map.json
vocab.txt

Do not upload them with names like:

adapter_config(1).json
adapter_model(1).safetensors

because PEFT expects the standard file names.

---

LoRA Configuration

Setting	Value
PEFT type	LoRA
LoRA rank r	16
LoRA alpha	32
LoRA dropout	0.05
Bias	none
Target modules	query, key, value
Base model class	BlipForConditionalGeneration

The adapter targets the attention projection layers of BLIP, allowing the model to adapt to COCO-style image captions while keeping most of the base model frozen.

---

Image Preprocessing

The model uses the BLIP image processor.

Setting	Value
Image size	384 × 384
RGB conversion	Yes
Resize	Yes
Rescale	Yes
Normalize	Yes
Processor class	BlipProcessor

---

Training Details

Setting	Value
Dataset	COCO 2014 captions
Training images	5,000
Validation images	500
Evaluation images	200
Epochs	3
Batch size	16
Gradient accumulation	2
Learning rate	5e-4
Weight decay	0.01
Max caption length	40
Generation max length	30
Beam size	5
Mixed precision	fp16
Gradient checkpointing	Enabled

---

Evaluation Results

Evaluation was performed on 200 validation images using beam search.

Metric	Score
BLEU-1	0.7600
BLEU-2	0.6027
BLEU-3	0.4697
BLEU-4	0.3654
ROUGE-L	0.5800
CIDEr	1.3512
CLIPScore	0.7521
RefCLIPScore	0.8066
Best validation loss	2.2685

Example generated captions:

a bed with a book on top of it in front of a window
a police car parked on the side of the road
a man standing in front of a street sign
a baseball player swinging a bat at a ball

---

How to Use

Install the required libraries:

pip install transformers peft accelerate safetensors pillow torch

Then load the base BLIP model and attach the LoRA adapter:

import torch
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
from peft import PeftModel

repo_id = "your-username/your-model-repo"
base_model_id = "Salesforce/blip-image-captioning-base"

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained(repo_id)

base_model = BlipForConditionalGeneration.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, repo_id)
model = model.to(device)
model.eval()

Generate a caption:

image = Image.open("example.jpg").convert("RGB")

inputs = processor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        pixel_values=inputs["pixel_values"],
        max_length=30,
        num_beams=5
    )

caption = processor.decode(output[0], skip_special_tokens=True)
print(caption)

---

Optional: Merge LoRA Adapter

For deployment, you can merge the LoRA adapter into the base model:

merged_model = model.merge_and_unload()

After merging, the model becomes a standard BLIP model with the LoRA weights applied.

---

Intended Use

This model is intended for:

• Image caption generation
• Vision-language learning experiments
• COCO-style captioning tasks
• Educational fine-tuning demonstrations
• Lightweight BLIP adaptation using LoRA

---

Limitations

This model has several limitations:

• It is a LoRA adapter, not a full standalone model.
• The base model Salesforce/blip-image-captioning-base is required for inference.
• Caption quality depends on the COCO 2014 training subset.
• The model may generate generic captions for complex scenes.
• The model may hallucinate objects that are not visible in the image.
• The model may perform poorly on images outside the COCO-style distribution.
• The model is not designed for safety-critical use cases.

---

Out-of-Scope Use

This model is not intended for:

• Medical image interpretation
• Legal or forensic image analysis
• Safety-critical decision-making
• Surveillance or biometric identification
• Production use without further evaluation

---

Citation

@misc{blip_lora_coco2014_captioning,
  title = {BLIP-LoRA Image Captioning Model Fine-Tuned on COCO 2014},
  author = {Ali Sedghiye},
  year = {2026},
  note = {LoRA adapter for Salesforce/blip-image-captioning-base}
}

---

Author

Developed by Ali Sedghiye as a LoRA fine-tuned BLIP image captioning model.