| # BLIP Image Captioning LoRA |
|
|
| A LoRA fine tuned version of **Salesforce/blip-image-captioning-base** for image caption generation using the **Flickr8k** dataset. This project demonstrates parameter efficient fine tuning with PEFT while keeping the base model frozen. |
|
|
| ## Model Details |
|
|
| * **Model:** `ciphermosaic/blip-image-captioning` |
| * **Task:** Image Captioning |
| * **Base Model:** `Salesforce/blip-image-captioning-base` |
| * **Fine Tuning Method:** LoRA (PEFT) |
| * **License:** Apache-2.0 |
|
|
| ## Dataset |
|
|
| The model was fine tuned on the **jxie/flickr8k** dataset. |
|
|
| Dataset split: |
|
|
| | Split | Samples | |
| | ---------- | ------: | |
| | Train | 6,000 | |
| | Validation | 1,000 | |
| | Test | 1,000 | |
|
|
| ## Training Configuration |
|
|
| * Framework: PyTorch |
| * Transformers |
| * PEFT (LoRA) |
| * Hardware: NVIDIA Tesla T4 (Google Colab) |
| * Epochs: 2 |
| * Batch Size: 4 |
| * Learning Rate: `5e-5` |
| * Optimizer: AdamW |
|
|
| ### LoRA Configuration |
|
|
| ```python |
| LoraConfig( |
| r=8, |
| lora_alpha=16, |
| lora_dropout=0.1, |
| target_modules=[ |
| "qkv", |
| "projection" |
| ], |
| bias="none" |
| ) |
| ``` |
|
|
| ## Evaluation |
|
|
| | Metric | Value | |
| | --------------- | ---------: | |
| | Validation Loss | **8.0478** | |
|
|
| This checkpoint is intended as an educational fine tuning project and baseline implementation for BLIP image captioning with LoRA. |
|
|
| ## Usage |
|
|
| ### Load with Transformers and PEFT |
|
|
| ```python |
| from transformers import BlipProcessor, BlipForConditionalGeneration |
| from peft import PeftModel |
| |
| base_model = BlipForConditionalGeneration.from_pretrained( |
| "Salesforce/blip-image-captioning-base" |
| ) |
| |
| model = PeftModel.from_pretrained( |
| base_model, |
| "ciphermosaic/blip-image-captioning" |
| ) |
| |
| processor = BlipProcessor.from_pretrained( |
| "Salesforce/blip-image-captioning-base" |
| ) |
| ``` |
|
|
| ### Example Inference |
|
|
| ```python |
| from PIL import Image |
| import requests |
| |
| image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB") |
| |
| inputs = processor(images=image, return_tensors="pt") |
| |
| outputs = model.generate(**inputs) |
| |
| caption = processor.decode(outputs[0], skip_special_tokens=True) |
| |
| print(caption) |
| ``` |
|
|
| The model can also be loaded using standard Hugging Face Transformers workflows together with the PEFT adapter. |
|
|
| ## Limitations |
|
|
| * Trained only on Flickr8k, therefore generalization to unseen domains may be limited. |
| * Caption quality depends on image content and may not accurately describe complex scenes. |
| * Primarily produces English captions. |
| * Performance may vary for medical, satellite, or highly specialized imagery. |
|
|
| ## Intended Uses |
|
|
| * Image caption generation |
| * Vision Language Model experimentation |
| * Learning PEFT and LoRA fine tuning |
| * Educational and research purposes |
| * Baseline for further image captioning improvements |
|
|
| ## Acknowledgements |
|
|
| This work builds upon: |
|
|
| * Salesforce BLIP |
| * Hugging Face Transformers |
| * Hugging Face PEFT |
| * Flickr8k Dataset |
|
|
| Special thanks to the open source community for making these resources available. |
|
|
| ## Author |
|
|
| **CipherMosaic** |
|
|
| GitHub: https://github.com/ciphermosaic |
|
|
