File size: 3,019 Bytes
71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 c018a30 71574c6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 | # BLIP Image Captioning LoRA
A LoRA fine tuned version of **Salesforce/blip-image-captioning-base** for image caption generation using the **Flickr8k** dataset. This project demonstrates parameter efficient fine tuning with PEFT while keeping the base model frozen.
## Model Details
* **Model:** `ciphermosaic/blip-image-captioning`
* **Task:** Image Captioning
* **Base Model:** `Salesforce/blip-image-captioning-base`
* **Fine Tuning Method:** LoRA (PEFT)
* **License:** Apache-2.0
## Dataset
The model was fine tuned on the **jxie/flickr8k** dataset.
Dataset split:
| Split | Samples |
| ---------- | ------: |
| Train | 6,000 |
| Validation | 1,000 |
| Test | 1,000 |
## Training Configuration
* Framework: PyTorch
* Transformers
* PEFT (LoRA)
* Hardware: NVIDIA Tesla T4 (Google Colab)
* Epochs: 2
* Batch Size: 4
* Learning Rate: `5e-5`
* Optimizer: AdamW
### LoRA Configuration
```python
LoraConfig(
r=8,
lora_alpha=16,
lora_dropout=0.1,
target_modules=[
"qkv",
"projection"
],
bias="none"
)
```
## Evaluation
| Metric | Value |
| --------------- | ---------: |
| Validation Loss | **8.0478** |
This checkpoint is intended as an educational fine tuning project and baseline implementation for BLIP image captioning with LoRA.
## Usage
### Load with Transformers and PEFT
```python
from transformers import BlipProcessor, BlipForConditionalGeneration
from peft import PeftModel
base_model = BlipForConditionalGeneration.from_pretrained(
"Salesforce/blip-image-captioning-base"
)
model = PeftModel.from_pretrained(
base_model,
"ciphermosaic/blip-image-captioning"
)
processor = BlipProcessor.from_pretrained(
"Salesforce/blip-image-captioning-base"
)
```
### Example Inference
```python
from PIL import Image
import requests
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
inputs = processor(images=image, return_tensors="pt")
outputs = model.generate(**inputs)
caption = processor.decode(outputs[0], skip_special_tokens=True)
print(caption)
```
The model can also be loaded using standard Hugging Face Transformers workflows together with the PEFT adapter.
## Limitations
* Trained only on Flickr8k, therefore generalization to unseen domains may be limited.
* Caption quality depends on image content and may not accurately describe complex scenes.
* Primarily produces English captions.
* Performance may vary for medical, satellite, or highly specialized imagery.
## Intended Uses
* Image caption generation
* Vision Language Model experimentation
* Learning PEFT and LoRA fine tuning
* Educational and research purposes
* Baseline for further image captioning improvements
## Acknowledgements
This work builds upon:
* Salesforce BLIP
* Hugging Face Transformers
* Hugging Face PEFT
* Flickr8k Dataset
Special thanks to the open source community for making these resources available.
## Author
**CipherMosaic**
GitHub: https://github.com/ciphermosaic
|