File size: 2,637 Bytes
95e6215 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: mit
tags:
- peft
- lora
- qwen2
- multilingual
- ocr
- translation
- safetensors
base_model: Qwen/Qwen2-VL-7B-Instruct
---
# Babel
## Model Description
Babel is a Qwen2-VL LoRA adapter fine-tuned for multilingual OCR (Optical Character Recognition) and translation tasks. It can extract text from images across multiple languages and translate between them, making it ideal for document digitization, cross-language content processing, and international business automation.
## Model Architecture
- **Base Model**: `Qwen/Qwen2-VL-7B-Instruct`
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation) via PEFT
- **Checkpoint**: Final checkpoint
- **Task**: Multilingual OCR + Translation (Vision-Language)
## Training Details
- **Framework**: HuggingFace PEFT + Transformers
- **Dataset**: Multilingual document images with text annotations and translations
- **Languages**: Multiple languages supported including English, Hindi, and more
- **Approach**: Vision-language fine-tuning with OCR and translation objectives
## Files
| File | Description |
|------|-------------|
| `adapter_model.safetensors` | LoRA adapter weights |
| `adapter_config.json` | PEFT adapter configuration |
| `tokenizer.json` | Tokenizer vocabulary |
| `tokenizer_config.json` | Tokenizer configuration |
## Usage
```python
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from peft import PeftModel
from PIL import Image
from huggingface_hub import snapshot_download
# Download adapter
adapter_dir = snapshot_download(repo_id='devanshty/Babel')
# Load base model
base_model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-7B-Instruct",
torch_dtype="auto",
device_map="auto"
)
processor = AutoProcessor.from_pretrained(adapter_dir)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_dir)
model.eval()
# OCR + Translate
image = Image.open("document.jpg")
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "Extract all text from this image and translate it to English."}
]
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(output[0], skip_special_tokens=True))
```
## Download & Use
```python
from huggingface_hub import hf_hub_download
adapter = hf_hub_download(repo_id='devanshty/Babel', filename='adapter_model.safetensors')
```
|