|
|
--- |
|
|
language: en |
|
|
license: other |
|
|
datasets: |
|
|
- DeepMount00/ner_training |
|
|
tags: |
|
|
- vision |
|
|
- multimodal |
|
|
- OCR |
|
|
- SmolVLM |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# SmolVLM Base - OCR Fine-tuned |
|
|
|
|
|
This is a merged version of SmolVLM-Base fine-tuned for OCR tasks. The model was trained using QLoRA on the DeepMount00/ner_training dataset. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: HuggingFaceTB/SmolVLM-Base |
|
|
- **Task**: Optical Character Recognition (OCR) |
|
|
- **Training Method**: QLoRA with 4-bit quantization |
|
|
- **Target Modules**: down_proj, o_proj, k_proj, q_proj, gate_proj, up_proj, v_proj |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoProcessor, Idefics3ForConditionalGeneration |
|
|
import torch |
|
|
from PIL import Image |
|
|
|
|
|
model_id = "DeepMount00/SmolVLM-Base-ocr_base" |
|
|
processor = AutoProcessor.from_pretrained(model_id) |
|
|
model = Idefics3ForConditionalGeneration.from_pretrained(model_id) |
|
|
|
|
|
# Load your image |
|
|
image = Image.open("path_to_your_image.jpg").convert("RGB") |
|
|
|
|
|
# Prepare the prompt |
|
|
messages = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "text", "text": "You are a model specialized in OCR"}, |
|
|
{"type": "image"}, |
|
|
{"type": "text", "text": "Extract the text from this image"} |
|
|
] |
|
|
} |
|
|
] |
|
|
|
|
|
# Process inputs |
|
|
inputs = processor(text=messages, images=[image], return_tensors="pt") |
|
|
|
|
|
# Generate |
|
|
with torch.no_grad(): |
|
|
outputs = model.generate(**inputs, max_new_tokens=512) |
|
|
|
|
|
# Decode and print the response |
|
|
print(processor.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
