README.md

---
library_name: peft
base_model: microsoft/Florence-2-base-ft
tags:
  - florence-2
  - lora
  - object-detection
  - subplot-detection
license: mit
---

# plot-visualization-florence-2-lora-32

This is a LoRA fine-tuned version of [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft) for detecting subplots in scientific figures.

## Model Details

- **Base Model:** microsoft/Florence-2-base-ft
- **LoRA Rank:** 32
- **Task:** Object Detection (subplot detection)
- **Labels:** `quantitative plot`, `qualitative plot`

## Usage

```python
from pathlib import Path
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoProcessor
import torch

# Load model
base_model = "microsoft/Florence-2-base-ft"
processor = AutoProcessor.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(model, "amayuelas/plot-visualization-florence-2-lora-32")
model = model.merge_and_unload()
model = model.to("cuda")

# Run inference
image = Image.open("your_image.jpg").convert("RGB")
prompt = "<OD>"
inputs = processor(text=prompt, images=image, return_tensors="pt").to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.float16)

with torch.no_grad():
    generated_ids = model.generate(
        input_ids=inputs["input_ids"],
        pixel_values=inputs["pixel_values"],
        max_new_tokens=1024,
        num_beams=3,
        use_cache=False,
    )

output = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
print(output)
```

## Training

This model was fine-tuned using PEFT/LoRA on a dataset of scientific figures with annotated subplots.