| --- |
| license: apache-2.0 |
| library_name: peft |
| tags: |
| - llama-3.2-vision |
| - plant-disease |
| - lora |
| - rocm |
| - mi300x |
| - amd |
| - vlm |
| - image-text-to-text |
| base_model: meta-llama/Llama-3.2-11B-Vision-Instruct |
| --- |
| |
| # Llama 3.2 Vision 11B LoRA — Plant Disease Diagnosis (MI300X fine-tune) |
|
|
| Fine-tuned **Llama 3.2 Vision 11B** with **LoRA** on a plant disease QA dataset |
| (cashew, cassava, maize, tomato — 22 disease classes) for visual diagnosis and |
| treatment recommendations. |
|
|
| Trained on a single **AMD Instinct MI300X** using PyTorch + ROCm, as a submission to the |
| lablab.ai AMD hackathon **Track 3 — Building AI-Powered Applications on AMD GPUs**. |
|
|
| ## 🌱 Try the live demo |
|
|
| This adapter powers the conversational layer of an interactive Plant Disease Assistant — upload a leaf photo to get a diagnosis and treatment guide: |
|
|
| **👉 [huggingface.co/spaces/lablab-ai-amd-developer-hackathon/merolav-space](https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/merolav-space)** |
|
|
| (The hosted Space runs the lighter DINOv2-L classifier on free CPU. Load this LoRA adapter on a GPU machine for the full conversational VLM experience — see the code under [`vision/`](https://github.com/genyarko/amd-merolav/tree/main/vision) in the training repo.) |
|
|
| ## Results |
|
|
| | Epoch | Train Loss | Val Loss | Throughput | Wall Time | |
| |------:|-----------:|---------:|:----------:|:---------:| |
| | 1 | 0.9530 | 0.0244 | 1.3 | 3331.1s | |
| | 2 | 0.0203 | 0.0177 | 1.3 | 3318.3s | |
| | 3 | 0.0151 | 0.0151 | 1.3 | 3321.8s | |
| | 4 | 0.0125 | 0.0147 | 1.3 | 3318.8s | |
| | 5 | 0.0109 | 0.0146 | 1.3 | 3317.5s | |
|
|
| **Best val_loss: 0.0146** |
| |
| ### Epoch 3 adapter |
| |
| An `epoch3_adapter/` checkpoint is included for A/B comparison. Epoch 3 had val_loss=0.0151 vs epoch 5's 0.0146 — the difference is marginal and epoch 3 may generalize equally well in practice. |
| |
| ## Training Details |
| |
| - **Base model:** meta-llama/Llama-3.2-11B-Vision-Instruct (11B params, ~4B vision + ~7B language) |
| - **Method:** LoRA (rank=16, alpha=32, dropout=0.05) |
| - **Target modules:** q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj |
| - **Precision:** bf16 (native MI300X) |
| - **Epochs:** 5 |
| - **Effective batch size:** 16 |
| - **Learning rate:** 2e-05 with cosine decay + 0.1 warmup |
| - **Optimizer:** AdamW (weight_decay=0.01) |
| - **Max sequence length:** 2048 |
| - **Hardware:** 1x AMD Instinct MI300X (192 GB HBM3) |
| |
| ## Usage |
| |
| ```python |
| from peft import PeftModel |
| from transformers import AutoProcessor, MllamaForConditionalGeneration |
| |
| base = MllamaForConditionalGeneration.from_pretrained( |
| "meta-llama/Llama-3.2-11B-Vision-Instruct", |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| model = PeftModel.from_pretrained(base, "best_adapter") |
| processor = AutoProcessor.from_pretrained("best_adapter") |
| ``` |
| |
| ## Artifacts |
|
|
| - `best_adapter/` — LoRA weights from the best validation epoch |
| - `epoch3_adapter/` — LoRA weights from epoch 3 (for A/B comparison) |
| - `config.yaml` — training hyperparameters |
| - `metrics.json` — per-epoch training history |
|
|
| See `config.yaml` for the full hyperparameter set. |
|
|
| ## Source |
|
|
| Training code: <https://github.com/genyarko/amd-merolav/tree/main/vision> |
|
|
