w4ashabii/SmolVLM256M_CropDisease

SmolVLM-256M — Nepali Crop Disease

A LoRA fine-tune of HuggingFaceTB/SmolVLM-256M-Instruct that answers "What crop is shown in this image, and does it have any disease?" for photos of Nepali crops. Trained on w4ashabii/nepali_crop_data, a ShareGPT-format visual-QA dataset covering 9 crops.

Model Details

Base model	HuggingFaceTB/SmolVLM-256M-Instruct (~256M params)
Fine-tuning method	LoRA (adapters only, base weights frozen)
Trainable params	3,769,344 / 260,254,272 (1.45%)
Framework	🤗 transformers Trainer + Unsloth FastVisionModel
LoRA targets	Attention + MLP modules across both vision encoder and language model
LoRA config	r=8, alpha=16, dropout=0.05, bias="none"
Precision	fp16
Hardware	1x NVIDIA T4 (Google Colab)
Language(s)	English (Q&A text); images of Nepali-grown crops
License	Apache 2.0 (inherited from base model)

Training Data

• Dataset: w4ashabii/nepali_crop_data
• Format: ShareGPT-style conversations (sharegpt.jsonl), one user turn (image + question) → one assistant turn (crop + disease answer)
• Train examples used: 39,987
• Eval examples used: 500 (held-out validation split)
• Image preprocessing: resized to longest_edge=512, image splitting disabled (not needed for single-leaf/crop photos)

Training Procedure

Hyperparameter	Value
Epochs	2
Per-device batch size	8
Gradient accumulation steps	2
Effective batch size	16
Learning rate	1e-4
LR schedule	cosine, 3% warmup
Optimizer	AdamW (Trainer default)
Total steps	5,000
Total training time	~3h 3m (11,012s)
Throughput	7.26 samples/sec

Checkpoints were saved locally every 500 steps and mirrored to this Hub repo roughly every hour during training via a custom TrainerCallback (each upload replaced the repo's prior contents), with a final clean upload of just the adapter + processor after training completed.

Training / Validation Loss

Step	Training Loss	Validation Loss
500	0.1948	0.1918
1000	0.1842	0.1871
1500	0.1878	0.1858
2000	0.1827	0.1865
2500	0.1685	0.1856
3000	0.1809	0.1841
3500	0.1825	0.1837
4000	0.1730	0.1836
4500	0.1758	0.1833
5000	0.1860	0.1833

Final train loss: 0.2363 (mean over all logged steps, includes early-training values before it stabilized). Validation loss plateaued around 0.183 from step ~3500 onward, suggesting the model converged with room left in the schedule rather than overfitting.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

BASE_MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
ADAPTER_REPO = "w4ashabii/SmolVLM256M_CropDisease"

processor = AutoProcessor.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForVision2Seq.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).to("cuda")
model.eval()

image = Image.open("your_crop_photo.jpg").convert("RGB")
question = "What crop is shown in this image, and does it have any disease? If so, name the disease."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=64)

answer = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer.strip())

With Unsloth (faster inference load)

from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained("w4ashabii/SmolVLM256M_CropDisease")
FastVisionModel.for_inference(model)
# ... same generate() call as above

Intended Use

• Assistive identification of crop type and visible disease symptoms from photos, for agricultural extension, research, or educational tooling focused on Nepali-grown crops.
• Not intended as a sole basis for treatment or pesticide decisions — outputs should be verified by an agronomist or local agricultural extension service, especially for high-stakes crop management decisions.

Limitations

• Small base model (256M params) and LoRA-only tuning trade some accuracy for speed/size; expect a lighter-weight, less nuanced answer than larger VLMs.
• Loss computed over the full sequence (question + answer) rather than answer-only, so the model was also lightly trained to reproduce the (fixed) question text — this doesn't appear to have hurt convergence here but is a simplification versus masked/answer-only supervision.
• Coverage limited to the crops and disease classes present in w4ashabii/nepali_crop_data; performance on out-of-distribution crops, lighting conditions, or camera angles is untested.
• Not evaluated for robustness to adversarial or low-quality images (blur, occlusion, multiple crops in frame, etc.).

Training Framework

Fine-tuned with 🤗 transformers.Trainer and LoRA adapters loaded/attached via Unsloth's FastVisionModel (use_gradient_checkpointing="unsloth") for reduced VRAM use and faster steps on a single T4 GPU.