WasteAssistant-InstructBLIP-LLaVA

WasteAssistant-InstructBLIP-LLaVA is an InstructBLIP-based visual question answering checkpoint fine-tuned for image-grounded waste identification and waste-management questions framed around Indian waste-management practices.

In this training variant, LLaVA (7B) was used to generate candidate/reference answers for the training pipeline. LLaVA is the answer-generation model; the model served from this repository is the fine-tuned InstructBLIP checkpoint.

• Base model: Salesforce/instructblip-flan-t5-xl
• Dataset: KhushKataruka/WasteVQA
• Research paper: arXiv:2607.10610
• Project repository: WasteAssistant on GitHub

Overview

The checkpoint is intended for research and prototyping of visual question answering (VQA) for waste items. Given an image and a natural-language question, it generates an answer based on visible visual information and learned associations from fine-tuning.

It is not a general-purpose legal authority or a substitute for local waste-collection guidance. Treat generated answers as model predictions and verify them before acting on them.

Question types represented in the evaluation split

The supplied evaluation split contains 2,784 image-question-answer records, 2,070 unique image paths, 15 question templates, and 21 waste labels. The categories and example templates below are based on the supplied test(1).json split; they describe evaluation coverage, not a guarantee of accuracy on each task.

1. Waste identification and material properties

• “What specific type of waste item do you see in this image?”
• “What materials is this waste object primarily made of?”
• “What visible characteristics like labels, shape, texture, color, and size help identify this item?”

2. Visible contamination

• “Are there any visible signs of contamination such as food residue, liquids, stains, or dirt on this object?”

3. Waste categories and SWM classification

• “Which main SWM 2016 waste category does this item belong to - Dry Waste, Wet Waste, Sanitary Waste, or Domestic Hazardous Waste?”
• “How would you classify this item as biodegradable, non-biodegradable, or hazardous according to SWM Rules 2016?”
• “Does this item fall under any special sub-categories like E-waste, Composite Waste, or Inert Waste and why?”

4. Preparation and special handling

• “What preprocessing steps like cleaning, drying, or wrapping are required before disposing of this item?”
• “Are there any special handling procedures needed for this item including separate collection methods?”

5. Recycling and compostability

• “How recyclable is this item through standard municipal recycling systems in India?”
• “What is the compostability potential of this item at household level and what is your reasoning?”

6. Bin selection

• “What colored bin should this waste be disposed in according to SWM Rules and what is the reasoning?”

7. Hazard level and environmental impact

• “On a scale of 1 to 10, what is the hazard level of this item and what hazard category does it belong to?”
• “What environmental impacts would occur if this item were disposed of improperly?”

8. Decomposition estimates

• “How long would this item take to decompose in a typical Indian landfill and what factors affect this?”

Waste labels in the supplied split

Cans, CardBoard, Chips Packets, CofeeCup, Construction scrap, Electrical Cables, Electronic chips, Electronics, Empty Medicine Packets, Glass, Gloves, Mask, Medicines, Metal, Paper, Plastic Bottles, Plastic Materials, Polythene, Syringes, WetWaste, Wire

These are annotation labels in the supplied split. Their presence does not mean that every class is equally represented or that the model performs equally well for each class.

Training-answer generation

The LLaVA variant used LLaVA (7B) to generate candidate answers for the training pipeline. Generated answers should be treated as model-generated annotations rather than unquestionable ground truth. Review and filter annotations before using them for training, benchmarking, or public-facing claims.

The answer-generation model (LLaVA) and the model checkpoint hosted in this repository (InstructBLIP) are different components of the pipeline.

Installation

Install a recent compatible version of PyTorch and Transformers:

pip install torch transformers pillow accelerate sentencepiece

For GPU inference, install the PyTorch build appropriate for your CUDA environment. The checkpoint may require substantial GPU memory; CPU inference can be slow.

Inference example

This example assumes that the repository contains the fine-tuned InstructBLIP weights and all required processor/tokenizer configuration files.

import torch
from PIL import Image
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration

model_id = "KhushKataruka/WasteAssistant-InstructBLIP-LLaVA"

processor = InstructBlipProcessor.from_pretrained(model_id)
model = InstructBlipForConditionalGeneration.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

image = Image.open("waste_image.jpg").convert("RGB")
question = "What specific type of waste item do you see in this image?"

inputs = processor(images=image, text=question, return_tensors="pt")
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=80)

answer = processor.batch_decode(output, skip_special_tokens=True)[0].strip()
print(answer)

If loading fails, confirm that the repository includes the checkpoint shards, shard index if present, config.json, processor configuration, tokenizer files, and any model-specific files created during training. Keep the model configuration consistent with the architecture used to save the checkpoint.

Intended use

• Research on visual question answering for waste identification and segregation.
• Prototyping image-grounded explanations about material properties, waste categories, recycling, compostability, and handling.
• Comparative evaluation of BLIP and InstructBLIP model variants using the same question templates and evaluation split.

Limitations and responsible use

• The question list describes coverage in the supplied evaluation split; it does not prove the model answers every question type accurately.
• Model-generated reference answers may be incomplete, inconsistent, or incorrect. Evaluate against a reviewed test set before reporting metrics.
• The model may hallucinate materials or properties that cannot be established from the image alone.
• Visible appearance cannot confirm hidden contamination, chemical composition, infection risk, or whether an item is safe to handle.
• Bin colours, collection arrangements, and legal requirements can depend on applicable rules and local authorities. Do not treat model outputs as authoritative legal guidance.
• Hazard ratings and decomposition estimates are approximate and may not be inferable from an image alone.
• Do not rely on this model alone for medical waste, sharps, hazardous substances, or other safety-critical disposal decisions.

Citation

If you use this checkpoint, cite the associated research paper and dataset. Verify the paper's official title and author metadata before publication.

@article{wasteassistant2026,
  title = {WasteAssistant},
  journal = {arXiv preprint arXiv:2607.10610},
  year = {2026},
  url = {https://arxiv.org/abs/2607.10610}
}

License

This repository declares the MIT license. Before public release, confirm that the checkpoint, training code, and underlying image/annotation sources can legally be redistributed under the selected license. Source datasets and base models may have separate terms.