license: apache-2.0 language:

bn tags:

bangla

vqa

llm

multimodal

evaluation pipeline_tag: text-generation

πŸ‡§πŸ‡© Evan Bangla VQA

Bangla Visual Question Answering using LLMs (Qwen vs TituLLM)


πŸ“Œ Overview

Evan Bangla VQA is a Bangla Visual Question Answering (VQA) system built using Large Language Models (LLMs). This project provides a complete, reproducible pipeline for answering Bangla visual questions and benchmarking two strong models:

  • Qwen (Qwen3.5-9B)
  • TituLLM (hishab/titulm-llama-3.2-3b-v2.0)

The repository includes:

βœ” Bangla prompt engineering βœ” optimized inference pipeline βœ” evaluation framework βœ” comparison analysis βœ” Hugging Face demo integration

This work is designed for research reproducibility, benchmarking, and publication.


πŸš€ Features

  • Bangla-first VQA prompting
  • Multi-model inference (Qwen + TituLLM)
  • Lightweight and reproducible pipeline
  • Hugging Face demo support
  • Paper-ready evaluation workflow
  • Semantic evaluation using BERT embeddings

πŸ“‚ Models Used

πŸ”Ή Qwen (Primary Model)

  • Model: Qwen/Qwen3.5-9B
  • Strong semantic reasoning
  • Lower hallucination rate

πŸ”Ή TituLLM (Baseline Model)

  • Model: hishab/titulm-llama-3.2-3b-v2.0
  • Lightweight baseline
  • Used for full evaluation comparison

βš™οΈ Installation

Clone the repository:

git clone https://huggingface.co/evanmahmud/evan-bangla-vqa
cd evan-bangla-vqa

Install dependencies:

pip install -r requirements.txt

🧠 Usage

πŸ”Ή Inference Example

from inference import answer

print(answer("ছবিঀে কী দেখা ΰ¦―ΰ¦Ύΰ¦šΰ§ΰ¦›ΰ§‡?", "Qwen"))
print(answer("ছবিঀে কী দেখা ΰ¦―ΰ¦Ύΰ¦šΰ§ΰ¦›ΰ§‡?", "TituLLM"))

πŸ“ Prompt Design

The system uses a Bangla-optimized prompt:

ΰ¦ͺ্রঢ্ন: {question}

সংক্ষিΰ¦ͺ্ঀ উঀ্ঀর দিনΰ₯€

উঀ্ঀর:

This improves:

βœ” response clarity βœ” brevity βœ” Bangla alignment


πŸ“Š Evaluation Metrics

The system is evaluated using:

  • Relaxed Accuracy
  • BLEU
  • ROUGE-L
  • BERTScore (Sentence-BERT cosine similarity)
  • Hallucination Rate

Hallucination Definition

Hallucination is computed using relaxed semantic matching:

Predictions are considered correct if:

βœ” substring overlap exists βœ” token overlap exists


πŸ“ˆ Results (Example)

Model Relaxed Accuracy BLEU ROUGE-L BERTScore
TituLLM 0.296 0.0176 0.000 0.572
Qwen Higher Higher Higher Higher

πŸ‘‰ Qwen shows stronger semantic alignment and lower hallucination.


πŸ§ͺ Demo (Hugging Face Space)

You can try the interactive demo:

πŸ‘‰ https://huggingface.co/spaces/YOURNAME/evan-bangla-vqa-demo


πŸ“Œ Reproducibility

This repository supports:

βœ” Kaggle GPU workflows βœ” quantized inference βœ” lightweight execution


πŸ“„ Citation

If you use this work:

@INPROCEEDINGS{11771448,
  author={Hoque, Md Mahmudul and Azrof, H.M. and Karmaker, Shuvo and Rafi, Redwan Ahmed and Khan, Sakib and Jyoti, Cornea},
  booktitle={2026 IEEE 4th International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings)}, 
  title={BanglaVQA-LLM: Enhancing Bangla Visual Question Answering with Instruction-Tuned Large Language Models}, 
  year={2026},
  volume={},
  number={},
  pages={1-5},
  keywords={Modeling;Visualization;Printing;Question answering (information retrieval);Multilingual;Testing;Training;Large language models;Tuning;Indexes;Bangla visual question answering;large language models;retrieval-augmented prompting;CLIP;low-resource languages;hallucination},
  doi={10.1109/AIBThings71029.2026.11771448}}

🀝 Contributions

Contributions are welcome:

βœ” improved Bangla datasets βœ” stronger evaluation methods βœ” optimized pipelines


⭐ Support

If you find this helpful, please ⭐ the repo.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for evanmahmud/evan-bangla-vqa

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1,448)
this model