MoME-BanglaFake: Mixture-of-Modality-Experts for Bengali Fake News Detection

---

Model Summary

MoME-BanglaFake is a trimodal (text + image + video) framework for detecting fake news and misinformation in the Bengali (Bangla) language. It introduces a Mixture-of-Modality-Experts (MoME) fusion architecture that dynamically routes multimodal representations to specialized expert sub-networks, enabling robust classification even when one or more modalities are missing.

The model operates in a two-phase pipeline:

1. Phase 1 — Continual Pre-Training (CPT): Adapts Google's Gemma-4-E2B foundation model to the Bengali language domain using Weight-Decomposed Low-Rank Adaptation (DoRA) on large-scale Bengali text corpora (~400K+ articles).
2. Phase 2 — Multimodal Fine-Tuning: Trains the full MoME classifier with three specialized encoders (Gemma-4-E2B for text, SigLIP2-SO400M for image, VideoMAEv2-Large for video) and a gated expert routing mechanism on the BTMD dataset using 5-fold stratified cross-validation.

Under five-fold stratified cross-validation, MoME-BanglaFake achieves:

• Binary Classification (Real vs. Fake): 91.48% Accuracy, 0.9148 Macro F1, 0.9682 ROC-AUC, 0.8298 MCC
• 4-Class Classification (Real, Misinformation, Satire, Clickbait): 83.25% Accuracy, 0.7901 Macro F1, 0.9517 ROC-AUC, 0.7405 MCC

---

Repository Structure

Faysal4200/mome-banglafake/
├── README.md                                # Comprehensive Model Card
│
├── continual-pretraining/                   # Phase 1: Continual Pre-Training (DoRA Adapter)
│   ├── adapter_config.json                  # DoRA config (use_dora: true, r=16, α=32)
│   ├── adapter_model.safetensors            # DoRA adapter weights (~150 MB)
│   ├── tokenizer.json                       # Tokenizer for Gemma-4-E2B
│   └── tokenizer_config.json
│
├── multimodal_finetuning_binary/            # Phase 2: Binary (Real vs Fake) 5-Fold Checkpoints
│   ├── fold_1/best_checkpoint.pt            # Fold 1 best model state dict (~578 MB)
│   ├── fold_2/best_checkpoint.pt            # Fold 2 best model state dict
│   ├── fold_3/best_checkpoint.pt            # Fold 3 best model state dict
│   ├── fold_4/best_checkpoint.pt            # Fold 4 best model state dict
│   └── fold_5/best_checkpoint.pt            # Fold 5 best model state dict
│
└── multimodal_finetuning_multiclass/        # Phase 2: Multi-Class (4-Class) 5-Fold Checkpoints
    ├── fold_1/best_checkpoint.pt            # Fold 1 best model state dict (~578 MB)
    ├── fold_2/best_checkpoint.pt            # Fold 2 best model state dict
    ├── fold_3/best_checkpoint.pt            # Fold 3 best model state dict
    ├── fold_4/best_checkpoint.pt            # Fold 4 best model state dict
    └── fold_5/best_checkpoint.pt            # Fold 5 best model state dict

---

Architecture Overview

Phase 1: Continual Pre-Training (CPT) with DoRA

Unlike standard LoRA, DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes pre-trained weight matrices into magnitude and directional components, applying low-rank updates strictly to the directional component. This provides learning capacity closer to full fine-tuning while maintaining parameter efficiency.

Parameter	Value
Base Model	google/gemma-4-E2B (5.12B params)
Adaptation Method	DoRA (Weight-Decomposed LoRA)
Rank (r)	16
Alpha (α)	32
Dropout	0.05
Learning Rate	7×10⁻⁵
LR Schedule	Cosine + 3% Warmup
Context Length	1,024 tokens
Precision	bfloat16
Target Modules	All linear layers (attention + MLP projections)
PEFT Version	0.19.1

Pre-Training Corpora:

• Bangla-TextBook — Educational Bengali prose (~10M tokens)
• XL-Sum (Bengali Subset) — BBC Bengali news articles
• Bangla Newspaper Dataset — 400,000+ crawled articles from prominent Bengali news publishers

Phase 2: Multimodal Mixture-of-Modality-Experts (MoME)

Modality	Encoder	Parameters	Output Dim
Text	Gemma-4-E2B + Phase 1 DoRA	5.12B (frozen base + DoRA)	768d
Image	SigLIP2-SO400M (patch14-384)	~400M (frozen)	768d
Video	VideoMAEv2-Large (5 chunks × 16 frames)	~305M (frozen)	768d

Mixture-of-Modality-Experts (MoME) Layer:

• Text Expert: Text representations (768d)
• Image Expert: Visual representations (768d)
• Video Expert: Spatiotemporal video representations (768d)
• Cross-Modal Expert: Joint concatenated multimodal representations (2304d)
• Masked Modality Router: Dynamically weights active experts and handles missing modalities via learnable null tokens.

---

Evaluation Results

Phase 1: Language Modeling Performance (Bengali Wikipedia)

Metric	Base Gemma-4-E2B	After CPT (Ours)	Absolute Δ	Relative Change
Perplexity (PPL) ↓	15.94	15.59	−0.35	−2.20%
Bits per Byte (BPB) ↓	0.4182	0.4149	−0.0033	−0.79%
Top-1 Accuracy ↑	47.46%	47.82%	+0.36	+0.76%
Top-5 Accuracy ↑	68.70%	68.75%	+0.05	+0.07%

---

Phase 2: 5-Fold Cross-Validation Performance on BTMD

Binary Classification (Real vs. Fake)

Metric	Mean ± Std	Overall	95% CI
Accuracy (%)	91.48 ± 0.65	91.48	[90.76, 92.19]
Balanced Accuracy (%)	91.49 ± 0.65	91.49	[90.76, 92.20]
F1 Score (Macro)	91.40 ± 0.62	91.48	[90.64, 92.13]
MCC	0.8300 ± 0.0131	0.8298	[0.8153, 0.8440]
AUC-ROC	0.9683 ± 0.0059	0.9682	[0.9640, 0.9724]

Multi-Class Classification (Real, Misinformation, Satire, Clickbait)

Metric	Mean ± Std	Overall	95% CI
Accuracy (%)	83.25 ± 1.77	83.25	[82.30, 84.19]
Balanced Accuracy (%)	80.86 ± 2.53	80.85	[79.56, 82.08]
F1 Score (Macro)	78.95 ± 2.50	79.01	[77.89, 80.24]
MCC	0.7414 ± 0.0263	0.7405	[0.7264, 0.7547]
AUC-ROC (OvR Macro)	0.9540 ± 0.0056	0.9517	[0.9475, 0.9559]

---

How to Use

1. Loading the Phase 1 DoRA Adapter (Bengali CPT Weights)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

repo_id = "Faysal4200/mome-banglafake"
base_model_name = "google/gemma-4-E2B"

# 1. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    repo_id, subfolder="continual-pretraining"
)

# 2. Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# 3. Load DoRA CPT adapter
model = PeftModel.from_pretrained(
    base_model, repo_id, subfolder="continual-pretraining"
)

print("DoRA CPT adapter loaded successfully!")

# Example: Generate Bengali text
inputs = tokenizer("বাংলাদেশের রাজধানী", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Downloading Phase 2 Binary Classification Checkpoints

import torch
from huggingface_hub import hf_hub_download

repo_id = "Faysal4200/mome-banglafake"

# Download Binary Fold 1 checkpoint
ckpt_path = hf_hub_download(
    repo_id=repo_id,
    filename="multimodal_finetuning_binary/fold_1/best_checkpoint.pt"
)

checkpoint = torch.load(ckpt_path, map_location="cpu")
print("Binary Checkpoint keys:", checkpoint.keys())

3. Downloading Phase 2 Multi-Class Classification Checkpoints

import torch
from huggingface_hub import hf_hub_download

repo_id = "Faysal4200/mome-banglafake"

# Download Multi-Class Fold 1 checkpoint
ckpt_path = hf_hub_download(
    repo_id=repo_id,
    filename="multimodal_finetuning_multiclass/fold_1/best_checkpoint.pt"
)

checkpoint = torch.load(ckpt_path, map_location="cpu")
print("Multi-Class Checkpoint keys:", checkpoint.keys())

---

Dataset

This model was trained on the BTMD (Bengali Trimodal Misinformation Dataset), a large-scale multimodal dataset containing 6,058 news instances across text, image, and video modalities.

• Dataset Repository: Faysal4200/BTMD
• Binary Classes: Real (3,029) / Fake (3,029)
• Fine-grained Classes: Real (3,029), Misinformation (1,859), Satire (594), Clickbait (576)
• Language: Bengali (Bangla)
• Inter-Annotator Agreement: Cohen's κ = 0.848

---

Citation

If you use this model or dataset in your research, please cite:

@article{ahmmed2025mome,
  title     = {MoME-BanglaFake: A Mixture-of-Modality-Experts Framework 
               for Bengali Fake News Detection Using a Novel Trimodal Dataset},
  author    = {Ahmmed, Faysal and Rafsan, Resadus Salehin and Akther, Airin 
               and Mansib, Muhtadi and Esika, Ainea Esrat and Mridha, F. M.},
  journal   = {International Journal of Information Management Data Insights},
  year      = {2026},
  note      = {Paper under review.}
}

---

Contact

• Faysal Ahmmed — 22-47069-1@student.aiub.edu | faysalahmmed4200@gmail.com
• F. M. Mridha — firoz.mridha@aiub.edu

Department of Computer Science, American International University-Bangladesh (AIUB)