Bangla Logical Fallacy — Blind Spots of Qwen3-1.7B

This bucket documents a small evaluation of Qwen/Qwen3-1.7B on Bangla logical fallacy detection. It contains the evaluation dataset and the results.

1. Blind Spot

In everyday Bangla, a common trick is using the same word with two different meanings inside one argument — for example গরম (hot tea) vs গরম (hot weather), or পাখা (bird's wing) vs পাখা (fan blade). This is called equivocation.

So a small multilingual model can label a normal causal fallacy correctly, but fail completely when the fallacy is hidden in a word with two meanings.

That is the blind spot I tested.

2. Evaluation

• Model: Qwen/Qwen3-1.7B (1.7B parameters, Apache-2.0)
• Platform: Kaggle
• Decoding: greedy, temperature = 0.0
• Output: Excel (label, confidence, reason)
• Dataset: 20 Bangla sentences across three fallacy types
  • equivocation: 3
  • false dilemma: 7
  • false cause: 10

Result

Overall: 15 / 20 correct (75%)

What went wrong

All 3 equivocation sentences were misclassified. Each time, the model picked a different label based on a surface feature of the sentence — if the sentence had "তাই" (therefore), it said false_cause; if the sentence contrasted two things, it said false_dilemma.

The model never noticed that the same word was being used with two different meanings. It reads words as having one fixed meaning, so a fallacy built on wordplay becomes invisible to it.

This is not a vocabulary problem. The words involved (গরম, পাখা, মিষ্টি) are common Bangla words that any multilingual model has seen.

3. Path Forward

Data

• Word-sense minimal pairs. Same sentence, same word, two meanings — one with the fallacy, one without. Teach the model to track meaning.
• Contrastive pairs. Same topic, same words, one with the fallacy and one without. This breaks the "তাই → false cause" shortcut.
• Target size: 25k–50k examples.

Fine-tuning

• Sense-tracking loss. Train the model to predict which meaning each word carries, given the full argument.
• Calibration. Use human disagreement as soft labels. When two labels are both plausible, the model should hedge, not guess confidently.

Architecture

• Two-stage decoder. First resolve word meanings, then classify the fallacy from the resolved representation.

Files

• bangla_fallacy_results.csv — all 20 rows with columns: label, input, explanation, model_prediction_label, model_explanation, correct/error, error_type, model_name, temperature

Reproducibility

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Qwen/Qwen3-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float16,
    device_map="auto", trust_remote_code=True,
).eval()

Prompt

You are an expert in Bangla informal logic. Read the argument. Identify the fallacy type. Reply ONLY with JSON.

Argument: {utterance}

Return JSON: {"label": one of ["equivocation", "false_dilemma", "false_cause", "no_fallacy"], "confidence": 0-100, "reason": "one sentence in English"}

Limitations

• Only 20 samples. This is a pilot, not a full benchmark.
• No no_fallacy control examples.
• One annotator.
• One model tested.

Note

• Dataset status: Work in progress.
• This bucket currently has a small pilot dataset (20 Bangla sentences).
• A full dataset is being prepared and will be released soon.