File size: 9,069 Bytes

22d3c3e

---

license: apache-2.0
language:
- hi
- bn
- en
base_model: google/muril-base-cased
tags:
- text-classification
- fact-verification
- code-mixed
- indic-languages
pipeline_tag: text-classification
---


# MuRIL Fine-Tuned for Claim-Evidence Classification (SUPPORTS / REFUTES)

Fine-tuned [`google/muril-base-cased`](https://huggingface.co/google/muril-base-cased) for binary claim-evidence fact verification on code-mixed Hindi–Bengali–English text. Given a **claim** and a piece of **evidence**, the model predicts whether the evidence **SUPPORTS** or **REFUTES** the claim.

## How to Use

```python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

import torch



MODEL_NAME = "nirnit-13/muril-supports-refutes"



tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)

model.eval()



claim = "भारत दुनिया में दूध का सबसे बड़ा उत्पादक देश है।"

evidence = "भारत विश्व में दुग्ध उत्पादन में प्रथम स्थान पर है।"



inputs = tokenizer(claim, evidence, truncation=True, max_length=192, padding="max_length", return_tensors="pt")



with torch.no_grad():

    outputs = model(**inputs)

    probs = torch.softmax(outputs.logits, dim=-1)

    pred_id = torch.argmax(probs, dim=-1).item()



id2label = {0: "SUPPORTS", 1: "REFUTES"}

print(id2label[pred_id], "| confidence:", probs[0][pred_id].item())

```

## Performance (Dev Set)

| Metric | Value |
|---|---|
| Accuracy | 0.822 |
| Macro F1 | 0.801 |
| F1 (SUPPORTS) | 0.865 |
| F1 (REFUTES) | 0.738 |

## Limitations

- Trained on informal, code-mixed, social-media-style claims. Performance degrades on out-of-domain, formally-written text or differently-structured evidence sources (see "Domain Generalization Caveat" below).
- Best used with `max_length=192` tokenization to match training conditions.

---

## Training & Tuning Details

The sections below document the hyperparameter tuning methodology used to produce this model.

## Overview

This document describes the hyperparameter tuning process used to fine-tune `google/muril-base-cased` for a binary claim-evidence classification task (SUPPORTS vs. REFUTES) on code-mixed Hindi–Bengali–English data.

- **Task:** Given a claim and a piece of evidence, classify whether the evidence SUPPORTS or REFUTES the claim.
- **Model:** `google/muril-base-cased` (encoder-only, fine-tuned via `AutoModelForSequenceClassification`)
- **Framework:** HuggingFace `transformers` + `datasets`, run on Google Colab (free-tier T4 GPU)
- **Search library:** [Optuna](https://optuna.org/) (TPE sampler), integrated via `Trainer.hyperparameter_search()`

## Dataset

| Split | File | Size | Notes |
|---|---|---|---|
| Train | `train_subtask1.json` | 4,536 | Fields: `ID`, `Text`, `Evidence`, `Label` |
| Dev | `dev_subtask1.json` | 1,134 | Same structure; used for validation during tuning |
| Label distribution (train) | — | SUPPORTS: 3,040 / REFUTES: 1,496 | ~2:1 class imbalance |

## Why Hyperparameter Tuning Was Needed

Initial manually-configured training runs plateaued around **Macro F1 ≈ 0.79–0.80**, with a persistent gap between SUPPORTS and REFUTES F1 scores (minority class underperforming). Manual tuning (adjusting one variable at a time — dropout, label smoothing, class weights) was slow and occasionally made results worse by stacking multiple untested changes at once. Optuna was introduced to systematically and efficiently search the hyperparameter space using smarter sampling (TPE) and early pruning of poor trials.

## Search Setup

### Tunable Hyperparameters and Search Space

```python

def optuna_hp_space(trial):

    return {

        "learning_rate": trial.suggest_float("learning_rate", 1e-5, 5e-5, log=True),

        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32]),

        "num_train_epochs": trial.suggest_int("num_train_epochs", 3, 16),

        "weight_decay": trial.suggest_float("weight_decay", 0.0, 0.1),

        "warmup_ratio": trial.suggest_float("warmup_ratio", 0.0, 0.2),

        "max_grad_norm": trial.suggest_float("max_grad_norm", 0.5, 2.0),

    }

```

### Fixed Settings (not tuned)

- `max_length = 192` (tokenization)
- `fp16 = True` (mixed precision, required for free-tier Colab GPU memory limits)
- `eval_strategy = "epoch"`, `save_strategy = "epoch"`
- `save_total_limit = 1` (only the best/most recent checkpoint retained)
- `metric_for_best_model = "f1"` (macro F1 used as the trial objective)
- Class-weighted `CrossEntropyLoss` (weights computed automatically from train label frequencies) to address the ~2:1 SUPPORTS:REFUTES imbalance

### Persistence

Since Colab's free tier disconnects after a time/usage limit, the Optuna study was configured with persistent SQLite storage so the search could resume across sessions without losing trial history:

```python

storage_path = "sqlite:////content/drive/MyDrive/muril_ckpt/optuna_study.db"



best_trial = trainer.hyperparameter_search(

    direction="maximize",

    backend="optuna",

    hp_space=optuna_hp_space,

    compute_objective=lambda metrics: metrics["eval_f1"],

    n_trials=15,

    study_name="muril_hp_search",

    storage=storage_path,

    load_if_exists=True,

)

```

- **Only the `.db` file is required to resume a search** across sessions/accounts — model checkpoints are not needed for this, since each trial trains a fresh model via `model_init()`.
- `n_trials` is a cumulative budget across all resumed sessions, not per-session.

## Results

**Total trials run:** 15 (several pruned early by Optuna due to clearly poor intermediate performance, e.g. high learning rates that failed to converge).

### Best Trial

| Hyperparameter | Value |
|---|---|
| `learning_rate` | 1.7991465194475932e-05 |
| `per_device_train_batch_size` | 32 |
| `num_train_epochs` | 16 |
| `weight_decay` | 0.0764751706565559 |
| `warmup_ratio` | 0.026509480810828625 |
| `max_grad_norm` | 1.2004964752527867 |

**Dev set performance (best trial):**

| Metric | Value |
|---|---|
| Accuracy | 0.822 |
| Macro F1 | 0.801 |
| F1 (SUPPORTS) | 0.865 |
| F1 (REFUTES) | 0.738 |

### Key Findings from the Search

- **Higher batch size (32) + moderate-high learning rate (~1.5–1.8e-5) + more epochs (10–16)** consistently produced the best results.
- **Very high learning rates (>3e-5) combined with small batch sizes** caused training to fail to converge (loss stuck near the random-guessing baseline of ~1.386) — these trials were pruned or scored near F1 ≈ 0.39–0.50.
- Smaller batch sizes (4–8) produced noisier, less reliable gradients and were more prone to early pruning.

## Known Limitation: Overfitting at High Epoch Counts

A full training run at the best trial's settings (16 epochs) showed a classic overfitting pattern when loss curves were inspected in detail:

- Training loss decreased continuously throughout all 16 epochs.
- Validation loss reached its minimum around **epoch 5–6** and increased steadily afterward.
- Despite this, Macro F1 continued to creep upward slightly through epoch 16 — meaning `metric_for_best_model="f1"` may have selected an already-overfit checkpoint rather than the true best-generalizing one (epoch 5–6, by validation loss).

**Recommendation for future runs:** either cap `num_train_epochs` closer to 6–8, or switch `metric_for_best_model` to `"eval_loss"` (with `greater_is_better=False`) to select the checkpoint with the best generalization rather than the highest (potentially overfit) F1.

## Domain Generalization Caveat

The tuned model performs well on data matching the training distribution (informal, code-mixed social-media-style claims) but was observed to generalize poorly to:
1. Clean, formally-written factual claims (out-of-domain test set) — near-total prediction collapse toward REFUTES.
2. A structurally different evidence-retrieval pipeline (news-article snippets with fusion-ranked retrieval scores, from a separate subtask) — Macro F1 dropped to ~0.60, with SUPPORTS recall collapsing to 0.45.

This indicates the tuned hyperparameters optimize well for the training/dev distribution, but do not by themselves solve domain-shift generalization — a separate concern from hyperparameter tuning.

## Reproducing This Search

1. Mount Google Drive and place `train_subtask1.json` / `dev_subtask1.json` in the configured folder.
2. Run the Optuna search cell — it will create (or resume, if `optuna_study.db` already exists at the given path) the study and run trials up to `n_trials`.
3. Retrieve `best_trial.hyperparameters` and plug them into a final `TrainingArguments` for a full training run.
4. Save the resulting model (`trainer.save_model(...)`) for downstream inference.