gaperon-mmbert-tamil

A multi-head text-quality classifier for Tamil, fine-tuned from jhu-clsp/mmBERT-base (a ModernBERT-architecture multilingual encoder) as part of the Gaperon data-curation pipeline. Given a passage of text, it predicts a low / medium / high rating along six quality dimensions, intended for scoring and filtering web/corpus text before it's used for further model training.

This is the best checkpoint (selected on validation mean_macro_f1) from training run mmbert_tamil_v1, saved at epoch 3.

Model description

• Backbone: jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, hidden size 768, 8192 max sequence length, mean pooling over token representations).
• Heads: six independent linear heads (Linear(768 → 3)), one per label dimension, applied to the mean-pooled backbone output. Each head is a 3-way classifier over {low: 0, medium: 1, high: 2}.
• Label dimensions: clarity, coherence, depth, grammar, usefulness, overall.

The backbone weights are in model.safetensors / config.json (standard HF format, loadable with AutoModel). The six classification heads are not part of the HF backbone class and are stored separately in heads.pt (see usage below).

Intended use

Rating the quality of Tamil text (e.g. web-scraped documents) along the six dimensions above, for filtering/weighting examples in a training corpus. Not intended as a general-purpose sentence encoder or for tasks unrelated to the quality-rating schema it was trained on.

Training data

Fine-tuned on an in-house tagged corpus of Tamil text (data_gaperon/tagged_v2/tamil), labeled with low/medium/high ratings for each of the six dimensions.

Hyperparameters

Hyperparameter	Value
max_length	512
batch_size	32
learning_rate	2e-5
weight_decay	0.01
dropout	0.1
frozen layers	0
warmup_ratio	0.1
seed	42

Evaluation results (best checkpoint, epoch 3)

Macro-F1 / accuracy per dimension on the validation set (no held-out test split was evaluated for this run):

Dimension	Val F1	Val Acc
clarity	0.5734	0.5873
coherence	0.5697	0.6019
depth	0.4731	0.7970
grammar	0.5276	0.6197
usefulness	0.5398	0.6592
overall	0.4993	0.6229
mean macro-F1	0.5305	–

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

repo_id = "<your-username>/gaperon-mmbert-tamil"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
backbone = AutoModel.from_pretrained(repo_id)

heads_path = hf_hub_download(repo_id, filename="heads.pt")
ckpt = torch.load(heads_path, map_location="cpu")
label_dims = ckpt["label_dims"]              # ["clarity", "coherence", "depth", "grammar", "usefulness", "overall"]
label2id = ckpt["label2id"]                  # per-dimension {"low": 0, "medium": 1, "high": 2}
heads_state = ckpt["heads_state_dict"]        # one Linear(768, 3) per dimension

heads = {dim: torch.nn.Linear(768, 3) for dim in label_dims}
for dim, head in heads.items():
    head.weight.data = heads_state[f"{dim}.weight"]
    head.bias.data = heads_state[f"{dim}.bias"]
    head.eval()

backbone.eval()
text = "Your input passage here."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    hidden = backbone(**inputs).last_hidden_state          # (1, seq_len, 768)
    mask = inputs["attention_mask"].unsqueeze(-1)
    pooled = (hidden * mask).sum(1) / mask.sum(1)           # mean pooling

id2label = {dim: {v: k for k, v in mapping.items()} for dim, mapping in label2id.items()}
for dim, head in heads.items():
    pred_id = head(pooled).argmax(-1).item()
    print(dim, "->", id2label[dim][pred_id])

Limitations

• Trained and evaluated only on Tamil text matching the tagging schema of the source corpus; quality ratings may not transfer to other domains or genres.
• Lowest-resource of the three Gaperon mmBERT models trained so far (3 epochs of saved checkpoints), with the lowest mean macro-F1 (0.5305) and no held-out test evaluation — treat its outputs as a noisier filtering signal than the English/Hindi models.
• heads.pt requires custom loading code (shown above) — it is not loadable via AutoModelForSequenceClassification.