phuongntc/Multi_EvalSumViet2

MultiEvalSumViet2: Multi-Criteria Evaluation for Vietnamese Summarization with Reinforcement Learning Applications

MultiEvalSumViet2 is a Vietnamese learned evaluator for scoring a candidate summary against its source document on three related and complementary criteria:

• Faithfulness (F): whether information stated in the summary is supported by the source; unsupported entities, relations, quantities, attributions, or causal claims reduce the score.
• Coherence (C): whether the summary is logically organized, self-contained, and linguistically well formed.
• Relevance (R): whether the summary preserves salient source information while avoiding tangential or low-value details.

A summary can therefore be factually supported but still receive a lower Relevance score if it omits important source content. The criteria are modeled separately but jointly from the same document-summary representation.

Output

For each (document, summary) pair, the evaluator returns:

• pred_faith in [0, 1]
• pred_coherence in [0, 1]
• pred_relevance in [0, 1]

Optional mapping to the original 1–5 scale:

score_1to5 = 4 * score_0to1 + 1

When one scalar is required for secondary analysis, filtering, preference construction, or reward computation, the paper uses:

Overall = 0.5 * F + 0.3 * R + 0.2 * C

The criterion-wise outputs remain the primary model outputs.

Architecture

MultiEvalSumViet2 uses a ViDeBERTa document-summary cross-encoder. The source document and candidate summary are jointly serialized so that source and summary tokens interact contextually before prediction.

document + summary
      ↓
ViDeBERTa cross-encoder
      ↓
masked mean pooling
      ↓
Linear(256) + GELU + Dropout(0.1)
      ↓
three independent criterion heads
      ↓
F, C, R

All three criterion scores are produced from one shared encoding in a single non-autoregressive forward pass.

Training objective

The model combines criterion-wise regression with within-document pairwise ranking:

L_hybrid = L_reg + λ L_rank

• L_reg calibrates predictions to the absolute human-reviewed score scale.
• L_rank preserves criterion-specific ordering among alternative summaries of the same source document.
• Ranking margin: m = 0.05.
• Ranking-loss weight: λ = 0.35, selected during development tuning.

The two constituent losses are standard objectives; the model design combines them in a criterion-wise, document-conditioned, group-balanced formulation.

Data and supervision

MultiEvalSumViet2 is developed from a News evaluation resource containing:

• 13,476 VnExpress source articles from 2022–2024.
• 6 candidate summaries per source.
• 80,856 (document, summary) evaluations in total.
• 72,768 evaluations from 12,128 source documents in the training-and-validation pool.
• 8,088 evaluations from 1,348 source documents in the leakage-safe held-out test split.
• Candidate systems: GPT-4o, Gemini, LLaMA-3.2 1B, LLaMA-3.2 3B, LLaMA-3.1 8B, and a ViT5-large summarizer fine-tuned on a filtered VNDS subset.

The evaluation prompt was calibrated on 100 candidate summaries using human ratings as reference; the selected Gemini prompt reached mean Cohen's kappa 0.78 across F/C/R. Gemini then provided initial 1–5 ratings, which were reviewed over the full corpus by 12 trained volunteer annotators; approximately 10% of initial labels were revised. A separate 450-pair blind audit evaluates rubric reproducibility without exposing the original labels to the re-annotators.

The released dataset is available at: https://huggingface.co/datasets/phuongntc/data_MultiEvalSumViet2

Evaluation

In-domain News

The held-out News test split contains 1,348 source documents / 8,088 document-summary pairs, separated by doc_id.

Out-of-domain IT-textbook benchmark

The current OOD benchmark contains 900 document-summary pairs = 150 IT source passages × 6 heterogeneous summarization systems, covering 13 IT subject areas. The six systems are GPT-4o, Gemini, LLaMA-3.2 1B, LLaMA-3.2 3B, LLaMA-3.1 8B, and the domain-adapted ViT5 model.

Against human-reviewed F/C/R ratings, the weighted Overall score reaches:

• Pearson r = 0.829
• Spearman ρ = 0.691
• MAE = 0.107

This benchmark provides evidence of transfer from news to technical educational text; it should not be interpreted as universal cross-domain robustness.

Efficiency profile

On a single NVIDIA T4, the released evaluator was profiled on 1,344 document-summary pairs:

• approximately 0.047 s/pair
• approximately 63 s total
• approximately 1.6 GB GPU memory

All F/C/R scores are produced together in one local forward pass.

Recommended usage

This repository contains custom criterion heads. Do not use it as a text-generation / summarization pipeline. Use the included modeling_summary_evaluator.py loader so that the backbone, trunk, and three criterion heads are restored correctly.

import os
import importlib.util
import torch
from huggingface_hub import snapshot_download
from transformers import DataCollatorWithPadding

REPO_ID = "phuongntc/Multi_EvalSumViet2"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

repo_dir = snapshot_download(repo_id=REPO_ID, repo_type="model")

loader_path = os.path.join(repo_dir, "modeling_summary_evaluator.py")
spec = importlib.util.spec_from_file_location("mse", loader_path)
mse = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mse)

model, tokenizer, _ = mse.load_for_inference(repo_dir, device=DEVICE)
model.eval()

docs = ["Văn bản nguồn ..."]
summaries = ["Bản tóm tắt ..."]

enc = mse.encode_pair(tokenizer, docs, summaries)
collator = DataCollatorWithPadding(tokenizer=tokenizer, padding=True)
features = [{k: enc[k][0] for k in enc.keys()}]
batch = collator(features)

with torch.inference_mode():
    scores = model(
        batch["input_ids"].to(DEVICE),
        batch["attention_mask"].to(DEVICE)
    )[0].detach().cpu().tolist()

print({
    "faithfulness": scores[0],
    "coherence": scores[1],
    "relevance": scores[2],
})

Intended use

Appropriate uses include:

• Vietnamese summarization evaluation;
• large-scale filtering and dataset curation;
• within-source preference construction;
• a frozen scoring/reward component in downstream optimization workflows.

A separately published verifier-guided PPO/GRPO study reused MultiEvalSumViet2 as a frozen reward evaluator; those policy-optimization results are downstream application evidence rather than training results of this repository.

Limitations

• Training data are news-centered.
• The IT benchmark adds one technical-educational OOD setting, but more genres are needed for broader generalization claims.
• The 450-pair blind audit evaluates reproducibility of the F/C/R rubric; it is not a direct estimate of corpus-label error.
• The reported efficiency profile is specific to the stated T4 configuration.
• As with any learned evaluator used in RL/filtering, it should not be the sole optimization signal when reward hacking is a concern.

Reproducibility

For paper reproduction, pin a specific repository revision:

repo_dir = snapshot_download(
    repo_id="phuongntc/Multi_EvalSumViet2",
    repo_type="model",
    revision="<COMMIT_HASH_OR_TAG>",
)

License

Model/code files in this repository are released under Apache License 2.0.

Citation

Model DOI: 10.57967/hf/7956

@misc{multievalsumviet2_model,
  title        = {MultiEvalSumViet2: Vietnamese Multi-Criteria Summary Evaluator},
  author       = {Thi Thu Phuong Tran},
  year         = {2026},
  howpublished = {Hugging Face Hub},
  url          = {https://huggingface.co/phuongntc/Multi_EvalSumViet2},
  doi          = {10.57967/hf/7956}
}

Contact

Maintainer: Thi Thu Phuong Tran
Affiliations: Hanoi Metropolitan University; VNU University of Engineering and Technology, Vietnam National University
Email: tttphuong2@daihocthudo.edu.vn