Upload README.md

README.md

@@ -0,0 +1,84 @@
+---
+license: cc-by-nc-4.0
+language:
+- vi
+pipeline_tag: translation
+tags:
+- dialect-normalization
+- vietnamese-dialect
+- mbart
+- social-media
+- low-resource
+datasets:
+- Biu3010/ViDia2Std
+base_model: facebook/mbart-large-50
+library_name: transformers
+---
+
+# mBART-ViDia2Std
+
+<p align="center">
+  <a href="https://huggingface.co/datasets/Biu3010/ViDia2Std">ViDia2Std Dataset</a> |
+  <a href="https://ojs.aaai.org/index.php/AAAI/article/view/40247">AAAI-26 Paper</a>
+</p>
+
+This is a version of [mBART-large-50](https://huggingface.co/facebook/mbart-large-50) fine-tuned on the entire [ViDia2Std](https://huggingface.co/datasets/Biu3010/ViDia2Std) corpus (13,657 pairs, all 63 Vietnamese provinces) plus data augmentation, to translate Vietnamese dialects and non-standard social media text into standard Vietnamese. It is released as the strongest version for real-world use, e.g. as a preprocessing step for downstream Vietnamese NLP tasks.
+
+> **Warning — data leakage.** This model was trained on **all splits of ViDia2Std, including `test`**. Do not use it as a baseline on the ViDia2Std test set; to reproduce the paper's baselines, train on the `train` split only.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+model_path = "Biu3010/dialect-normalizer"
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to(device)
+
+def normalize_text(text):
+    inputs = tokenizer(text, return_tensors="pt", max_length=128, truncation=True)
+    input_ids = inputs.input_ids.to(device)
+    attention_mask = inputs.attention_mask.to(device)
+
+    with torch.no_grad():
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_length=128,
+            num_beams=3,
+            early_stopping=True,
+        )
+
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+dialect_text = "răng mà bựa ni trời nắng rứa hề"
+print(f"Dialect : {dialect_text}")
+print(f"Standard: {normalize_text(dialect_text)}")
+# Expected output: "sao mà hôm nay trời nắng thế nhỉ"
+```
+
+If you encounter language-token issues, set `tokenizer.src_lang = "vi_VN"` and pass `forced_bos_token_id=tokenizer.lang_code_to_id["vi_VN"]` to `model.generate()`.
+
+## Reference results
+
+The paper's mBART-large-50 baseline (trained on the `train` split only) achieves **BLEU 0.8166 · ROUGE-L 0.9384 · METEOR 0.8925** on ViDia2Std. This full-corpus model cannot be legitimately evaluated on that test set.
+
+## Citation
+
+```bibtex
+@article{Anh_Ta_Van_Dinh_Nguyen_2026,
+  title     = {ViDia2Std: A Parallel Corpus and Methods for Low-Resource Vietnamese Dialect-to-Standard Translation},
+  author    = {Anh Ta, Khoa and Van Dinh, Nguyen and Nguyen, Kiet Van},
+  journal   = {Proceedings of the AAAI Conference on Artificial Intelligence},
+  volume    = {40},
+  number    = {36},
+  pages     = {29995--30004},
+  year      = {2026},
+  month     = {Mar.},
+  url       = {https://ojs.aaai.org/index.php/AAAI/article/view/40247},
+  doi       = {10.1609/aaai.v40i36.40247}
+}
+```