CS221DoAn/Do_an_group_mBERT

mBERT-FoodNER: Multilingual Baseline for Vietnamese Food Order Extraction

Table of contents

1. Introduction
2. Dataset Overview
3. Entity Label Space
4. Empirical Evaluation
5. Training Hyperparameters
6. Using mBERT-FoodNER with transformers
7. Architecture Limitations (mBERT vs PhoBERT)
8. Authors & Citation

---

1. Introduction

CS221DoAn/Do_an_group_mBERT is a fine-tuned Named Entity Recognition (NER) model based on the pre-trained Multilingual BERT (mBERT) architecture. It is specifically trained for Token Classification on domain-specific Vietnamese unstructured text: Online Food Delivery Orders and Messages.

In our research project for the CS221.Q21 course (Natural Language Processing), this mBERT model serves as a strong deep-learning baseline. It is used to compare and evaluate the effectiveness of monolingual models (like PhoBERT) versus multilingual architectures in handling the complex morphology of the Vietnamese language.

2. Dataset Overview

The model was fine-tuned on a custom, manually annotated dataset consisting of 2,325 real-world food ordering messages extracted from Facebook comments (specifically from Sơn Nguyễn vegetarian restaurant). The dataset exhibits strong social media characteristics including teencode, abbreviations, and missing diacritics.

• Train Set: 1,860 samples (80%)
• Validation Set: 232 samples (10%)
• Test Set: 233 samples (10%)

3. Entity Label Space

The model utilizes the standard BIO (Begin - Inside - Outside) tagging scheme across 15 distinct entity labels, mapping to 7 core information classes:

Tag	Entity Class	Description	Examples
B-FOOD, I-FOOD	FOOD	Names of dishes, drinks, toppings	cơm sườn, trà sữa, trân châu
B-QUANTITY, I-QUANTITY	QUANTITY	Portions, servings, item counts	1p, 2 ly, một hộp, 3 suất
B-NOTE, I-NOTE	NOTE	Special requests, flavor modifications	không, ít ngọt, nhiều
B-PLACE, I-PLACE	PLACE	Delivery location, addresses	Ký túc xá khu A, tòa D6, rào b4
B-PHONE, I-PHONE	PHONE	Receiver's contact number	0794987xxx, 0903123xxx
B-TIME, I-TIME	TIME	Expected/requested delivery time	lúc 11h30, trưa nay, 18h
B-PRICE, I-PRICE	PRICE	Monetary cost, item prices	35k, 40000, 25 ngàn
O	OUTSIDE	Non-entity words, syntax connecting words	cho em, giao qua, với, ạ, nhé

4. Empirical Evaluation

The model was rigorously evaluated on an unseen Test Set using strict Entity-level F1-Scores via the seqeval framework.

• mBERT Macro F1-Score: 0.9880
• Comparison: While mBERT achieved an excellent Macro F1-score of 0.9880, it slightly underperformed compared to the monolingual PhoBERT architecture (0.9913) due to limitations in handling Vietnamese compound words during tokenization.

5. Training Hyperparameters

The model was fine-tuned with the following configuration:

• Learning Rate: 2e-5
• Batch Size (Train/Eval): 16
• Epochs: 10
• Weight Decay: 0.01
• Optimizer: AdamW
• Evaluation Strategy: Epoch-based (loading the best model based on F1-score at the end)

6. Using mBERT-FoodNER with transformers

Note on Inference: Unlike PhoBERT, this mBERT pipeline does NOT require the py_vncorenlp word segmenter. It applies the WordPiece algorithm directly to raw syllables.

Installation

Install the required libraries using pip:

pip install -q transformers torch

Example usage (Full Inference Pipeline)

import re
import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

# 1. Setup Device and Load Model from Hugging Face Hub
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
MODEL_ID = "CS221DoAn/Do_an_group_mBERT"

print("Loading Model and Tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForTokenClassification.from_pretrained(MODEL_ID).to(device)
model.eval()

def predict_food_order(raw_text):
    print(f"\nInput: {raw_text}")
    
    # Preprocessing & Text Cleaning
    clean_text = re.sub(r'([.,()!?:+])', r' \1 ', raw_text.replace('_', ' '))
    clean_text = re.sub(r'\s+', ' ', clean_text).strip()
    
    # Syllable-level tokenization (Whitespace split for WordPiece)
    tokens = clean_text.split()              
    
    # Tokenization & Subword Alignment
    input_ids = [tokenizer.cls_token_id]
    word_ids = [None]
    
    for i, word in enumerate(tokens):
        sub_ids = tokenizer.encode(word, add_special_tokens=False)
        input_ids.extend(sub_ids)
        word_ids.extend([i] * len(sub_ids))
        
    input_ids.append(tokenizer.sep_token_id)
    word_ids.append(None)
    
    # Forward Pass / Inference
    inputs = torch.tensor([input_ids]).to(device)
    with torch.no_grad():
        logits = model(inputs).logits
        preds = logits.argmax(dim=-1)[0].tolist()
        
    # Display Extracted Entities
    prev_idx = None
    for i, idx in enumerate(word_ids):
        if idx is not None and idx != prev_idx:
            tag_id = preds[i]
            label = model.config.id2label.get(tag_id, "O")
            print(f"{tokens[idx]:<20} {label}")
            prev_idx = idx

# --- EXECUTE TEST CASES ---
test_cases = [
    "1p cải xào, gà lát chiên giòn, chả giò, cơm thêm, Start Cf, 0384293xxx, giao lúc 12h15 ạ",
    "em 1p sườn ngào, bầu xào, chả giò, rau củ kho + cơm thêm. giao rào b4, 11h30. 0773570xxx.",
    "1p 15k thập cẩm ( cơm thêm) giao rào D6 0585115xxx ạ"
]

for sample in test_cases:
    predict_food_order(sample)

7. Architecture Limitations (mBERT vs PhoBERT)

While mBERT is a powerful multilingual model trained on 104 languages, our empirical analysis reveals a specific limitation when applied to Vietnamese NER tasks compared to monolingual architectures:

• WordPiece vs. Vietnamese Morphology: mBERT utilizes the WordPiece tokenization algorithm and does not natively support Vietnamese word segmentation tools (like VnCoreNLP). As a result, it applies WordPiece directly to discrete syllables.
• Boundary Breaking: When processing compound food names (e.g., cơm sườn, rau muống), the lack of prior word segmentation causes the model to fragment these cohesive terms, which slightly degrades the ability to preserve semantic boundaries compared to PhoBERT.

8. Authors & Citation

This project was developed for the Natural Language Processing (CS221.Q21) course at the University of Information Technology (UIT) - VNU-HCM.

• Students: Võ Thành Lộc (24520989), Nguyễn Anh Nguyên (24521185)
• Instructor: Ph.D. Nguyễn Trọng Chỉnh
• Date: July 2026

If you use this model in your academic research or projects, please cite our project and the original BERT paper:

@misc{cs221_food_order_ner_mbert,
  author = {Vo Thanh Loc and Nguyen Anh Nguyen},
  title = {Food Order Extraction: Multilingual BERT Baseline for Vietnamese NER},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = https://huggingface.co/CS221DoAn/Do_an_group_mBERT
}

@inproceedings{bert,
  title = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author = {Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
  booktitle = {Proceedings of NAACL-HLT 2019},
  year = {2019},
  pages = {4171--4186}
}