ViLegalBERT-ext

ViLegalBERT-ext is a Vietnamese legal-domain encoder language model obtained via continual pretraining of Qualcomm-AI-Research/BamiBERT on a large, cleaned, deduplicated corpus of Vietnamese legal text.

The name "ext" refers to the two key limits this model extends relative to a previous model - ViLegalBERT:

• Context length: 256 tokens → 2048 tokens
• Legal pretraining corpus size: 16GB → 20.57GB (~7.15B legal-domain tokens)

Unlike PhoBERT-based models, ViLegalBERT-ext requires no external Vietnamese word segmentation as a preprocessing step — it operates directly on raw text via byte-level BPE tokenizer.

Model Details

• Architecture: BERT-base (12 Transformer layers), RoBERTa-style masked language modeling objective (dynamic masking, no next-sentence-prediction)
• Base model: Qualcomm-AI-Research/BamiBERT (~103M parameters), itself trained from scratch on 129GB of general-domain Vietnamese text
• Tokenizer: Byte-level BPE extended from PhoGPT's Vietnamese tokenizer, vocab size 20,481 (inherited unchanged from BamiBERT)
• Max sequence length: 2048 tokens
• Precision: trained in BF16 mixed precision

Continual Pretraining Data

• Corpus: Vietnamese legal text
• Size: 20.57GB of raw legal text, tokenized and packed (~7.15B legal-domain tokens)

Training Procedure

• Objective: Masked language modeling with dynamic masking (15% mask probability), consistent with BamiBERT's original RoBERTa-style pretraining recipe
• Optimizer: AdamW
• Learning rate: 2e-5 (peak), linear schedule with warmup — chosen conservatively (~1/7 of BamiBERT's original 1.5e-4 peak LR) to reduce the risk of catastrophic forgetting during domain-adaptive continual pretraining
• Effective batch size: 144 (per-device batch size 48 × gradient accumulation 3)
• Hardware: single NVIDIA H100 80GB GPU

Intended Use

ViLegalBERT-ext is an encoder model intended for fill-mask and as a base for fine-tuning on downstream Vietnamese legal NLP tasks, such as:

• Legal document classification
• Legal named entity recognition
• Legal question answering / retrieval (as an embedding backbone)
• Legal text similarity / clause matching

It is not designed for open-ended text generation.

Evaluation

Evaluation results (downstream task benchmarks) coming soon.

Limitations

• Inherits any limitations of the base BamiBERT model, including potential underperformance on non-standard Vietnamese dialects and general temporal concept drift.
• Continual pretraining was conducted on legal text; general-domain qualitative checks did not reveal meaningful catastrophic forgetting, but no comprehensive general-domain benchmark evaluation has been run.

License

This model is a derivative of Qualcomm-AI-Research/BamiBERT and is released under the same license terms: BSD 3-Clause Clear and the Qualcomm Responsible AI License. See the base model's license for full terms.

Citation

If you use this model, please cite the original BamiBERT model.

  @article{BamiBERT,
  title    = {{BamiBERT: A New BERT-based Language Model for Vietnamese}},
  author   = {Dat Quoc Nguyen and Thinh Pham and Chi Tran and Linh The Nguyen},
  journal  = {arXiv preprint},
  volume   = {arXiv:2607.02259},
  year     = {2026}
}