README.md

---
language:
- vi
license: other
license_name: bsd-3-clause-clear-and-qualcomm-responsible-ai
license_link: https://huggingface.co/Qualcomm-AI-Research/BamiBERT#license
base_model: Qualcomm-AI-Research/BamiBERT
tags:
- fill-mask
- masked-language-modeling
- bert
- roberta
- vietnamese
- legal
- legal-nlp
pipeline_tag: fill-mask
---

# ViLegalBERT-ext

**ViLegalBERT-ext** is a Vietnamese legal-domain encoder language model obtained via **continual pretraining** of [`Qualcomm-AI-Research/BamiBERT`](https://huggingface.co/Qualcomm-AI-Research/BamiBERT) on a large, cleaned, deduplicated corpus of Vietnamese legal text.

The name **"ext"** refers to the two key limits this model extends relative to a previous model - [ViLegalBERT](https://huggingface.co/ntphuc149/ViLegalBERT):
- **Context length**: 256 tokens → **2048 tokens**
- **Legal pretraining corpus size**: 16GB → **20.57GB** (~7.15B legal-domain tokens)

Unlike PhoBERT-based models, ViLegalBERT-ext requires **no external Vietnamese word segmentation** as a preprocessing step — it operates directly on raw text via byte-level BPE tokenizer.

## Model Details

- **Architecture**: BERT-base (12 Transformer layers), RoBERTa-style masked language modeling objective (dynamic masking, no next-sentence-prediction)
- **Base model**: [`Qualcomm-AI-Research/BamiBERT`](https://huggingface.co/Qualcomm-AI-Research/BamiBERT) (~103M parameters), itself trained from scratch on 129GB of general-domain Vietnamese text
- **Tokenizer**: Byte-level BPE extended from PhoGPT's Vietnamese tokenizer, vocab size 20,481 (inherited unchanged from BamiBERT)
- **Max sequence length**: 2048 tokens
- **Precision**: trained in BF16 mixed precision

## Continual Pretraining Data

- **Corpus**: Vietnamese legal text
- **Size**: 20.57GB of raw legal text, tokenized and packed (~7.15B legal-domain tokens)

## Training Procedure

- **Objective**: Masked language modeling with dynamic masking (15% mask probability), consistent with BamiBERT's original RoBERTa-style pretraining recipe
- **Optimizer**: AdamW
- **Learning rate**: 2e-5 (peak), linear schedule with warmup — chosen conservatively (~1/7 of BamiBERT's original 1.5e-4 peak LR) to reduce the risk of catastrophic forgetting during domain-adaptive continual pretraining
- **Effective batch size**: 144 (per-device batch size 48 × gradient accumulation 3)
- **Hardware**: single NVIDIA H100 80GB GPU

## Intended Use

ViLegalBERT-ext is an encoder model intended for **fill-mask** and as a base for fine-tuning on downstream Vietnamese legal NLP tasks, such as:
- Legal document classification
- Legal named entity recognition
- Legal question answering / retrieval (as an embedding backbone)
- Legal text similarity / clause matching

It is **not** designed for open-ended text generation.

## Evaluation

*Evaluation results (downstream task benchmarks) coming soon.*

## Limitations

- Inherits any limitations of the base BamiBERT model, including potential underperformance on non-standard Vietnamese dialects and general temporal concept drift.
- Continual pretraining was conducted on legal text; general-domain qualitative checks did not reveal meaningful catastrophic forgetting, but no comprehensive general-domain benchmark evaluation has been run.

## License

This model is a derivative of `Qualcomm-AI-Research/BamiBERT` and is released under the same license terms: **BSD 3-Clause Clear** and the **Qualcomm Responsible AI License**. See the [base model's license](https://huggingface.co/Qualcomm-AI-Research/BamiBERT#license) for full terms.

## Citation

If you use this model, please cite the original BamiBERT model.

```bibtex
  @article{BamiBERT,
  title    = {{BamiBERT: A New BERT-based Language Model for Vietnamese}},
  author   = {Dat Quoc Nguyen and Thinh Pham and Chi Tran and Linh The Nguyen},
  journal  = {arXiv preprint},
  volume   = {arXiv:2607.02259},
  year     = {2026}
}
```