README.md

---
license: mit
language:
- bo
base_model: KoichiYasuoka/bert-base-tibetan
tags:
- token-classification
- tibetan
- word-segmentation
- bert
- onnx
library_name: onnxruntime
---

# Tibetan Word Segmentation BUDA

Fine-tuned `KoichiYasuoka/bert-base-tibetan` for syllable-level Tibetan word segmentation. Predicts **B** (begin-word) or **I** (inside-word) for each syllable. Used as the segmentation engine in [TradutorBUDA](https://tashirabten.github.io/BUDA_APPs_Port/).

**99.95% accuracy** — B F1: 99.92% | I F1: 99.97% — on 20,903 validation sequences.

## Model Files

| File | Size | Description |
|------|------|-------------|
| `model_quantized.onnx` | 104.5 MB | INT8 quantized ONNX (deployed) |
| `tokenizer.json` | 0.8 MB | HuggingFace tokenizer |
| `vocab.txt` | 0.4 MB | Vocabulary |
| `config.json` | — | Model configuration |
| `segmentation_config.json` | — | Label mappings (`B`/`I`) |
| `dictionary_words.json` | 16.6 MB | 353K Tibetan words for constraint decoding |

> The FP32 ONNX (413 MB) is not included — use the INT8 model for CPU deployment.

## Architecture

Base: `KoichiYasuoka/bert-base-tibetan` (BERT-base, 110M parameters, pre-trained on 4.63 GB Tibetan text)
Task head: token classification → 2 labels (B / I)

## Results

| Metric | Score |
|--------|-------|
| Accuracy | 99.95% |
| B precision | 99.97% |
| B recall | 99.88% |
| **B F1** | **99.92%** |
| I precision | 99.95% |
| I recall | 99.99% |
| I F1 | 99.97% |

## Dataset

- **Train:** 208,077 sequences — **Val:** 20,903 sequences
- 200K synthetic sequences generated from 353K Tibetan dictionary terms (B/I labels derived from word boundaries)
- 16K silver-standard sequences from [Botok](https://github.com/Esukhia/botok) segmentation output
- Input format: space-separated Tibetan syllables; each syllable ends with a tsheg (་)

Example:
```
Syllables: བྱང་  ཆུབ་  སེམས་  དཔའི་  སྤྱོད་  པ
Labels:    B     I     B      I      B      I
Words:     བྱང་ཆུབ་  /  སེམས་དཔའི་  /  སྤྱོད་པ
```

## Training Configuration

- **Base model:** `KoichiYasuoka/bert-base-tibetan`
- **Optimizer:** AdamW (lr=3e-5, weight_decay=0.01)
- **LR schedule:** 10% linear warmup → constant
- **Epochs:** 5 (early stopping patience=3; did not trigger)
- **Batch size:** 64
- **Max sequence length:** 256 syllable-tokens
- **Mixed precision:** FP16 on NVIDIA A100 (40 GB)
- **Best checkpoint metric:** B F1

## Usage (ONNX / Java — TradutorBUDA)

The model is loaded by `TibetanSegmentationService.java` using ONNX Runtime:

```java
OrtSession session = env.createSession("models/segmentation_onnx/model_quantized.onnx", opts);
```

Inputs: `input_ids`, `attention_mask`, `token_type_ids` (shape: `[batch, 256]`)  
Output: `logits` (shape: `[batch, 256, 2]`) — argmax gives B=0 / I=1 per token.

## Usage (Python)

```python
import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np
import json, re

tokenizer = AutoTokenizer.from_pretrained("trabten/tibetan_segmentation_BUDA")
session = ort.InferenceSession("model_quantized.onnx", providers=["CPUExecutionProvider"])

MAX_LENGTH = 256
ID2LABEL = {0: "B", 1: "I"}

def segment(text):
    syllables = [s for s in re.split(r'(?<=་)', text.strip()) if s.strip()]
    token_ids, word_ids = [], []
    for i, syl in enumerate(syllables):
        ids = tokenizer.encode(syl, add_special_tokens=False)
        token_ids.extend(ids)
        word_ids.extend([i] * len(ids))

    cls, sep = tokenizer.cls_token_id, tokenizer.sep_token_id
    input_ids = [cls] + token_ids[:MAX_LENGTH-2] + [sep]
    word_ids  = [-1]  + word_ids[:MAX_LENGTH-2]  + [-1]
    attn_mask = [1] * len(input_ids)
    tok_types = [0] * len(input_ids)

    pad = MAX_LENGTH - len(input_ids)
    input_ids += [tokenizer.pad_token_id] * pad
    attn_mask += [0] * pad
    tok_types += [0] * pad

    feeds = {
        "input_ids":      np.array([input_ids],  dtype=np.int64),
        "attention_mask": np.array([attn_mask],  dtype=np.int64),
        "token_type_ids": np.array([tok_types],  dtype=np.int64),
    }
    logits = session.run(None, feeds)[0][0]
    preds = logits.argmax(axis=-1)

    syl_labels, prev = [], None
    for wid, pred in zip(word_ids, preds):
        if wid == -1: continue
        if wid != prev:
            syl_labels.append(ID2LABEL[pred])
        prev = wid

    words, current = [], []
    for syl, lbl in zip(syllables, syl_labels):
        if lbl == "B" and current:
            words.append("".join(current))
            current = [syl]
        else:
            current.append(syl)
    if current:
        words.append("".join(current))
    return words

print(segment("བྱང་ཆུབ་སེམས་དཔའི་སྤྱོད་པ"))
# ['བྱང་ཆུབ་', 'སེམས་དཔའི་', 'སྤྱོད་པ']
```

## Background

Tibetan has no spaces between words — segmentation is required before dictionary lookup, POS tagging, or machine translation. Traditional rule-based segmenters (e.g. Botok) rely on hand-curated lexicons and miss out-of-vocabulary terms. This model learns segmentation patterns from context, achieving near-perfect accuracy on Buddhist canonical vocabulary.

## References

- Yasuoka, K. "Universal Dependencies for Classical Tibetan." (base model)
- Esukhia. [Botok — Tibetan NLP toolkit](https://github.com/Esukhia/botok) (silver-standard data source)
- Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers." (ICLR 2019)