---
language:
- vi
- ede
tags:
- vanilla-transformer
- morpheme-tokenizer
- information-retrieval
- cross-lingual-retrieval
- EViRAL
- ede
- vietnamese
---

# Vanilla Transformer + Morpheme Tokenizer v4 — EViRAL

Cross-lingual retrieval model:

**Ede query → Vietnamese passage retrieval**

## Files

| File | Description |
|---|---|
| `mlm.pt` | MLM pre-trained encoder |
| `align.pt` | Cross-lingual aligned encoder |
| `finetune.pt` | Contrastive fine-tuned encoder (best) |
| `eval_results.csv` | Full val/test metrics |
| `eval_100queries.csv` | Metrics on 100 sampled test queries |
| `morph_tokenizer/vocab.json` | Token → id mapping (size: 57,029) |
| `morph_tokenizer/synonym_graph.json` | Ede–Ede synonym graph via Vi-pivot (IDF-weighted) |
| `morph_tokenizer/mi_table.json` | Global MI boundary scores (Upgrade A) |
| `morph_tokenizer/tokenizer_config.json` | Special tokens, IDs, hyperparams, model dims |

## Tokenizer

**Morpheme tokenizer v4**:

- Global MI boundary scoring (`is_plausible_boundary` + corpus-wide MI table)
- IDF-weighted Ede–Ede synonym graph via Vi pivot
- Gumbel stochastic segmentation at train, MAP (deterministic) at eval
- Vi side: whitespace passthrough (syllable-level)
- Morpheme-aware vocab: `MIN_FREQ_MORPH=3` for fragments (≤4 chars), `MIN_FREQ_WORD=2` for whole words

## Architecture

- Vanilla Transformer encoder
- `d_model=512`, `n_heads=8`, `n_layers=6`, `d_ff=2048`, `dropout=0.1`
- `max_length=128`
- Special tokens: ['[PAD]', '[UNK]', '[CLS]', '[SEP]', '[MASK]']

## Training pipeline

1. MLM pre-training (Ede + Vi)
2. Cross-lingual alignment (Ede query ↔ Vi query, InfoNCE)
3. Contrastive fine-tuning on (Ede query, Vi passage) pairs

## Reload tokenizer

```python
import json
vocab          = json.load(open("morph_tokenizer/vocab.json"))
synonym_graph  = json.load(open("morph_tokenizer/synonym_graph.json"))
cfg            = json.load(open("morph_tokenizer/tokenizer_config.json"))

# mi_table: decode "word|cut_idx" -> (word, int(cut_idx))
mi_raw = json.load(open("morph_tokenizer/mi_table.json"))
mi_table = {tuple([k.rsplit("|", 1)[0], int(k.rsplit("|", 1)[1])]): v
             for k, v in mi_raw.items()}
```