Morpheus-TR-50K

Morpheus is a neural morpheme-aware tokenizer and word embedder for Turkish — an agglutinative language whose semantic content is densely packed into productive suffix chains. It combines unsupervised morphological supervision (Morfessor) with self-supervised objectives (SGNS, contrastive, MLM) to learn segmentations that are simultaneously morphologically aligned and language-modeling-friendly. Because it is neural, the same forward pass that tokenizes also yields a structured word embedding — so Morpheus is a tokenizer and an embedding model at once.

evlerimizdekiler  →  ev | ler | imiz | de | ki | ler
                     root  PL   POSS   LOC  REL  PL
                    ("the ones in our houses")

Where classical BPE/WordPiece fragment morphologically rich Turkish words into statistically convenient but linguistically opaque subwords, Morpheus produces interpretable, surface-preserving morpheme-level segmentations — and its decode is exactly invertible: decode(encode(w)) == w holds by construction.

Headline Results

Morpheus is the only lossless, morphology-aware tokenizer for Turkish that is usable in a generative LLM. Among reversible tokenizers (the only ones valid for generation), it achieves the lowest BPC, the highest frequency-weighted token purity, the strongest morphological alignment, and ~19% lower GPU memory than 64K-vocab subword tokenizers — while also producing structured word embeddings for free.

The two tokenizers that appear to beat it buy those numbers with information loss, which disqualifies them for generation:

Tokenizer	Roundtrip decode(encode(w))==w	Valid for generation?
Morpheus	100.0%	✓
BPE / ByteBPE / Unigram	100.0%	✓
TurkishTokenizer	95.4% (lossy canonical decode)	✗
WordPiece	58.2% (strips ç/ğ/ı/ö/ş/ü)	✗

Restricted to the reversible subset, Morpheus leads where it matters:

Metric	Morpheus	Best reversible baseline
BPC ↓ (equal 10K steps)	1.425	1.436 (BPE)
MorphScore macro-F1 ↑ (UD gold)	0.61	~0.32 (subwords)
TR-MMLU %Pure (freq-weighted) ↑	83.5	~49 (subwords)
Peak GPU memory (B=32 gen) ↓	~3,020 MB	3,723 MB (64K subword)
Structured root-family embeddings	✓	✗

Morpheus as a word embedder

The same forward pass that tokenizes emits a 320-dim word embedding. Evaluated frozen against BERTurk (768-d) and the multilingual retriever BGE-M3 (1024-d), the picture splits cleanly by task character — Morpheus dominates lexical / root-level tasks, the heavier contextual encoders lead on context- and inflection-dependent tasks:

Task	Morpheus (320)	BERTurk (768)	BGE-M3 (1024)
Root-family retrieval (MAP ↑)	0.85	0.49	0.80
Same-root verification (ROC-AUC ↑)	1.00	0.70	0.98
Number probing (acc ↑)	0.59	0.95	0.91
Case probing (acc ↑)	0.22	0.89	0.81
WikiANN-tr NER (macro-F1 ↑)	0.48	0.79	0.76

This is a deliberate, architectural trade-off: the root-identity contrastive objective pulls a root's inflections together — sharpening root geometry (hence the retrieval/dedup wins) while collapsing the inflectional contrasts a probe reads — and the static per-word vector lacks the sentence context NER needs. Morpheus is therefore complementary to contextual encoders: ideal for the lexical index of a multi-vector RAG system, paired with a dense semantic encoder for context.

Model Details

• Developed by: Tolga Şakar
• Model type: Neural morpheme-aware tokenizer + word embedder (CharEncoder → BoundaryDetector → SegmentEncoder)
• Language(s): Turkish (tr)
• License: MIT
• Trained from: Scratch (no base model)
• Vocabulary size: 50,000 morpheme-level tokens
• Word embedding dimension: 320

Repository

• Code: TurkishMorpheus
• Corpus collection tooling: CorpusCollector
• Paper: Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

Usage

Morpheus is a neural tokenizer — it requires both a vocab file and a trained PyTorch model checkpoint. Standard AutoTokenizer.from_pretrained() does not apply.

Install

git clone https://github.com/lonewolf-rd/TurkishMorpheus.git
cd TurkishMorpheus
pip install -e .

Download artifacts from this repo

from huggingface_hub import snapshot_download

local_dir = snapshot_download(repo_id="lonewolflab/Morpheus-TR-50K")
# Downloads:
#   - morpheus_50k/vocab.json
#   - morpheus_50k/tokenizer_config.json
#   - turkish_morpheus_a100_v3_best.pt

Load and tokenize

import torch, sys
from src.model_development.model.morpheus import Morpheus
from src.model_development.training.trainer import TrainingConfig
from src.model_development.tokenization.morpheus_tokenizer import MorpheusTokenizer

# TrainingConfig pickle workaround (checkpoint contains the config object)
sys.modules["__main__"].TrainingConfig = TrainingConfig

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

ckpt = torch.load(f"{local_dir}/turkish_morpheus_a100_v3_best.pt",
                  map_location=device, weights_only=False)
cfg = ckpt["config"]
morpheus = Morpheus(
    char_dim=cfg.char_dim, char_embed_dim=cfg.char_embed_dim,
    case_embed_dim=cfg.case_embed_dim,
    n_layers_encoder=cfg.n_layers_encoder,
    n_layers_detector=cfg.n_layers_detector,
    num_heads=cfg.num_heads, max_word_len=cfg.max_word_len,
    max_segs=cfg.max_segs, dropout=cfg.dropout,
    threshold=cfg.threshold, pos_weight=cfg.pos_weight,
    count_loss_w=getattr(cfg, "count_loss_w", 0.3),
)
morpheus.load_state_dict(ckpt["model_state"])
morpheus.to(device).eval()

tokenizer = MorpheusTokenizer.load(
    f"{local_dir}/morpheus_50k", morpheus_model=morpheus, device=device,
)

text = "evlerimizdekiler kitapçısı muvaffakiyetsizleştiriciler"
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text, add_special_tokens=False)
decoded = tokenizer.decode(ids)
assert decoded == text   # lossless by construction

Word embeddings

# Morpheus produces a 320-dim embedding per word from the same forward pass
words = ["kitap", "kitaplar"]
ids, flags, lens = [], [], []
for w in words:
    i, f, rl = tokenizer.helper.word_to_char_ids(w, max_len=32)
    ids.append(i); flags.append(f); lens.append(rl)
out = morpheus(
    char_ids=torch.tensor(ids, device=device),
    case_flags=torch.tensor(flags, device=device),
    real_lengths=torch.tensor(lens, device=device),
)
emb = out["word_embeddings"]   # (2, 320)

Architecture

char_ids, case_flags  (B=batch, L=32 max word chars)
        │
        ▼
   CharEncoder              Multi-scale CNN + 3 × RoPE self-attention
        │                   Output: (B, L, 320) context-aware char vectors
        ▼
  BoundaryDetector          4 × RoPE attention with adjacent-pair scoring
        │                   Deep-supervised aux loss vs Morfessor labels
        │                   Output: boundary_probs ∈ [0,1] of shape (B, L−1)
        ▼
  SegmentEncoder            Poisson-binomial DP → soft segment membership
        │                   Char-level attention pooling per segment
        │                   Mean over valid segments + FFN + LayerNorm
        ▼
   word_embedding ∈ ℝ³²⁰

Soft segmentation via Poisson-binomial dynamic programming: given boundary probabilities p_i ∈ [0,1] between adjacent characters, the probability that character i belongs to segment k is computed via:

f[i, k] = f[i−1, k] · (1 − p_i) + f[i−1, k−1] · p_i

This gives a differentiable membership matrix that converges to one-hot segment assignments as p_i → {0,1}, recovering hard segmentation at inference without an architectural switch. Because the operator only groups characters (never rewrites them), segmentation is surface-preserving and exactly invertible.

Training Data

Trained on a large-scale monolingual Turkish corpus combining a multi-register author corpus with the full cleaned Turkish Wikipedia (~10 GB raw text), covering four registers:

• Ekşisözlük — informal/colloquial Turkish (rich morphological constructs)
• Dergipark — academic/formal Turkish (diverse derivational morphology)
• Turkish news sites — standard journalistic register
• Turkish Wikipedia — encyclopedic, broad vocabulary (aggressively cleaned/filtered + dedup)

The web-sourced registers were collected with the companion repository CorpusCollector, which documents source URLs, scraping protocol, rate-limiting, and per-source preprocessing for full reproducibility. Split: 95% train / 5% test, fixed seed.

Training Procedure

Hyperparameters

Parameter	Value
Epochs	10
Effective batch size	512 (256 × grad_accum 2)
Optimizer	AdamW
Learning rate	cosine-decayed
Char dim	320
Encoder / Detector layers	3 / 4
Attention heads	5
Max word len	32
Max segments per word	12
Dropout	0.1
Mixed precision	TF32 (matmul + cuDNN); loss in FP32; AMP off

Loss

L = w_aux · L_aux + w_sgns · L_sgns + w_ctr · L_contrastive + w_mlm · L_mlm

Component	Role	Weight schedule
L_aux	Boundary BCE + count MSE vs Morfessor (root-corrected, deep-supervised)	Decays 0.50 → 0.08 over 10 epochs
L_sgns	Skip-gram with 16 negatives, ±6 window, 120K context vocab	Constant 0.7
L_ctr	InfoNCE on root identity, temperature 0.10	Constant 0.3
L_mlm	Char autoregressive reconstruction of masked words (20% mask rate)	Constant 1.0

The aux schedule realizes a curriculum: early epochs anchor on Morfessor; as it decays, distributional signals (SGNS, MLM) take over to shape semantic geometry, so the model becomes teacher-free and generalizes to OOV.

Hardware

• Single NVIDIA A100 80GB, ~30 min/epoch × 10 epochs ≈ 5 hours total.

Evaluation

Reversibility (the generation gate)

decode(encode(w)) == w over 30,204 inflected wordforms: Morpheus 100%, BPE/ByteBPE/Unigram 100%, TurkishTokenizer 95.4% (lossy canonicalization), WordPiece 58.2% (diacritic stripping).

Gold morphology — MorphScore (UD_Turkish-Kenet, 30K words)

Model	Macro-F1
TurkishTokenizer (lossy)	0.648
Morpheus	0.608
Morfessor	0.589
BPE / Unigram / ByteBPE	~0.32
WordPiece (lossy)	0.270

Intrinsic — morphological alignment (stratified test set)

Stratum	n	Boundary F1 vs Morfessor	MAS %
seen	800	0.611	67.5
oov	400	0.715	69.0
curated_oov	10	0.742	74.2
nonce	5	0.625	55.6

Downstream language modeling — BPC (param-equalized 58M GPT, equal 10K steps)

Tokenizer	vocab	BPC ↓	Fertility (tok/word)
Morpheus	50K	1.425	1.73
BPE	64K	1.436	1.51
Unigram	64K	1.437	1.52
TurkishTokenizer (lossy)	33K	1.442	1.98
Morfessor	29K	1.446	1.91
ByteBPE	64K	1.449	1.53
WordPiece (lossy)	64K	1.384	1.39

Among reversible tokenizers Morpheus has the lowest BPC. WordPiece's lower raw value is an artifact of modeling diacritic-stripped, lower-entropy text.

Inference / efficiency

Tokenizer	Gen char/s (B=32)	Peak GPU mem (B=32)
Morpheus	14,444	~3,020 MB
64K subword	~22,000	3,723 MB

Morpheus uses ~19% less GPU memory; its higher fertility lowers raw generation throughput — the deliberate trade-off of morpheme-level tokenization.

Intended Uses

Recommended

• Turkish NLU / classification (morpheme-aligned tokens aid attention specialization)
• Lexical / keyword retrieval, stemming, morphological dedup — and the lexical index of a multi-vector RAG system
• Pretraining small-to-medium Turkish language models where faithful decoding and morphology matter
• Linguistic research / corpus annotation at scale; educational morphology tools
• Memory-constrained inference (~19% lower GPU memory vs 64K-vocab tokenizers)

Not recommended

• Real-time generation where latency dominates (~1.6× slower char throughput than BPE)
• A drop-in replacement for a contextual encoder on NER or inflectional-feature tasks — pair Morpheus with BERTurk/BGE-M3 there
• Multilingual models (Turkish-specific by design)

Limitations

• Single language: trained for Turkish only.
• Slightly Higher fertility (~1.73 vs subword ~1.5 tokens/word) → longer sequences and lower raw generation throughput.
• Root-centric embedding: excels at lexical retrieval/dedup but underperforms contextual encoders on number/case probing and NER — by architectural design, not a bug.

Citation

@misc{sakar2026morpheus,
  title  = {Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish},
  author = {Şakar, Tolga},
  year   = {2026},
  note   = {Preprint forthcoming on arXiv}
}

Acknowledgments

• Morfessor (Creutz & Lagus, 2002, 2007) — unsupervised morphological supervisor
• SentencePiece (Kudo & Richardson, 2018) and HuggingFace tokenizers — baseline implementations
• SIGMORPHON 2022 Turkish task — inflection gold standard
• BERTurk and BGE-M3 — embedding baselines
• The Turkish NLP community (BERTurk, TURNA, Zemberek, TRMorph) that motivated this study

License

MIT License. See LICENSE.

---

Repository: TurkishMorpheus · Author: Tolga Şakar