# TLHdig 0.2 — Triple-Parallel Corpus Exploitation

**Zenodo 15459134** (TLHdig Beta 0.2) — 315K Hittite transliteration line.
**Fotoğraf yok AMA cuneiform Unicode + transliteration + phonetic** üçlü paralel corpus.

## Keşif

TLHdig 0.2 dataset'i sadece transliteration değil, **üçlü-paralel** corpus:

```
Cuneiform Unicode:    𒈨𒈾𒄴𒄩𒀭𒁕𒆠𒅖𒊭
Transliteration:      me-na-aḫ-ḫa-an-da ki-iš-ša-
Phonetic:             menaḫḫanta kišša~
```

Bu 3 modalite arasında **doğrudan supervised training** yapılabilir — fotoğraf **gerekmeden**.

## Üretilen corpora (`datasets/processed/tlhdig_corpora/`)

| Corpus | Kayıt | Kullanım |
|---|---:|---|
| `cuneiform_to_transliteration.jsonl` | 315,108 | Seq2seq: Unicode → ABZ transliteration |
| `hittite_parallel.jsonl` | **280,624** | ⭐ Hitit-only parallel (gold) |
| `transliteration_to_phonetic.jsonl` | 279,168 | Transliteration → Pronunciation |
| `cuneiform_lm.jsonl` | 347,015 | Cuneiform Unicode LM |
| `transliteration_lm.jsonl` | 341,409 | KenLM 5-gram input |
| `phonetic_lm.jsonl` | 279,168 | Phonetic LM |
| `language_id.jsonl` | 341,409 | Hit/Akk/Hur/Hat/Luw/Sum classification |
| `sign_abz_pairs.jsonl` | 2,742,928 | Cuneiform char frequency + ABZ discovery |

**Hittite cuneiform unique chars**: 337 (ABZ sign repertoire doğrulanabilir).

Top-10 Hitit sign'ı:
```
𒀭 (AN/DINGIR)     176,832 occurrence
𒀀 (A)             116,937
𒀸 (AŠ)             76,579
𒈾 (NA)             76,167
𒍣 (ZI)             73,829
𒈠 (MA)             60,485
𒉡 (NU)             52,674
𒄿 (I)              51,676
𒊭 (ŠA)             47,490
𒅀 (IA)             46,857
```

## 5 Training Signal

### 1. Cuneiform Unicode → Transliteration (seq2seq)
- **Input**: Cuneiform character sequence `𒈨𒈾𒄴...`
- **Output**: Transliteration `me-na-aḫ-...`
- **Train size**: 280,624 Hitit pairs
- **Model**: ByT5 char-to-char (no image required!)
- **Kullanım**: 
  - Seq2seq decoder initialization
  - Image-to-text seq2seq pipeline'ın text-to-text pretraining
  - Downstream: image tanıma yapılamıyorsa text yoluyla hatasız

### 2. Transliteration → Phonetic
- **Input**: `me-na-aḫ-ḫa-an-da`
- **Output**: `menaḫḫanta`
- **Train size**: 279,168
- **Kullanım**: Morphological analysis, lemmatization

### 3. Cuneiform LM (char-level)
- **Size**: 315K line × avg 20 char = 6M+ char
- **ByT5 / GPT-2 small** ön-eğitim
- **Kullanım**: Sign sequence priors (bigram/trigram probabilities)

### 4. Transliteration LM (KenLM 5-gram)
- **Input data**: 341K line transliteration
- **Output**: `hitit_kenlm_5gram.binary`
- **Kullanım**: Viterbi lattice rescoring (Stage 4, v3/v4)

### 5. Sign ABZ Discovery
- 337 Hitit cuneiform char → ABZ code mapping
- Unicode U+12000-U+1254F range
- Frequency-weighted: top 100 sign = %95+ coverage

## Entegrasyon — pipeline v4'te nerede

### Stage 2 (Classification) desteği
- Cuneiform char frequency → class priors
- ABZ class label mapping (unified_label) verify

### Stage 4 (Transliteration) ana motor
- ByT5 model initialization: Hittite LM pretrain
- KenLM 5-gram ready
- Phonetic reading output (bonus)

### Seq2seq pipeline (BEYOND_95)
- **Pretraining**: 280K cuneiform↔transliteration (text-only)
- **Fine-tuning**: 83 Hitit tablet image-text pair (from build_pairs.py)
- **Inference**: image → transliteration direct

## Faydalar (realistic gain)

| Kullanım | Beklenen gain | Maliyet |
|---|---|---|
| ByT5 Hitit LM pretrain (char-level) | +2-4% seq2seq | 1 GPU hafta |
| KenLM 5-gram Viterbi rescoring | +1-3% (v4 config'de hedef) | 1 CPU gün |
| Cuneiform char frequency prior | +0.3-0.5% | 0 |
| Phonetic reading side-task | +0.3-0.5% multi-task | marjinal |
| Language ID auxiliary | filtering only | 0 |

**Toplam realistic**: +3-5% downstream (seq2seq paradigm + LM rescoring dahil).

## Özel değer

- **CC-BY-4.0** (free for research + commercial with attribution)
- **280K parallel pair** — cuneiform OCR için rekor
- **Foto gerekmiyor** — hemen başlatılabilir
- **Würzburg authoritative** (Schwemer grubu)
- **98%+ published Hittite fragments** coverage

## Hemen yapılabilecek (veri bizde)

```bash
# 1. KenLM 5-gram Hitit LM
python3 -c "
import json
with open('datasets/processed/tlhdig_corpora/transliteration_lm.jsonl') as f, \
     open('/tmp/hitit_translit.txt', 'w') as g:
    for line in f:
        r = json.loads(line)
        if r.get('lang') == 'Hit':
            g.write(r['text'] + '\n')
"
# Pip install via GitHub: pip install https://github.com/kpu/kenlm/archive/master.zip
# kenlm/bin/lmplz -o 5 < /tmp/hitit_translit.txt > hitit.arpa
# kenlm/bin/build_binary hitit.arpa hitit_kenlm_5gram.binary

# 2. ByT5 pretrain — 280K cuneiform→translit
# Training script: hitit_ocr/src/seq2seq/train_text_only.py (TODO)

# 3. Cuneiform char embedding (supervised)
# Each of 337 unique cuneiform chars → embedding via usage context
```

## Status

- ✅ TLHdig 0.2 zaten indirildi (datasets/sources/tlhdig/)
- ✅ 8 corpus türetildi (datasets/processed/tlhdig_corpora/)
- ✅ Hitit parallel 280K kayıt hazır
- ⏳ ByT5 pretraining script TODO
- ⏳ KenLM binary generation TODO (pip install kenlm)

## Referans

- [Zenodo TLHdig 0.2](https://zenodo.org/records/15459134)
- License: CC-BY-4.0
- Authors: Müller, Prechel, Rieken, Schwemer (Würzburg/Mainz/Marburg)
- Web: https://www.hethport.uni-wuerzburg.de/TLHdig/