| # TLHdig 0.2 — Triple-Parallel Corpus Exploitation |
|
|
| **Zenodo 15459134** (TLHdig Beta 0.2) — 315K Hittite transliteration line. |
| **Fotoğraf yok AMA cuneiform Unicode + transliteration + phonetic** üçlü paralel corpus. |
|
|
| ## Keşif |
|
|
| TLHdig 0.2 dataset'i sadece transliteration değil, **üçlü-paralel** corpus: |
|
|
| ``` |
| Cuneiform Unicode: 𒈨𒈾𒄴𒄩𒀭𒁕𒆠𒅖𒊭 |
| Transliteration: me-na-aḫ-ḫa-an-da ki-iš-ša- |
| Phonetic: menaḫḫanta kišša~ |
| ``` |
|
|
| Bu 3 modalite arasında **doğrudan supervised training** yapılabilir — fotoğraf **gerekmeden**. |
|
|
| ## Üretilen corpora (`datasets/processed/tlhdig_corpora/`) |
| |
| | Corpus | Kayıt | Kullanım | |
| |---|---:|---| |
| | `cuneiform_to_transliteration.jsonl` | 315,108 | Seq2seq: Unicode → ABZ transliteration | |
| | `hittite_parallel.jsonl` | **280,624** | ⭐ Hitit-only parallel (gold) | |
| | `transliteration_to_phonetic.jsonl` | 279,168 | Transliteration → Pronunciation | |
| | `cuneiform_lm.jsonl` | 347,015 | Cuneiform Unicode LM | |
| | `transliteration_lm.jsonl` | 341,409 | KenLM 5-gram input | |
| | `phonetic_lm.jsonl` | 279,168 | Phonetic LM | |
| | `language_id.jsonl` | 341,409 | Hit/Akk/Hur/Hat/Luw/Sum classification | |
| | `sign_abz_pairs.jsonl` | 2,742,928 | Cuneiform char frequency + ABZ discovery | |
|
|
| **Hittite cuneiform unique chars**: 337 (ABZ sign repertoire doğrulanabilir). |
|
|
| Top-10 Hitit sign'ı: |
| ``` |
| 𒀭 (AN/DINGIR) 176,832 occurrence |
| 𒀀 (A) 116,937 |
| 𒀸 (AŠ) 76,579 |
| 𒈾 (NA) 76,167 |
| 𒍣 (ZI) 73,829 |
| 𒈠 (MA) 60,485 |
| 𒉡 (NU) 52,674 |
| 𒄿 (I) 51,676 |
| 𒊭 (ŠA) 47,490 |
| 𒅀 (IA) 46,857 |
| ``` |
|
|
| ## 5 Training Signal |
|
|
| ### 1. Cuneiform Unicode → Transliteration (seq2seq) |
| - **Input**: Cuneiform character sequence `𒈨𒈾𒄴...` |
| - **Output**: Transliteration `me-na-aḫ-...` |
| - **Train size**: 280,624 Hitit pairs |
| - **Model**: ByT5 char-to-char (no image required!) |
| - **Kullanım**: |
| - Seq2seq decoder initialization |
| - Image-to-text seq2seq pipeline'ın text-to-text pretraining |
| - Downstream: image tanıma yapılamıyorsa text yoluyla hatasız |
|
|
| ### 2. Transliteration → Phonetic |
| - **Input**: `me-na-aḫ-ḫa-an-da` |
| - **Output**: `menaḫḫanta` |
| - **Train size**: 279,168 |
| - **Kullanım**: Morphological analysis, lemmatization |
|
|
| ### 3. Cuneiform LM (char-level) |
| - **Size**: 315K line × avg 20 char = 6M+ char |
| - **ByT5 / GPT-2 small** ön-eğitim |
| - **Kullanım**: Sign sequence priors (bigram/trigram probabilities) |
|
|
| ### 4. Transliteration LM (KenLM 5-gram) |
| - **Input data**: 341K line transliteration |
| - **Output**: `hitit_kenlm_5gram.binary` |
| - **Kullanım**: Viterbi lattice rescoring (Stage 4, v3/v4) |
|
|
| ### 5. Sign ABZ Discovery |
| - 337 Hitit cuneiform char → ABZ code mapping |
| - Unicode U+12000-U+1254F range |
| - Frequency-weighted: top 100 sign = %95+ coverage |
|
|
| ## Entegrasyon — pipeline v4'te nerede |
|
|
| ### Stage 2 (Classification) desteği |
| - Cuneiform char frequency → class priors |
| - ABZ class label mapping (unified_label) verify |
| |
| ### Stage 4 (Transliteration) ana motor |
| - ByT5 model initialization: Hittite LM pretrain |
| - KenLM 5-gram ready |
| - Phonetic reading output (bonus) |
| |
| ### Seq2seq pipeline (BEYOND_95) |
| - **Pretraining**: 280K cuneiform↔transliteration (text-only) |
| - **Fine-tuning**: 83 Hitit tablet image-text pair (from build_pairs.py) |
| - **Inference**: image → transliteration direct |
| |
| ## Faydalar (realistic gain) |
| |
| | Kullanım | Beklenen gain | Maliyet | |
| |---|---|---| |
| | ByT5 Hitit LM pretrain (char-level) | +2-4% seq2seq | 1 GPU hafta | |
| | KenLM 5-gram Viterbi rescoring | +1-3% (v4 config'de hedef) | 1 CPU gün | |
| | Cuneiform char frequency prior | +0.3-0.5% | 0 | |
| | Phonetic reading side-task | +0.3-0.5% multi-task | marjinal | |
| | Language ID auxiliary | filtering only | 0 | |
| |
| **Toplam realistic**: +3-5% downstream (seq2seq paradigm + LM rescoring dahil). |
| |
| ## Özel değer |
| |
| - **CC-BY-4.0** (free for research + commercial with attribution) |
| - **280K parallel pair** — cuneiform OCR için rekor |
| - **Foto gerekmiyor** — hemen başlatılabilir |
| - **Würzburg authoritative** (Schwemer grubu) |
| - **98%+ published Hittite fragments** coverage |
| |
| ## Hemen yapılabilecek (veri bizde) |
| |
| ```bash |
| # 1. KenLM 5-gram Hitit LM |
| python3 -c " |
| import json |
| with open('datasets/processed/tlhdig_corpora/transliteration_lm.jsonl') as f, \ |
| open('/tmp/hitit_translit.txt', 'w') as g: |
| for line in f: |
| r = json.loads(line) |
| if r.get('lang') == 'Hit': |
| g.write(r['text'] + '\n') |
| " |
| # Pip install via GitHub: pip install https://github.com/kpu/kenlm/archive/master.zip |
| # kenlm/bin/lmplz -o 5 < /tmp/hitit_translit.txt > hitit.arpa |
| # kenlm/bin/build_binary hitit.arpa hitit_kenlm_5gram.binary |
| |
| # 2. ByT5 pretrain — 280K cuneiform→translit |
| # Training script: hitit_ocr/src/seq2seq/train_text_only.py (TODO) |
| |
| # 3. Cuneiform char embedding (supervised) |
| # Each of 337 unique cuneiform chars → embedding via usage context |
| ``` |
| |
| ## Status |
| |
| - ✅ TLHdig 0.2 zaten indirildi (datasets/sources/tlhdig/) |
| - ✅ 8 corpus türetildi (datasets/processed/tlhdig_corpora/) |
| - ✅ Hitit parallel 280K kayıt hazır |
| - ⏳ ByT5 pretraining script TODO |
| - ⏳ KenLM binary generation TODO (pip install kenlm) |
|
|
| ## Referans |
|
|
| - [Zenodo TLHdig 0.2](https://zenodo.org/records/15459134) |
| - License: CC-BY-4.0 |
| - Authors: Müller, Prechel, Rieken, Schwemer (Würzburg/Mainz/Marburg) |
| - Web: https://www.hethport.uni-wuerzburg.de/TLHdig/ |
|
|
