Hugging Face's logo

savastakan
/
hitit-cuneiform-ocr

Image Classification
PyTorch
Turkish
English
object-detection
ocr
cuneiform
hittite
archaeology
digital-humanities
dinov3
convnextv2
siglip
eva02
yolo
Model card Files
xet
 Community
hitit-cuneiform-ocr / code /TLHDIG_EXPLOITATION.md
savastakan's picture
savastakan
Initial upload: code + 5 record checkpoints + fuse
f211247 verified
preview code
|
Raw
History Blame Contribute Delete
5.45 kB

TLHdig 0.2 — Triple-Parallel Corpus Exploitation

Zenodo 15459134 (TLHdig Beta 0.2) — 315K Hittite transliteration line. Fotoğraf yok AMA cuneiform Unicode + transliteration + phonetic üçlü paralel corpus.

Keşif

TLHdig 0.2 dataset'i sadece transliteration değil, üçlü-paralel corpus:

Cuneiform Unicode:    𒈨𒈾𒄴𒄩𒀭𒁕𒆠𒅖𒊭
Transliteration:      me-na-aḫ-ḫa-an-da ki-iš-ša-
Phonetic:             menaḫḫanta kišša~

Bu 3 modalite arasında doğrudan supervised training yapılabilir — fotoğraf gerekmeden.

Üretilen corpora (datasets/processed/tlhdig_corpora/)

Corpus Kayıt Kullanım
cuneiform_to_transliteration.jsonl 315,108 Seq2seq: Unicode → ABZ transliteration
hittite_parallel.jsonl 280,624 ⭐ Hitit-only parallel (gold)
transliteration_to_phonetic.jsonl 279,168 Transliteration → Pronunciation
cuneiform_lm.jsonl 347,015 Cuneiform Unicode LM
transliteration_lm.jsonl 341,409 KenLM 5-gram input
phonetic_lm.jsonl 279,168 Phonetic LM
language_id.jsonl 341,409 Hit/Akk/Hur/Hat/Luw/Sum classification
sign_abz_pairs.jsonl 2,742,928 Cuneiform char frequency + ABZ discovery

Hittite cuneiform unique chars: 337 (ABZ sign repertoire doğrulanabilir).

Top-10 Hitit sign'ı:

𒀭 (AN/DINGIR)     176,832 occurrence
𒀀 (A)             116,937
𒀸 (AŠ)             76,579
𒈾 (NA)             76,167
𒍣 (ZI)             73,829
𒈠 (MA)             60,485
𒉡 (NU)             52,674
𒄿 (I)              51,676
𒊭 (ŠA)             47,490
𒅀 (IA)             46,857

5 Training Signal

1. Cuneiform Unicode → Transliteration (seq2seq)

  • Input: Cuneiform character sequence 𒈨𒈾𒄴...
  • Output: Transliteration me-na-aḫ-...
  • Train size: 280,624 Hitit pairs
  • Model: ByT5 char-to-char (no image required!)
  • Kullanım:
    • Seq2seq decoder initialization
    • Image-to-text seq2seq pipeline'ın text-to-text pretraining
    • Downstream: image tanıma yapılamıyorsa text yoluyla hatasız

2. Transliteration → Phonetic

  • Input: me-na-aḫ-ḫa-an-da
  • Output: menaḫḫanta
  • Train size: 279,168
  • Kullanım: Morphological analysis, lemmatization

3. Cuneiform LM (char-level)

  • Size: 315K line × avg 20 char = 6M+ char
  • ByT5 / GPT-2 small ön-eğitim
  • Kullanım: Sign sequence priors (bigram/trigram probabilities)

4. Transliteration LM (KenLM 5-gram)

  • Input data: 341K line transliteration
  • Output: hitit_kenlm_5gram.binary
  • Kullanım: Viterbi lattice rescoring (Stage 4, v3/v4)

5. Sign ABZ Discovery

  • 337 Hitit cuneiform char → ABZ code mapping
  • Unicode U+12000-U+1254F range
  • Frequency-weighted: top 100 sign = %95+ coverage

Entegrasyon — pipeline v4'te nerede

Stage 2 (Classification) desteği

  • Cuneiform char frequency → class priors
  • ABZ class label mapping (unified_label) verify

Stage 4 (Transliteration) ana motor

  • ByT5 model initialization: Hittite LM pretrain
  • KenLM 5-gram ready
  • Phonetic reading output (bonus)

Seq2seq pipeline (BEYOND_95)

  • Pretraining: 280K cuneiform↔transliteration (text-only)
  • Fine-tuning: 83 Hitit tablet image-text pair (from build_pairs.py)
  • Inference: image → transliteration direct

Faydalar (realistic gain)

Kullanım Beklenen gain Maliyet
ByT5 Hitit LM pretrain (char-level) +2-4% seq2seq 1 GPU hafta
KenLM 5-gram Viterbi rescoring +1-3% (v4 config'de hedef) 1 CPU gün
Cuneiform char frequency prior +0.3-0.5% 0
Phonetic reading side-task +0.3-0.5% multi-task marjinal
Language ID auxiliary filtering only 0

Toplam realistic: +3-5% downstream (seq2seq paradigm + LM rescoring dahil).

Özel değer

  • CC-BY-4.0 (free for research + commercial with attribution)
  • 280K parallel pair — cuneiform OCR için rekor
  • Foto gerekmiyor — hemen başlatılabilir
  • Würzburg authoritative (Schwemer grubu)
  • 98%+ published Hittite fragments coverage

Hemen yapılabilecek (veri bizde) 

# 1. KenLM 5-gram Hitit LM
python3 -c "
import json
with open('datasets/processed/tlhdig_corpora/transliteration_lm.jsonl') as f, \
     open('/tmp/hitit_translit.txt', 'w') as g:
    for line in f:
        r = json.loads(line)
        if r.get('lang') == 'Hit':
            g.write(r['text'] + '\n')
"
# Pip install via GitHub: pip install https://github.com/kpu/kenlm/archive/master.zip
# kenlm/bin/lmplz -o 5 < /tmp/hitit_translit.txt > hitit.arpa
# kenlm/bin/build_binary hitit.arpa hitit_kenlm_5gram.binary

# 2. ByT5 pretrain — 280K cuneiform→translit
# Training script: hitit_ocr/src/seq2seq/train_text_only.py (TODO)

# 3. Cuneiform char embedding (supervised)
# Each of 337 unique cuneiform chars → embedding via usage context

Status

  • ✅ TLHdig 0.2 zaten indirildi (datasets/sources/tlhdig/)
  • ✅ 8 corpus türetildi (datasets/processed/tlhdig_corpora/)
  • ✅ Hitit parallel 280K kayıt hazır
  • ⏳ ByT5 pretraining script TODO
  • ⏳ KenLM binary generation TODO (pip install kenlm)

Referans