HITIT-OCR v2 — %90+ Top-1 Accuracy Pipeline
Hedef: 198-class Hitit gold subset (3253 sample, in_gold_eval=True) üzerinde top-1 accuracy ≥ 0.90.
Realistik hedef tanımı
⚠️ Önemli: Literatürde 3300 sınıf macro-F1 %90 raporlanmadı — ulaşılamaz.
Gerçekçi:
- Primary target: 198-class Hitit gold top-1 ≥ 0.90 (no rejection, pure)
- Enhanced target: Rejection@τ=0.6 ile selective accuracy ≥ 0.94, coverage ≥ 0.85
- Full 3300 macro-F1: 0.60-0.65 (tier-stratified)
Baseline (v1 pipeline)
- DINOv3+LoRA+Hierarchical+CB-Focal→LDAM → tahmini top-1 ~%77-80 (DeepScribe extrapolated)
- Gap to target: +%10-13 puan
%90'a ulaştıran SOTA teknikler
Gain tablosu (198-class Hitit gold)
| # | Teknik | Beklenen Gain | Maliyet | Paper |
|---|---|---|---|---|
| 1 | EMA + SWA averaging | +1.0-1.5% | ~0 | Izmailov 2018 |
| 2 | Label smoothing ε=0.1 | +0.5-1.0% | ~0 | Szegedy 2016 |
| 3 | Multi-scale TTA (224+320+384+448) | +3-5% | 4× infer | Shanmugam CVPR21 |
| 4 | 3-model ensemble (DINOv3+ConvNeXt-V2+SigLIP2) | +4-7% | 3× train | Kaggle consensus |
| 5 | Knowledge Distillation (ensemble→single) | +1-2% tail +5-8% | +1× train | DIST NeurIPS22 |
| 6 | Pseudo-labeling self-training | +2-4% tail | 3 iteration | FixMatch |
| 7 | TLHdig LM rescoring (Viterbi) | +3-5% | LM 1 gün | Gordin 2020 |
| 8 | Rejection @τ=0.6 (selective acc) | +3-6% | 0 | Geifman 2017 |
| 9 | Label cleanlab noise detection | +1-2% | cleanlab | Northcutt 2021 |
| 10 | Tablet-level context aggregation | +1-3% | 1× train | Yuan ICDAR21 |
| 11 | Active learning (5K re-annotate) | +2-4% | manual | Gal 2017 |
| 12 | Synthetic rare class generation | +3-5% tail | 500 GPU-h | DeepAramaic |
| 13 | Paleographic style conditioning (FiLM) | +1-2% cross-source | minor | — |
| 14 | Hierarchical fallback (L2→L3) | +2-4% | 0 | Zhu ICCV23 |
| TOPLAM (overlap dedüksiyon sonrası) | +12-18% | — | — |
Baseline 77% + 15% average gain = ~92% (güvenli tahmin).
Güncellenen Mimari
┌─────────────────────────────────────────────────────────────────────┐
│ INPUT: Tablet image │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 0 — Preprocessing (v1'den aynı + label cleanlab) │
│ + Label noise detection (cleanlab cross-val confident learning) │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 1 — Detection (v1'den aynı) │
│ YOLO11-P2 + SAHI │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 2 — Classification (ENHANCED) │
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌────────────────┐ │
│ │ DINOv3-L/14 │ │ ConvNeXt-V2-L │ │ SigLIP2-SO400M │ │
│ │ +LoRA+Hier │ │ +Hier │ │ +Hier │ │
│ │ (w=0.45) │ │ (w=0.30) │ │ (w=0.25) │ │
│ └───────┬───────┘ └───────┬───────┘ └────────┬───────┘ │
│ └──────────────────┼────────────────────┘ │
│ ▼ │
│ Weighted logit average │
│ ▼ │
│ EMA + SWA checkpoint averaging │
│ ▼ │
│ Multi-scale TTA (224+320+384+448) │
│ ▼ │
│ FiLM paleographic style conditioning │
│ ▼ │
│ Knowledge Distillation (→single DINOv3-L) │
│ ▼ │
│ Pseudo-label iteration (3×) │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 3 — Line Assembly + OOD + Rejection │
│ + Tablet-level context aggregation (graph pooling) │
│ + Selective classification rejection (τ=0.6) │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 4 — Transliteration (ENHANCED) │
│ ByT5 + Viterbi + TLHdig 3-gram LM fusion │
│ + Hierarchical L2 (25 paleographic clusters) fallback │
└─────────────────────────────────────────────────────────────────────┘
Stage 2 — Detaylı (en çok değişen)
Ensemble: 3 backbone
| Model | Params | Role | Weight | Eğitim süresi |
|---|---|---|---|---|
| DINOv3-L/14 | 300M | Primary SSL | 0.45 | 800 GPU-h |
| ConvNeXt-V2-Large | 198M | CNN inductive bias | 0.30 | 400 GPU-h |
| SigLIP2-SO400M | 400M | Multilingual symbol | 0.25 | 300 GPU-h |
Ensemble formülü:
logits_final = 0.45 * logits_dino + 0.30 * logits_convnext + 0.25 * logits_siglip
# weights val set ile fit — Bayesian optimization
EMA + SWA
# Son %25 epoch'ta EMA decay 0.9999
from torch.optim.swa_utils import AveragedModel, SWALR
model = base_model
swa_model = AveragedModel(model)
swa_start_epoch = 60 # 80 total, son 20 epoch SWA
swa_scheduler = SWALR(optimizer, swa_lr=5e-5)
for epoch in range(80):
train_epoch(model)
if epoch >= swa_start_epoch:
swa_model.update_parameters(model)
swa_scheduler.step()
# Checkpoint averaging sonrası BN update
torch.optim.swa_utils.update_bn(train_loader, swa_model)
Multi-scale TTA
def tta_predict(model, image):
scales = [224, 320, 384, 448]
augments = [
lambda x: x, # identity
lambda x: F.adjust_gamma(x, 0.9), # gamma darker
lambda x: F.adjust_gamma(x, 1.1), # gamma brighter
lambda x: TF.rotate(x, angle=3), # +3° rotation
lambda x: TF.rotate(x, angle=-3), # -3° rotation
]
# NO horizontal flip (cuneiform asymmetric)
all_logits = []
for scale in scales:
resized = F.resize(image, (scale, scale))
for aug in augments:
aug_img = aug(resized)
logits = model(aug_img.unsqueeze(0))
all_logits.append(F.softmax(logits, dim=-1))
# Average over 20 augmented versions
return torch.stack(all_logits).mean(0)
Knowledge Distillation (ensemble → single DINOv3-L)
# Distillation config
teacher: ensemble (DINOv3-L + ConvNeXt-V2 + SigLIP2)
student: DINOv3-L/14
loss:
soft_target_weight: 0.7
hard_target_weight: 0.3
temperature: 4.0
method: DIST # Pearson correlation (NeurIPS 2022)
epochs: 30
lr: 1e-4
Son deployment tek DINOv3-L — inference 3× hızlı.
Pseudo-labeling (3 iteration self-training)
Iteration 1:
- Ensemble'la unlabeled hitit_local + ebl_ocr üzerinde predict
- confidence > 0.90 (head/mid), > 0.75 (tail) → pseudo-label
- Training set'e ekle (15K yeni sample expected)
- Re-train classifier
Iteration 2-3:
- Önceki pseudo-labels içinde cleanlab noise detection
- Remove noisy, add new confident → iterate
- Typical convergence 3 iterations
Label noise cleaning
# cleanlab ile label noise detection
from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues
# 5-fold cross-val predictions
pred_probs = cross_val_predict(classifier, X, y, cv=5, method='predict_proba')
label_issues = find_label_issues(
labels=y,
pred_probs=pred_probs,
filter_by='both',
return_indices_ranked_by='self_confidence'
)
# Beklenen: %3-8 noise found
# Strategy: top-50% noise → remove, diğerleri → relabel
Paleographic style conditioning (FiLM)
class FiLMClassifier(nn.Module):
def __init__(self, backbone, n_classes, n_styles=11):
self.backbone = backbone
self.style_embed = nn.Embedding(n_styles, 256)
self.film_gamma = nn.Linear(256, 768)
self.film_beta = nn.Linear(256, 768)
self.head = nn.Linear(768, n_classes)
def forward(self, x, source_id):
feat = self.backbone(x) # (B, 768)
style = self.style_embed(source_id)
gamma = self.film_gamma(style)
beta = self.film_beta(style)
feat = gamma * feat + beta # style modulation
return self.head(feat)
11 source_id: hitit_local, deepscribe, ebl, ebl_v2, maicubeda, heicubeda, cuneiml, compvis, old_babylonian_zip, transliterated_fragments, yeni_veri.
Stage 3 — Rejection + Tablet context
Rejection (selective classification)
def predict_with_rejection(model, x, threshold=0.6):
logits = model(x)
probs = F.softmax(logits, dim=-1)
max_prob, pred = probs.max(dim=-1)
if max_prob < threshold:
return "REJECT", max_prob
return pred, max_prob
# Evaluation
n_correct_non_reject = sum(1 for p, y, r in predictions
if p == y and r != "REJECT")
n_non_reject = sum(1 for _, _, r in predictions if r != "REJECT")
selective_accuracy = n_correct_non_reject / n_non_reject
coverage = n_non_reject / total
Hedef: selective_accuracy ≥ 0.94 at coverage ≥ 0.85
Tablet-level context aggregation
# Graph pooling: aynı tablet_id'deki bbox'lar arasında attention
class TabletContextAggregator(nn.Module):
def __init__(self, feat_dim=768):
self.attention = nn.MultiheadAttention(feat_dim, num_heads=8)
self.proj = nn.Linear(feat_dim, feat_dim)
def forward(self, bbox_feats, tablet_mask):
# bbox_feats: (N_bbox, 768)
# tablet_mask: attention only within same tablet
attn_out, _ = self.attention(bbox_feats, bbox_feats, bbox_feats,
attn_mask=tablet_mask)
return self.proj(attn_out + bbox_feats) # residual
Stage 4 — TLHdig LM rescoring
3-gram LM training
# TLHdig 3M token corpus → KenLM
python3 -c "
import json
with open('datasets/sources/tlhdig/corpus.jsonl') as f, open('tlhdig_text.txt', 'w') as g:
for line in f:
r = json.loads(line)
g.write(r['cuneiform'] + '\n') # or phonetic_reading
"
kenlm/bin/lmplz -o 3 < tlhdig_text.txt > tlhdig.arpa
kenlm/bin/build_binary tlhdig.arpa tlhdig.binary
Viterbi rescoring
import kenlm
lm = kenlm.Model('tlhdig.binary')
def rescore_lattice(lattice, lambda_lm=0.3):
"""
lattice: list of (top-5 ABZ, log_probs) per position
Find best path via Viterbi with LM prior.
"""
# Dynamic programming
n = len(lattice)
best = {} # (position, prev_abz) → (score, path)
best[(0, '<s>')] = (0.0, [])
for i in range(n):
for prev in best:
if prev[0] != i: continue
score_prev, path_prev = best[prev]
for j, (abz, logp) in enumerate(lattice[i]):
# Acoustic score (classifier)
acoustic = logp
# LM score
context = ' '.join(path_prev[-2:] + [abz])
lm_score = lm.score(context, bos=False, eos=False)
# Combined
new_score = score_prev + acoustic + lambda_lm * lm_score
key = (i+1, abz)
if key not in best or new_score > best[key][0]:
best[key] = (new_score, path_prev + [abz])
# Final: max over all (n, *)
final = max([(s, p) for (pos, _), (s, p) in best.items() if pos == n])
return final[1] # best path
Evaluation protokolü v2
# hitit_ocr/src/evaluate_e2e_v2.py
def evaluate():
# 1. Pure baseline (tek model, no TTA, no rejection)
acc_baseline = eval_top1(model=dinov3_lora, data=gold_set)
# 2. + EMA+SWA
acc_swa = eval_top1(model=dinov3_swa, data=gold_set)
# 3. + TTA
acc_tta = eval_top1_tta(model=dinov3_swa, scales=[224,320,384,448], data=gold_set)
# 4. + Ensemble
acc_ensemble = eval_top1_ensemble(
models=[dinov3_swa, convnext_swa, siglip_swa],
weights=[0.45, 0.30, 0.25],
tta=True,
data=gold_set
)
# 5. + LM rescoring
acc_lm = eval_with_lm(ensemble, lm=tlhdig_3gram, lambda_lm=0.3, data=gold_set)
# 6. + Rejection (selective accuracy)
sel_acc, coverage = eval_selective(ensemble, threshold=0.6, data=gold_set)
# 7. Primary: pure top-1 (no rejection)
print(f"Primary top-1: {acc_lm}") # target ≥ 0.90
print(f"Selective acc @ 85% cov: {sel_acc}") # target ≥ 0.94
Training schedule v2 (12 hafta)
| Hafta | Aşama | GPU-saat |
|---|---|---|
| 1 | Label cleanlab + hitit_local 2× aug | 40 |
| 2-3 | DINOv3 continual SSL | 400 |
| 4-5 | YOLO11-P2 5-fold CV | 600 |
| 5-6 | DINOv3-L+LoRA classifier (primary) | 400 |
| 6-7 | ConvNeXt-V2-L classifier | 300 |
| 7-8 | SigLIP2-SO400M classifier | 300 |
| 8 | EMA+SWA averaging | 20 |
| 8-9 | Ensemble weight optimization (Bayesian) | 40 |
| 9 | Pseudo-label iter 1 → retrain | 200 |
| 10 | Pseudo-label iter 2, 3 | 300 |
| 10-11 | Knowledge distillation → single model | 300 |
| 11 | TLHdig LM training + Viterbi | 50 |
| 11-12 | ByT5 transliteration | 100 |
| 12 | Evaluation + rejection calibration | 50 |
Total: ~3,100 GPU-saat A100, 32 gün 4×A100.
Performans hedefleri (realistik projeksiyonlar)
| Metrik | v1 baseline | v2 target | Mekanizma |
|---|---|---|---|
| Top-1 acc (198 gold) | ~77% | ≥90% | Ensemble + TTA + SWA + LM |
| Top-5 acc | ~89% | ≥97% | Ensemble |
| Selective acc @ 85% coverage | N/A | ≥94% | Rejection @τ=0.6 |
| 3300-class macro-F1 | ~55-60% | ≥65% | LDAM + pseudo |
| Head tier F1 | ~88% | ≥95% | Ensemble |
| Tail tier F1 | ~40% | ≥60% | KD + synthetic |
| Rare tier F1 | ~25% | ≥45% | Prototype + pseudo |
| Open-set AUROC | N/A | ≥0.85 | Energy |
| CER (hit) | ~12% | ≤7% | ByT5 + LM |
Pipeline güncellemeleri (kod organizasyonu)
hitit_ocr/
├── PIPELINE.md # v1 (baseline)
├── PIPELINE_V2_90PCT.md # BU (v2, %90 hedef)
├── configs/
│ ├── classification_hitit_v1.yaml # single DINOv3
│ ├── classification_ensemble_v2.yaml (YENİ)
│ ├── classification_convnext.yaml (YENİ)
│ ├── classification_siglip2.yaml (YENİ)
│ ├── distillation_v2.yaml (YENİ)
│ ├── pseudo_label_v2.yaml (YENİ)
│ ├── lm_rescoring_v2.yaml (YENİ)
│ └── rejection_v2.yaml (YENİ)
└── src/
├── classification/
│ ├── ensemble.py (YENİ — 3 backbone)
│ ├── ema_swa.py (YENİ)
│ ├── tta_inference.py (YENİ)
│ ├── film_conditioning.py (YENİ)
│ ├── distillation.py (YENİ)
│ ├── pseudo_label.py (YENİ)
│ └── cleanlab_noise.py (YENİ)
├── pipeline/
│ ├── rejection.py (YENİ)
│ └── tablet_context.py (YENİ)
└── lm/
├── train_kenlm.py (YENİ)
└── viterbi_rescore.py (YENİ)
Kaynaklar
- Izmailov et al. SWA (arXiv:1803.05407)
- Szegedy et al. Label Smoothing (CVPR 2016)
- Shanmugam et al. TTA for Documents (CVPR21 workshop)
- Huang et al. DIST KD (arXiv:2205.10536, NeurIPS 2022)
- Sohn et al. FixMatch (arXiv:2001.07685, NeurIPS 2020)
- Northcutt et al. Confident Learning (cleanlab, JAIR 2021)
- Geifman & El-Yaniv Selective Classification (arXiv:1705.08500, NeurIPS 2017)
- Zhu et al. OpenMix (CVPR 2023)
- Gordin et al. Akkadian cuneiform seq2seq (PLOS ONE 2020)
- Yuan et al. Contextual Cuneiform (ICDAR 2021)
- ConvNeXt-V2 (arXiv:2301.00808)
- SigLIP2 (arXiv:2502.14786)
- DeepAramaic (arXiv:2407.07124)