# HITIT-OCR v2 — %90+ Top-1 Accuracy Pipeline **Hedef**: 198-class Hitit gold subset (3253 sample, `in_gold_eval=True`) üzerinde **top-1 accuracy ≥ 0.90**. ## Realistik hedef tanımı ⚠️ **Önemli**: Literatürde **3300 sınıf macro-F1 %90 raporlanmadı** — ulaşılamaz. Gerçekçi: - **Primary target**: 198-class Hitit gold top-1 ≥ 0.90 (no rejection, pure) - **Enhanced target**: Rejection@τ=0.6 ile selective accuracy ≥ 0.94, coverage ≥ 0.85 - **Full 3300 macro-F1**: 0.60-0.65 (tier-stratified) ## Baseline (v1 pipeline) - DINOv3+LoRA+Hierarchical+CB-Focal→LDAM → tahmini top-1 ~%77-80 (DeepScribe extrapolated) - Gap to target: +%10-13 puan ## %90'a ulaştıran SOTA teknikler ### Gain tablosu (198-class Hitit gold) | # | Teknik | Beklenen Gain | Maliyet | Paper | |---|---|---|---|---| | 1 | EMA + SWA averaging | +1.0-1.5% | ~0 | Izmailov 2018 | | 2 | Label smoothing ε=0.1 | +0.5-1.0% | ~0 | Szegedy 2016 | | 3 | Multi-scale TTA (224+320+384+448) | +3-5% | 4× infer | Shanmugam CVPR21 | | 4 | 3-model ensemble (DINOv3+ConvNeXt-V2+SigLIP2) | +4-7% | 3× train | Kaggle consensus | | 5 | Knowledge Distillation (ensemble→single) | +1-2% tail +5-8% | +1× train | DIST NeurIPS22 | | 6 | Pseudo-labeling self-training | +2-4% tail | 3 iteration | FixMatch | | 7 | TLHdig LM rescoring (Viterbi) | +3-5% | LM 1 gün | Gordin 2020 | | 8 | Rejection @τ=0.6 (selective acc) | +3-6% | 0 | Geifman 2017 | | 9 | Label cleanlab noise detection | +1-2% | cleanlab | Northcutt 2021 | | 10 | Tablet-level context aggregation | +1-3% | 1× train | Yuan ICDAR21 | | 11 | Active learning (5K re-annotate) | +2-4% | manual | Gal 2017 | | 12 | Synthetic rare class generation | +3-5% tail | 500 GPU-h | DeepAramaic | | 13 | Paleographic style conditioning (FiLM) | +1-2% cross-source | minor | — | | 14 | Hierarchical fallback (L2→L3) | +2-4% | 0 | Zhu ICCV23 | | **TOPLAM (overlap dedüksiyon sonrası)** | **+12-18%** | — | — | **Baseline 77%** + **15% average gain** = **~92%** (güvenli tahmin). ## Güncellenen Mimari ``` ┌─────────────────────────────────────────────────────────────────────┐ │ INPUT: Tablet image │ └──────────────────────────┬──────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Stage 0 — Preprocessing (v1'den aynı + label cleanlab) │ │ + Label noise detection (cleanlab cross-val confident learning) │ └──────────────────────────┬──────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Stage 1 — Detection (v1'den aynı) │ │ YOLO11-P2 + SAHI │ └──────────────────────────┬──────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Stage 2 — Classification (ENHANCED) │ │ │ │ ┌───────────────┐ ┌───────────────┐ ┌────────────────┐ │ │ │ DINOv3-L/14 │ │ ConvNeXt-V2-L │ │ SigLIP2-SO400M │ │ │ │ +LoRA+Hier │ │ +Hier │ │ +Hier │ │ │ │ (w=0.45) │ │ (w=0.30) │ │ (w=0.25) │ │ │ └───────┬───────┘ └───────┬───────┘ └────────┬───────┘ │ │ └──────────────────┼────────────────────┘ │ │ ▼ │ │ Weighted logit average │ │ ▼ │ │ EMA + SWA checkpoint averaging │ │ ▼ │ │ Multi-scale TTA (224+320+384+448) │ │ ▼ │ │ FiLM paleographic style conditioning │ │ ▼ │ │ Knowledge Distillation (→single DINOv3-L) │ │ ▼ │ │ Pseudo-label iteration (3×) │ └──────────────────────────┬──────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Stage 3 — Line Assembly + OOD + Rejection │ │ + Tablet-level context aggregation (graph pooling) │ │ + Selective classification rejection (τ=0.6) │ └──────────────────────────┬──────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Stage 4 — Transliteration (ENHANCED) │ │ ByT5 + Viterbi + TLHdig 3-gram LM fusion │ │ + Hierarchical L2 (25 paleographic clusters) fallback │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Stage 2 — Detaylı (en çok değişen) ### Ensemble: 3 backbone | Model | Params | Role | Weight | Eğitim süresi | |---|---|---|---|---| | **DINOv3-L/14** | 300M | Primary SSL | 0.45 | 800 GPU-h | | **ConvNeXt-V2-Large** | 198M | CNN inductive bias | 0.30 | 400 GPU-h | | **SigLIP2-SO400M** | 400M | Multilingual symbol | 0.25 | 300 GPU-h | Ensemble formülü: ```python logits_final = 0.45 * logits_dino + 0.30 * logits_convnext + 0.25 * logits_siglip # weights val set ile fit — Bayesian optimization ``` ### EMA + SWA ```python # Son %25 epoch'ta EMA decay 0.9999 from torch.optim.swa_utils import AveragedModel, SWALR model = base_model swa_model = AveragedModel(model) swa_start_epoch = 60 # 80 total, son 20 epoch SWA swa_scheduler = SWALR(optimizer, swa_lr=5e-5) for epoch in range(80): train_epoch(model) if epoch >= swa_start_epoch: swa_model.update_parameters(model) swa_scheduler.step() # Checkpoint averaging sonrası BN update torch.optim.swa_utils.update_bn(train_loader, swa_model) ``` ### Multi-scale TTA ```python def tta_predict(model, image): scales = [224, 320, 384, 448] augments = [ lambda x: x, # identity lambda x: F.adjust_gamma(x, 0.9), # gamma darker lambda x: F.adjust_gamma(x, 1.1), # gamma brighter lambda x: TF.rotate(x, angle=3), # +3° rotation lambda x: TF.rotate(x, angle=-3), # -3° rotation ] # NO horizontal flip (cuneiform asymmetric) all_logits = [] for scale in scales: resized = F.resize(image, (scale, scale)) for aug in augments: aug_img = aug(resized) logits = model(aug_img.unsqueeze(0)) all_logits.append(F.softmax(logits, dim=-1)) # Average over 20 augmented versions return torch.stack(all_logits).mean(0) ``` ### Knowledge Distillation (ensemble → single DINOv3-L) ```yaml # Distillation config teacher: ensemble (DINOv3-L + ConvNeXt-V2 + SigLIP2) student: DINOv3-L/14 loss: soft_target_weight: 0.7 hard_target_weight: 0.3 temperature: 4.0 method: DIST # Pearson correlation (NeurIPS 2022) epochs: 30 lr: 1e-4 ``` Son deployment **tek DINOv3-L** — inference 3× hızlı. ### Pseudo-labeling (3 iteration self-training) ``` Iteration 1: - Ensemble'la unlabeled hitit_local + ebl_ocr üzerinde predict - confidence > 0.90 (head/mid), > 0.75 (tail) → pseudo-label - Training set'e ekle (15K yeni sample expected) - Re-train classifier Iteration 2-3: - Önceki pseudo-labels içinde cleanlab noise detection - Remove noisy, add new confident → iterate - Typical convergence 3 iterations ``` ### Label noise cleaning ```python # cleanlab ile label noise detection from cleanlab.classification import CleanLearning from cleanlab.filter import find_label_issues # 5-fold cross-val predictions pred_probs = cross_val_predict(classifier, X, y, cv=5, method='predict_proba') label_issues = find_label_issues( labels=y, pred_probs=pred_probs, filter_by='both', return_indices_ranked_by='self_confidence' ) # Beklenen: %3-8 noise found # Strategy: top-50% noise → remove, diğerleri → relabel ``` ### Paleographic style conditioning (FiLM) ```python class FiLMClassifier(nn.Module): def __init__(self, backbone, n_classes, n_styles=11): self.backbone = backbone self.style_embed = nn.Embedding(n_styles, 256) self.film_gamma = nn.Linear(256, 768) self.film_beta = nn.Linear(256, 768) self.head = nn.Linear(768, n_classes) def forward(self, x, source_id): feat = self.backbone(x) # (B, 768) style = self.style_embed(source_id) gamma = self.film_gamma(style) beta = self.film_beta(style) feat = gamma * feat + beta # style modulation return self.head(feat) ``` 11 source_id: hitit_local, deepscribe, ebl, ebl_v2, maicubeda, heicubeda, cuneiml, compvis, old_babylonian_zip, transliterated_fragments, yeni_veri. ## Stage 3 — Rejection + Tablet context ### Rejection (selective classification) ```python def predict_with_rejection(model, x, threshold=0.6): logits = model(x) probs = F.softmax(logits, dim=-1) max_prob, pred = probs.max(dim=-1) if max_prob < threshold: return "REJECT", max_prob return pred, max_prob # Evaluation n_correct_non_reject = sum(1 for p, y, r in predictions if p == y and r != "REJECT") n_non_reject = sum(1 for _, _, r in predictions if r != "REJECT") selective_accuracy = n_correct_non_reject / n_non_reject coverage = n_non_reject / total ``` **Hedef**: `selective_accuracy ≥ 0.94` at `coverage ≥ 0.85` ### Tablet-level context aggregation ```python # Graph pooling: aynı tablet_id'deki bbox'lar arasında attention class TabletContextAggregator(nn.Module): def __init__(self, feat_dim=768): self.attention = nn.MultiheadAttention(feat_dim, num_heads=8) self.proj = nn.Linear(feat_dim, feat_dim) def forward(self, bbox_feats, tablet_mask): # bbox_feats: (N_bbox, 768) # tablet_mask: attention only within same tablet attn_out, _ = self.attention(bbox_feats, bbox_feats, bbox_feats, attn_mask=tablet_mask) return self.proj(attn_out + bbox_feats) # residual ``` ## Stage 4 — TLHdig LM rescoring ### 3-gram LM training ```bash # TLHdig 3M token corpus → KenLM python3 -c " import json with open('datasets/sources/tlhdig/corpus.jsonl') as f, open('tlhdig_text.txt', 'w') as g: for line in f: r = json.loads(line) g.write(r['cuneiform'] + '\n') # or phonetic_reading " kenlm/bin/lmplz -o 3 < tlhdig_text.txt > tlhdig.arpa kenlm/bin/build_binary tlhdig.arpa tlhdig.binary ``` ### Viterbi rescoring ```python import kenlm lm = kenlm.Model('tlhdig.binary') def rescore_lattice(lattice, lambda_lm=0.3): """ lattice: list of (top-5 ABZ, log_probs) per position Find best path via Viterbi with LM prior. """ # Dynamic programming n = len(lattice) best = {} # (position, prev_abz) → (score, path) best[(0, '')] = (0.0, []) for i in range(n): for prev in best: if prev[0] != i: continue score_prev, path_prev = best[prev] for j, (abz, logp) in enumerate(lattice[i]): # Acoustic score (classifier) acoustic = logp # LM score context = ' '.join(path_prev[-2:] + [abz]) lm_score = lm.score(context, bos=False, eos=False) # Combined new_score = score_prev + acoustic + lambda_lm * lm_score key = (i+1, abz) if key not in best or new_score > best[key][0]: best[key] = (new_score, path_prev + [abz]) # Final: max over all (n, *) final = max([(s, p) for (pos, _), (s, p) in best.items() if pos == n]) return final[1] # best path ``` ## Evaluation protokolü v2 ```python # hitit_ocr/src/evaluate_e2e_v2.py def evaluate(): # 1. Pure baseline (tek model, no TTA, no rejection) acc_baseline = eval_top1(model=dinov3_lora, data=gold_set) # 2. + EMA+SWA acc_swa = eval_top1(model=dinov3_swa, data=gold_set) # 3. + TTA acc_tta = eval_top1_tta(model=dinov3_swa, scales=[224,320,384,448], data=gold_set) # 4. + Ensemble acc_ensemble = eval_top1_ensemble( models=[dinov3_swa, convnext_swa, siglip_swa], weights=[0.45, 0.30, 0.25], tta=True, data=gold_set ) # 5. + LM rescoring acc_lm = eval_with_lm(ensemble, lm=tlhdig_3gram, lambda_lm=0.3, data=gold_set) # 6. + Rejection (selective accuracy) sel_acc, coverage = eval_selective(ensemble, threshold=0.6, data=gold_set) # 7. Primary: pure top-1 (no rejection) print(f"Primary top-1: {acc_lm}") # target ≥ 0.90 print(f"Selective acc @ 85% cov: {sel_acc}") # target ≥ 0.94 ``` ## Training schedule v2 (12 hafta) | Hafta | Aşama | GPU-saat | |---|---|---| | 1 | Label cleanlab + hitit_local 2× aug | 40 | | 2-3 | DINOv3 continual SSL | 400 | | 4-5 | YOLO11-P2 5-fold CV | 600 | | 5-6 | DINOv3-L+LoRA classifier (primary) | 400 | | 6-7 | ConvNeXt-V2-L classifier | 300 | | 7-8 | SigLIP2-SO400M classifier | 300 | | 8 | EMA+SWA averaging | 20 | | 8-9 | Ensemble weight optimization (Bayesian) | 40 | | 9 | Pseudo-label iter 1 → retrain | 200 | | 10 | Pseudo-label iter 2, 3 | 300 | | 10-11 | Knowledge distillation → single model | 300 | | 11 | TLHdig LM training + Viterbi | 50 | | 11-12 | ByT5 transliteration | 100 | | 12 | Evaluation + rejection calibration | 50 | **Total**: ~3,100 GPU-saat A100, 32 gün 4×A100. ## Performans hedefleri (realistik projeksiyonlar) | Metrik | v1 baseline | v2 target | Mekanizma | |---|---|---|---| | **Top-1 acc (198 gold)** | ~77% | **≥90%** | Ensemble + TTA + SWA + LM | | Top-5 acc | ~89% | ≥97% | Ensemble | | Selective acc @ 85% coverage | N/A | **≥94%** | Rejection @τ=0.6 | | 3300-class macro-F1 | ~55-60% | **≥65%** | LDAM + pseudo | | Head tier F1 | ~88% | ≥95% | Ensemble | | Tail tier F1 | ~40% | ≥60% | KD + synthetic | | Rare tier F1 | ~25% | ≥45% | Prototype + pseudo | | Open-set AUROC | N/A | ≥0.85 | Energy | | CER (hit) | ~12% | ≤7% | ByT5 + LM | ## Pipeline güncellemeleri (kod organizasyonu) ``` hitit_ocr/ ├── PIPELINE.md # v1 (baseline) ├── PIPELINE_V2_90PCT.md # BU (v2, %90 hedef) ├── configs/ │ ├── classification_hitit_v1.yaml # single DINOv3 │ ├── classification_ensemble_v2.yaml (YENİ) │ ├── classification_convnext.yaml (YENİ) │ ├── classification_siglip2.yaml (YENİ) │ ├── distillation_v2.yaml (YENİ) │ ├── pseudo_label_v2.yaml (YENİ) │ ├── lm_rescoring_v2.yaml (YENİ) │ └── rejection_v2.yaml (YENİ) └── src/ ├── classification/ │ ├── ensemble.py (YENİ — 3 backbone) │ ├── ema_swa.py (YENİ) │ ├── tta_inference.py (YENİ) │ ├── film_conditioning.py (YENİ) │ ├── distillation.py (YENİ) │ ├── pseudo_label.py (YENİ) │ └── cleanlab_noise.py (YENİ) ├── pipeline/ │ ├── rejection.py (YENİ) │ └── tablet_context.py (YENİ) └── lm/ ├── train_kenlm.py (YENİ) └── viterbi_rescore.py (YENİ) ``` ## Kaynaklar - Izmailov et al. SWA (arXiv:1803.05407) - Szegedy et al. Label Smoothing (CVPR 2016) - Shanmugam et al. TTA for Documents (CVPR21 workshop) - Huang et al. DIST KD (arXiv:2205.10536, NeurIPS 2022) - Sohn et al. FixMatch (arXiv:2001.07685, NeurIPS 2020) - Northcutt et al. Confident Learning (cleanlab, JAIR 2021) - Geifman & El-Yaniv Selective Classification (arXiv:1705.08500, NeurIPS 2017) - Zhu et al. OpenMix (CVPR 2023) - Gordin et al. Akkadian cuneiform seq2seq (PLOS ONE 2020) - Yuan et al. Contextual Cuneiform (ICDAR 2021) - ConvNeXt-V2 (arXiv:2301.00808) - SigLIP2 (arXiv:2502.14786) - DeepAramaic (arXiv:2407.07124)