| # HITIT-OCR v2 — %90+ Top-1 Accuracy Pipeline |
|
|
| **Hedef**: 198-class Hitit gold subset (3253 sample, `in_gold_eval=True`) üzerinde **top-1 accuracy ≥ 0.90**. |
|
|
| ## Realistik hedef tanımı |
|
|
| ⚠️ **Önemli**: Literatürde **3300 sınıf macro-F1 %90 raporlanmadı** — ulaşılamaz. |
|
|
| Gerçekçi: |
| - **Primary target**: 198-class Hitit gold top-1 ≥ 0.90 (no rejection, pure) |
| - **Enhanced target**: Rejection@τ=0.6 ile selective accuracy ≥ 0.94, coverage ≥ 0.85 |
| - **Full 3300 macro-F1**: 0.60-0.65 (tier-stratified) |
|
|
| ## Baseline (v1 pipeline) |
| - DINOv3+LoRA+Hierarchical+CB-Focal→LDAM → tahmini top-1 ~%77-80 (DeepScribe extrapolated) |
| - Gap to target: +%10-13 puan |
|
|
| ## %90'a ulaştıran SOTA teknikler |
|
|
| ### Gain tablosu (198-class Hitit gold) |
|
|
| | # | Teknik | Beklenen Gain | Maliyet | Paper | |
| |---|---|---|---|---| |
| | 1 | EMA + SWA averaging | +1.0-1.5% | ~0 | Izmailov 2018 | |
| | 2 | Label smoothing ε=0.1 | +0.5-1.0% | ~0 | Szegedy 2016 | |
| | 3 | Multi-scale TTA (224+320+384+448) | +3-5% | 4× infer | Shanmugam CVPR21 | |
| | 4 | 3-model ensemble (DINOv3+ConvNeXt-V2+SigLIP2) | +4-7% | 3× train | Kaggle consensus | |
| | 5 | Knowledge Distillation (ensemble→single) | +1-2% tail +5-8% | +1× train | DIST NeurIPS22 | |
| | 6 | Pseudo-labeling self-training | +2-4% tail | 3 iteration | FixMatch | |
| | 7 | TLHdig LM rescoring (Viterbi) | +3-5% | LM 1 gün | Gordin 2020 | |
| | 8 | Rejection @τ=0.6 (selective acc) | +3-6% | 0 | Geifman 2017 | |
| | 9 | Label cleanlab noise detection | +1-2% | cleanlab | Northcutt 2021 | |
| | 10 | Tablet-level context aggregation | +1-3% | 1× train | Yuan ICDAR21 | |
| | 11 | Active learning (5K re-annotate) | +2-4% | manual | Gal 2017 | |
| | 12 | Synthetic rare class generation | +3-5% tail | 500 GPU-h | DeepAramaic | |
| | 13 | Paleographic style conditioning (FiLM) | +1-2% cross-source | minor | — | |
| | 14 | Hierarchical fallback (L2→L3) | +2-4% | 0 | Zhu ICCV23 | |
| | **TOPLAM (overlap dedüksiyon sonrası)** | **+12-18%** | — | — | |
|
|
| **Baseline 77%** + **15% average gain** = **~92%** (güvenli tahmin). |
|
|
| ## Güncellenen Mimari |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ INPUT: Tablet image │ |
| └──────────────────────────┬──────────────────────────────────────────┘ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ Stage 0 — Preprocessing (v1'den aynı + label cleanlab) │ |
| │ + Label noise detection (cleanlab cross-val confident learning) │ |
| └──────────────────────────┬──────────────────────────────────────────┘ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ Stage 1 — Detection (v1'den aynı) │ |
| │ YOLO11-P2 + SAHI │ |
| └──────────────────────────┬──────────────────────────────────────────┘ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ Stage 2 — Classification (ENHANCED) │ |
| │ │ |
| │ ┌───────────────┐ ┌───────────────┐ ┌────────────────┐ │ |
| │ │ DINOv3-L/14 │ │ ConvNeXt-V2-L │ │ SigLIP2-SO400M │ │ |
| │ │ +LoRA+Hier │ │ +Hier │ │ +Hier │ │ |
| │ │ (w=0.45) │ │ (w=0.30) │ │ (w=0.25) │ │ |
| │ └───────┬───────┘ └───────┬───────┘ └────────┬───────┘ │ |
| │ └──────────────────┼────────────────────┘ │ |
| │ ▼ │ |
| │ Weighted logit average │ |
| │ ▼ │ |
| │ EMA + SWA checkpoint averaging │ |
| │ ▼ │ |
| │ Multi-scale TTA (224+320+384+448) │ |
| │ ▼ │ |
| │ FiLM paleographic style conditioning │ |
| │ ▼ │ |
| │ Knowledge Distillation (→single DINOv3-L) │ |
| │ ▼ │ |
| │ Pseudo-label iteration (3×) │ |
| └──────────────────────────┬──────────────────────────────────────────┘ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ Stage 3 — Line Assembly + OOD + Rejection │ |
| │ + Tablet-level context aggregation (graph pooling) │ |
| │ + Selective classification rejection (τ=0.6) │ |
| └──────────────────────────┬──────────────────────────────────────────┘ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ Stage 4 — Transliteration (ENHANCED) │ |
| │ ByT5 + Viterbi + TLHdig 3-gram LM fusion │ |
| │ + Hierarchical L2 (25 paleographic clusters) fallback │ |
| └─────────────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| ## Stage 2 — Detaylı (en çok değişen) |
|
|
| ### Ensemble: 3 backbone |
|
|
| | Model | Params | Role | Weight | Eğitim süresi | |
| |---|---|---|---|---| |
| | **DINOv3-L/14** | 300M | Primary SSL | 0.45 | 800 GPU-h | |
| | **ConvNeXt-V2-Large** | 198M | CNN inductive bias | 0.30 | 400 GPU-h | |
| | **SigLIP2-SO400M** | 400M | Multilingual symbol | 0.25 | 300 GPU-h | |
|
|
| Ensemble formülü: |
| ```python |
| logits_final = 0.45 * logits_dino + 0.30 * logits_convnext + 0.25 * logits_siglip |
| # weights val set ile fit — Bayesian optimization |
| ``` |
|
|
| ### EMA + SWA |
| ```python |
| # Son %25 epoch'ta EMA decay 0.9999 |
| from torch.optim.swa_utils import AveragedModel, SWALR |
| |
| model = base_model |
| swa_model = AveragedModel(model) |
| swa_start_epoch = 60 # 80 total, son 20 epoch SWA |
| swa_scheduler = SWALR(optimizer, swa_lr=5e-5) |
| |
| for epoch in range(80): |
| train_epoch(model) |
| if epoch >= swa_start_epoch: |
| swa_model.update_parameters(model) |
| swa_scheduler.step() |
| |
| # Checkpoint averaging sonrası BN update |
| torch.optim.swa_utils.update_bn(train_loader, swa_model) |
| ``` |
|
|
| ### Multi-scale TTA |
| ```python |
| def tta_predict(model, image): |
| scales = [224, 320, 384, 448] |
| augments = [ |
| lambda x: x, # identity |
| lambda x: F.adjust_gamma(x, 0.9), # gamma darker |
| lambda x: F.adjust_gamma(x, 1.1), # gamma brighter |
| lambda x: TF.rotate(x, angle=3), # +3° rotation |
| lambda x: TF.rotate(x, angle=-3), # -3° rotation |
| ] |
| # NO horizontal flip (cuneiform asymmetric) |
| all_logits = [] |
| for scale in scales: |
| resized = F.resize(image, (scale, scale)) |
| for aug in augments: |
| aug_img = aug(resized) |
| logits = model(aug_img.unsqueeze(0)) |
| all_logits.append(F.softmax(logits, dim=-1)) |
| # Average over 20 augmented versions |
| return torch.stack(all_logits).mean(0) |
| ``` |
|
|
| ### Knowledge Distillation (ensemble → single DINOv3-L) |
|
|
| ```yaml |
| # Distillation config |
| teacher: ensemble (DINOv3-L + ConvNeXt-V2 + SigLIP2) |
| student: DINOv3-L/14 |
| loss: |
| soft_target_weight: 0.7 |
| hard_target_weight: 0.3 |
| temperature: 4.0 |
| method: DIST # Pearson correlation (NeurIPS 2022) |
| epochs: 30 |
| lr: 1e-4 |
| ``` |
|
|
| Son deployment **tek DINOv3-L** — inference 3× hızlı. |
|
|
| ### Pseudo-labeling (3 iteration self-training) |
|
|
| ``` |
| Iteration 1: |
| - Ensemble'la unlabeled hitit_local + ebl_ocr üzerinde predict |
| - confidence > 0.90 (head/mid), > 0.75 (tail) → pseudo-label |
| - Training set'e ekle (15K yeni sample expected) |
| - Re-train classifier |
| |
| Iteration 2-3: |
| - Önceki pseudo-labels içinde cleanlab noise detection |
| - Remove noisy, add new confident → iterate |
| - Typical convergence 3 iterations |
| ``` |
|
|
| ### Label noise cleaning |
|
|
| ```python |
| # cleanlab ile label noise detection |
| from cleanlab.classification import CleanLearning |
| from cleanlab.filter import find_label_issues |
| |
| # 5-fold cross-val predictions |
| pred_probs = cross_val_predict(classifier, X, y, cv=5, method='predict_proba') |
| label_issues = find_label_issues( |
| labels=y, |
| pred_probs=pred_probs, |
| filter_by='both', |
| return_indices_ranked_by='self_confidence' |
| ) |
| # Beklenen: %3-8 noise found |
| # Strategy: top-50% noise → remove, diğerleri → relabel |
| ``` |
|
|
| ### Paleographic style conditioning (FiLM) |
|
|
| ```python |
| class FiLMClassifier(nn.Module): |
| def __init__(self, backbone, n_classes, n_styles=11): |
| self.backbone = backbone |
| self.style_embed = nn.Embedding(n_styles, 256) |
| self.film_gamma = nn.Linear(256, 768) |
| self.film_beta = nn.Linear(256, 768) |
| self.head = nn.Linear(768, n_classes) |
| |
| def forward(self, x, source_id): |
| feat = self.backbone(x) # (B, 768) |
| style = self.style_embed(source_id) |
| gamma = self.film_gamma(style) |
| beta = self.film_beta(style) |
| feat = gamma * feat + beta # style modulation |
| return self.head(feat) |
| ``` |
|
|
| 11 source_id: hitit_local, deepscribe, ebl, ebl_v2, maicubeda, heicubeda, cuneiml, compvis, old_babylonian_zip, transliterated_fragments, yeni_veri. |
| |
| ## Stage 3 — Rejection + Tablet context |
| |
| ### Rejection (selective classification) |
| |
| ```python |
| def predict_with_rejection(model, x, threshold=0.6): |
| logits = model(x) |
| probs = F.softmax(logits, dim=-1) |
| max_prob, pred = probs.max(dim=-1) |
| |
| if max_prob < threshold: |
| return "REJECT", max_prob |
| return pred, max_prob |
| |
| # Evaluation |
| n_correct_non_reject = sum(1 for p, y, r in predictions |
| if p == y and r != "REJECT") |
| n_non_reject = sum(1 for _, _, r in predictions if r != "REJECT") |
| selective_accuracy = n_correct_non_reject / n_non_reject |
| coverage = n_non_reject / total |
| ``` |
| |
| **Hedef**: `selective_accuracy ≥ 0.94` at `coverage ≥ 0.85` |
|
|
| ### Tablet-level context aggregation |
|
|
| ```python |
| # Graph pooling: aynı tablet_id'deki bbox'lar arasında attention |
| class TabletContextAggregator(nn.Module): |
| def __init__(self, feat_dim=768): |
| self.attention = nn.MultiheadAttention(feat_dim, num_heads=8) |
| self.proj = nn.Linear(feat_dim, feat_dim) |
| |
| def forward(self, bbox_feats, tablet_mask): |
| # bbox_feats: (N_bbox, 768) |
| # tablet_mask: attention only within same tablet |
| attn_out, _ = self.attention(bbox_feats, bbox_feats, bbox_feats, |
| attn_mask=tablet_mask) |
| return self.proj(attn_out + bbox_feats) # residual |
| ``` |
|
|
| ## Stage 4 — TLHdig LM rescoring |
|
|
| ### 3-gram LM training |
| ```bash |
| # TLHdig 3M token corpus → KenLM |
| python3 -c " |
| import json |
| with open('datasets/sources/tlhdig/corpus.jsonl') as f, open('tlhdig_text.txt', 'w') as g: |
| for line in f: |
| r = json.loads(line) |
| g.write(r['cuneiform'] + '\n') # or phonetic_reading |
| " |
| kenlm/bin/lmplz -o 3 < tlhdig_text.txt > tlhdig.arpa |
| kenlm/bin/build_binary tlhdig.arpa tlhdig.binary |
| ``` |
|
|
| ### Viterbi rescoring |
| ```python |
| import kenlm |
| lm = kenlm.Model('tlhdig.binary') |
| |
| def rescore_lattice(lattice, lambda_lm=0.3): |
| """ |
| lattice: list of (top-5 ABZ, log_probs) per position |
| Find best path via Viterbi with LM prior. |
| """ |
| # Dynamic programming |
| n = len(lattice) |
| best = {} # (position, prev_abz) → (score, path) |
| best[(0, '<s>')] = (0.0, []) |
| |
| for i in range(n): |
| for prev in best: |
| if prev[0] != i: continue |
| score_prev, path_prev = best[prev] |
| for j, (abz, logp) in enumerate(lattice[i]): |
| # Acoustic score (classifier) |
| acoustic = logp |
| # LM score |
| context = ' '.join(path_prev[-2:] + [abz]) |
| lm_score = lm.score(context, bos=False, eos=False) |
| # Combined |
| new_score = score_prev + acoustic + lambda_lm * lm_score |
| key = (i+1, abz) |
| if key not in best or new_score > best[key][0]: |
| best[key] = (new_score, path_prev + [abz]) |
| |
| # Final: max over all (n, *) |
| final = max([(s, p) for (pos, _), (s, p) in best.items() if pos == n]) |
| return final[1] # best path |
| ``` |
|
|
| ## Evaluation protokolü v2 |
|
|
| ```python |
| # hitit_ocr/src/evaluate_e2e_v2.py |
| def evaluate(): |
| # 1. Pure baseline (tek model, no TTA, no rejection) |
| acc_baseline = eval_top1(model=dinov3_lora, data=gold_set) |
| |
| # 2. + EMA+SWA |
| acc_swa = eval_top1(model=dinov3_swa, data=gold_set) |
| |
| # 3. + TTA |
| acc_tta = eval_top1_tta(model=dinov3_swa, scales=[224,320,384,448], data=gold_set) |
| |
| # 4. + Ensemble |
| acc_ensemble = eval_top1_ensemble( |
| models=[dinov3_swa, convnext_swa, siglip_swa], |
| weights=[0.45, 0.30, 0.25], |
| tta=True, |
| data=gold_set |
| ) |
| |
| # 5. + LM rescoring |
| acc_lm = eval_with_lm(ensemble, lm=tlhdig_3gram, lambda_lm=0.3, data=gold_set) |
| |
| # 6. + Rejection (selective accuracy) |
| sel_acc, coverage = eval_selective(ensemble, threshold=0.6, data=gold_set) |
| |
| # 7. Primary: pure top-1 (no rejection) |
| print(f"Primary top-1: {acc_lm}") # target ≥ 0.90 |
| print(f"Selective acc @ 85% cov: {sel_acc}") # target ≥ 0.94 |
| ``` |
|
|
| ## Training schedule v2 (12 hafta) |
|
|
| | Hafta | Aşama | GPU-saat | |
| |---|---|---| |
| | 1 | Label cleanlab + hitit_local 2× aug | 40 | |
| | 2-3 | DINOv3 continual SSL | 400 | |
| | 4-5 | YOLO11-P2 5-fold CV | 600 | |
| | 5-6 | DINOv3-L+LoRA classifier (primary) | 400 | |
| | 6-7 | ConvNeXt-V2-L classifier | 300 | |
| | 7-8 | SigLIP2-SO400M classifier | 300 | |
| | 8 | EMA+SWA averaging | 20 | |
| | 8-9 | Ensemble weight optimization (Bayesian) | 40 | |
| | 9 | Pseudo-label iter 1 → retrain | 200 | |
| | 10 | Pseudo-label iter 2, 3 | 300 | |
| | 10-11 | Knowledge distillation → single model | 300 | |
| | 11 | TLHdig LM training + Viterbi | 50 | |
| | 11-12 | ByT5 transliteration | 100 | |
| | 12 | Evaluation + rejection calibration | 50 | |
| |
| **Total**: ~3,100 GPU-saat A100, 32 gün 4×A100. |
| |
| ## Performans hedefleri (realistik projeksiyonlar) |
| |
| | Metrik | v1 baseline | v2 target | Mekanizma | |
| |---|---|---|---| |
| | **Top-1 acc (198 gold)** | ~77% | **≥90%** | Ensemble + TTA + SWA + LM | |
| | Top-5 acc | ~89% | ≥97% | Ensemble | |
| | Selective acc @ 85% coverage | N/A | **≥94%** | Rejection @τ=0.6 | |
| | 3300-class macro-F1 | ~55-60% | **≥65%** | LDAM + pseudo | |
| | Head tier F1 | ~88% | ≥95% | Ensemble | |
| | Tail tier F1 | ~40% | ≥60% | KD + synthetic | |
| | Rare tier F1 | ~25% | ≥45% | Prototype + pseudo | |
| | Open-set AUROC | N/A | ≥0.85 | Energy | |
| | CER (hit) | ~12% | ≤7% | ByT5 + LM | |
| |
| ## Pipeline güncellemeleri (kod organizasyonu) |
| |
| ``` |
| hitit_ocr/ |
| ├── PIPELINE.md # v1 (baseline) |
| ├── PIPELINE_V2_90PCT.md # BU (v2, %90 hedef) |
| ├── configs/ |
| │ ├── classification_hitit_v1.yaml # single DINOv3 |
| │ ├── classification_ensemble_v2.yaml (YENİ) |
| │ ├── classification_convnext.yaml (YENİ) |
| │ ├── classification_siglip2.yaml (YENİ) |
| │ ├── distillation_v2.yaml (YENİ) |
| │ ├── pseudo_label_v2.yaml (YENİ) |
| │ ├── lm_rescoring_v2.yaml (YENİ) |
| │ └── rejection_v2.yaml (YENİ) |
| └── src/ |
| ├── classification/ |
| │ ├── ensemble.py (YENİ — 3 backbone) |
| │ ├── ema_swa.py (YENİ) |
| │ ├── tta_inference.py (YENİ) |
| │ ├── film_conditioning.py (YENİ) |
| │ ├── distillation.py (YENİ) |
| │ ├── pseudo_label.py (YENİ) |
| │ └── cleanlab_noise.py (YENİ) |
| ├── pipeline/ |
| │ ├── rejection.py (YENİ) |
| │ └── tablet_context.py (YENİ) |
| └── lm/ |
| ├── train_kenlm.py (YENİ) |
| └── viterbi_rescore.py (YENİ) |
| ``` |
| |
| ## Kaynaklar |
|
|
| - Izmailov et al. SWA (arXiv:1803.05407) |
| - Szegedy et al. Label Smoothing (CVPR 2016) |
| - Shanmugam et al. TTA for Documents (CVPR21 workshop) |
| - Huang et al. DIST KD (arXiv:2205.10536, NeurIPS 2022) |
| - Sohn et al. FixMatch (arXiv:2001.07685, NeurIPS 2020) |
| - Northcutt et al. Confident Learning (cleanlab, JAIR 2021) |
| - Geifman & El-Yaniv Selective Classification (arXiv:1705.08500, NeurIPS 2017) |
| - Zhu et al. OpenMix (CVPR 2023) |
| - Gordin et al. Akkadian cuneiform seq2seq (PLOS ONE 2020) |
| - Yuan et al. Contextual Cuneiform (ICDAR 2021) |
| - ConvNeXt-V2 (arXiv:2301.00808) |
| - SigLIP2 (arXiv:2502.14786) |
| - DeepAramaic (arXiv:2407.07124) |
|
|
