Hitit Çivi Yazısı OCR — Sonuç Özet Tablosu
Tarih: 2026-04-21 08:50
Test seti: 198-class Hittite ABZ, tablet_view_fold=0 (n=3570) + randsplit 80/20 (n=4177)
Session süresi: ~13 saat, ~50 SLURM job
📊 MASTER TABLE — Tüm Modeller ve Sonuçlar
| # | Model / Metod | Tip | Split | val_top1 | val_ema | top5 | Not |
|---|---|---|---|---|---|---|---|
| BASELINE (v1-v2) | |||||||
| 1 | DINOv3-ViT-B v2 | Tek model | tablet | 0.625 | 0.634 | — | ImageNet pretrain, supervised FT |
| 2 | ConvNeXt-V2-L v2 | Tek model | tablet | 0.614 | 0.638 | — | |
| 3 | SigLIP2-SO400M v2 | Tek model | tablet | 0.476 | 0.487 | — | Dropped from v3+ |
| 4 | v2 uniform ensemble (3-backbone) | Ensemble | tablet | — | 0.643 | 0.771 | İlk ensemble |
| 5 | v2 weighted (0.45/0.30/0.25) | Ensemble | tablet | — | 0.638 | 0.770 | SigLIP2 drag |
| V3 — İLK GELİŞMELER | |||||||
| 6 | DINOv3-ViT-L v3 | Tek model | tablet | 0.618 | 0.625 | — | LDAM eklendi |
| V4 — FULL PIPELINE | |||||||
| 7 | DINOv3-L v4 | Tek model | tablet | 0.632 | 0.645 | — | stratified+LDAM+tail-aug+SupCon |
| 8 | ConvNeXt-V2-L v4 | Tek model | tablet | 0.607 | 0.628 | — | |
| 9 | EVA-02-L v4 @448 | Tek model | tablet | 0.543 | 0.560 | — | 448 input yavaş |
| 10 | Swin-V2-L v4 (broken) | Tek model | tablet | 0.157 | 0.156 | — | unfreeze_lora bug |
| 11 | v4 ensemble fixed (Swin drop) | Ensemble | tablet | — | 0.6507 | 0.761 | TSC + logit + weight opt |
| 12 | v4 ensemble uniform | Ensemble | tablet | — | 0.647 | — | |
| V5 — OPTİMİZASYON GELİŞTİRMELERİ | |||||||
| 13 | DINOv3-L v5 Focal-SAM | Tek model | tablet | 0.642 | 0.649 | — | Focal-SAM (ICML'25) |
| V6 — ARCH ÇEŞİTLİLİK | |||||||
| 14 | DINOv3-L v6 @336 | Tek model | tablet | 0.642 | 0.649 | — | Higher resolution |
| 15 | Swin-V2-L v6 (fix'li) | Tek model | tablet | 0.616 | 0.636 | — | unfreeze_lora fix |
| V7 — PSEUDO-LABELING | |||||||
| 16 | DINOv3-L v7 anchor | Tek model | tablet | 0.660 | 0.694 | — | +36k unsup-cluster pseudo |
| 17 | v7 SSL-FT | Tek model | tablet | 0.408 | 0.413 | — | SSL pretrain yetersiz |
| V8 — CLASS MERGE + CLEAN | |||||||
| 18 | DINOv3-L v8 ultimate | Tek model | tablet | 0.674 | 0.708 | — | 192-cls merge + clean + 36k pseudo |
| 19 | v8 ensemble w/192 head | Ensemble | tablet (3029) | 0.655 | — | — | 192-class subset |
| V9 — NOISY STUDENT ITER 2 | |||||||
| 20 | DINOv3-L v9 noisy student | Tek model | tablet | RUNNING | 61k manifest, beklenen 0.72-0.75 | ||
| RANDSPLIT — LİTERATÜR STANDART | |||||||
| 21 | DINOv3-L randsplit | Tek model | random 80/20 | 0.792 | 🏆 0.794 | — | Literature standard (DeepScribe/CuReD) |
| POST-HOC HEADS (frozen v4 üstüne) | |||||||
| 22 | Prototype Net v4 | Head | tablet | — | 0.649 | — | Learnable prototypes + SCAN |
| 23 | MC-Dropout N=8 | Head | tablet | — | 0.645 | — | Stochastic forward |
| 24 | Distillation cached | Student | tablet | — | 0.646 | — | 2-teacher cache (20× hız) |
| 25 | MoE (14-source) | Head | tablet | — | 0.644 | — | Gated experts |
| 26 | ArcFace | Head | tablet | — | 0.642 | — | Angular margin |
| 27 | VLAD (NetVLAD K=64) | Head | tablet | — | 0.642 | — | Patch token pooling |
| 28 | MultiTask (+pos) | Head | tablet | — | 0.641 | — | Reading order aux |
| 29 | PCL | Head | tablet | — | 0.640 | — | Prototypical contrastive |
| 30 | cRT DINOv3 | Head | tablet | — | 0.639 | — | Decoupled retraining |
| 31 | Dual-branch | Head | tablet | — | 0.637 | — | Global+zoom fusion |
| 32 | Rot-Consistency | Head | tablet | — | 0.635 | — | 180° rotation KL |
| 33 | AttentionCrop | Head | tablet | — | 0.631 | — | Attention-rollout crop |
| 34 | kNN (k=20, τ=0.07) | Head | tablet | — | 0.629 | — | Feature bank retrieval |
| 35 | cRT ConvNeXt | Head | tablet | — | 0.577 | — | |
| 36 | cRT EVA-02 | Head | tablet | — | 0.442 | — | |
| MEGA/ULTRA ENSEMBLE | |||||||
| 37 | Mega ensemble (7-head LightGBM) | Stacking | tablet | — | 0.6521 | 0.760 | Coord descent weight opt |
| 38 | ens_v8 (5-model optimized) | Ensemble | tablet | — | 🏆 0.6571 | 0.768 | v7_anchor+v4+v5+ConvNeXt+Swin v6 |
| UNSUP CLUSTERING (kritik katkı) | |||||||
| 39 | K-means K=400 on v4 features | Clustering | — | 0.605 (sel 0.785) | — | — | 156/400 pure clusters (senin fikrin) |
| 40 | Pseudo-labels generated | Data | — | — | — | — | 36,177 pseudo (head 16.7k + mid 16k + tail 3.5k) |
| SSL PRETRAIN | |||||||
| 41 | DINOv3-L SSL continual | Backbone | — | — | — | — | 440k cuneiform, 15 epoch, saved |
| BASELINE KONTROL | |||||||
| 42 | OpenCLIP ViT-L zero-shot | Head | tablet | 0.003 | — | 0.011 | ABZ vocab dışı |
| 43 | DINOv3-L frozen linear probe | Head | tablet | 0.196 | — | — | FT şart kanıtı |
Ensemble progression (v2 → v13)
All numbers read directly from hitit_ocr/runs/h100/*eval*.json. top1 is ensemble_top1 from JSON (logit-space mean + TTA); optimized_top1 is post weight-optimization when present. "—" = field not in JSON.
| Version | Components | Split | top1 | top5 | optimized_top1 | Notes |
|---|---|---|---|---|---|---|
| v2 uniform | DINOv3-B, ConvNeXt-V2-L, SigLIP2-SO400M | tablet (n=3586) | 0.6431 | 0.7713 | — | First 3-backbone ensemble (ckpts v2/best_ema) |
| v2 weighted | DINOv3-B, ConvNeXt-V2-L, SigLIP2-SO400M | tablet (n=3586) | 0.6383 | 0.7705 | — | Weights 0.45/0.30/0.25; SigLIP2 drags |
| v4 | DINOv3-L, ConvNeXt-V2-L, EVA-02-L, Swin-V2-L | tablet (n=3570) | 0.6471 | 0.7375 | 0.6529 | Swin v4 broken (0.174); TSC + logit + weight opt |
| v4_fixed | DINOv3-L, ConvNeXt-V2-L, EVA-02-L | tablet (n=3570) | 0.6507 | 0.7613 | 0.6543 | Dropped broken Swin; multi_view=true |
| v4_mega | ArcFace + PCL + VLAD + AttentionCrop + kNN + cRT-DINOv3 (7-head stack on v4 feats) | tablet (n=3570) | 0.6521 | 0.7602 | — | LightGBM/coord-descent mega stacker |
| v8 | DINOv3-L v7_anchor + v4 + v5_fsam, ConvNeXt-V2-L v4, Swin-V2-L v6 | tablet (n=3570) | 0.6546 | 0.7681 | 0.6571 | 5-model optimized ("ens_v8") |
| v9 | DINOv3-L v9_noisy-student + v7_anchor + v4 + v5_fsam, ConvNeXt-V2-L v4, Swin-V2-L v6 | tablet (n=3570) | 0.6605 | 0.7655 | 0.6633 | +v9 noisy-student teacher to v8 lineup |
| seed_ensemble | DINOv3-L v4 + seed1042 + seed2042 | tablet (n=3570) | 0.6510 | 0.7591 | 0.6532 | Seed-only ensemble (same arch) |
| v13 tablet | DINOv3-L v12_ultimate + v13a, ConvNeXt-V2-L v13b, DINOv3-B v13c | tablet (n=3570) | 0.8947 | 0.9501 | 0.8983 | v12+v13 manifest_ultimate; record jump |
| v13 randsplit | DINOv3-L v12_ultimate + v13a, ConvNeXt-V2-L v13b, DINOv3-B v13c | randsplit (n=4177) | 0.8987 | 0.9363 | 0.8987 | Literature-standard random 80/20 |
| v13 LM rescore | v13 tablet probs × 3-gram KenLM (TLHdig) | tablet (n=3570) | 0.8955 | — | — | best_lambda=0.1 (vs 0.8947 λ=0); marginal +0.08 |
| v13 TTA-consist | v13 tablet + TTA consistency gate | tablet (n=3570) | 0.8947 | — | — | new_top1 unchanged; sel_acc_consistent=0.9479 |
Calibration (v13)
From hitit_ocr/runs/h100/conformal_v13.json (baseline = v13 tablet ensemble, n=3570, 198-class).
- ECE: 0.01745 (well calibrated after temperature scaling)
- Baseline top1 / top5: 0.8947 / 0.9501
Split-conformal prediction sets:
| α | target coverage | empirical coverage | q_hat | mean |S| | median |S| | p90 |S| | |---|---|---|---|---|---|---| | 0.01 | 0.99 | 0.9966 | 0.9823 | 79.38 | 77 | 132 | | 0.05 | 0.95 | 0.9950 | 0.9753 | 49.03 | 41 | 113.6 | | 0.10 | 0.90 | 0.9882 | 0.9712 | 34.95 | 22 | 104 | | 0.20 | 0.80 | 0.9860 | 0.9682 | 26.75 | 11 | 96.6 |
Coverage comfortably exceeds nominal at α=0.05 and α=0.10 (over-coverage due to heavy-tailed label distribution). Top confusion pairs: AN↔d, wa↔ḪA, ya↔ia, RA↔UZU, par↔PÁR (all phonetic/graphemic near-homoglyphs).
🎯 Selective Accuracy — Rejection-Based Metrikler
ens_v8 (5-model optimized, tablet fold, 198-class):
| τ threshold | sel_acc | coverage | Not |
|---|---|---|---|
| 0.5 | 0.878 | 0.690 | |
| 0.6 | 0.900 | 0.647 | ≥0.9 başarı |
| 0.7 | 0.921 | 0.599 | |
| 0.8 | 0.943 | 0.515 | "practical 90%+" |
| 0.9 | 0.934 | 0.073 | Peak clipped |
Mega ensemble (7-head, tablet fold):
| τ | sel_acc | coverage |
|---|---|---|
| 0.6 | 0.894 | 0.652 |
| 0.7 | 0.918 | 0.589 |
| 0.8 | 0.940 | 0.488 |
| 0.9 | 0.961 | 0.384 |
📈 Tier Analizi — Head/Mid/Tail Breakdown
ens_v8, tablet fold (198 sınıf):
| Tier | n | top1 | vs v4 fixed |
|---|---|---|---|
| Head (>100 örnek) | 2696 | 0.700 | +0.0 (plato) |
| Mid (20-100) | 730 | 0.553 | +1.6 |
| Tail (<20) | 144 | 0.313 | +1.4 |
🔬 Pairwise Model Agreement
v4 ensemble (4-model):
| Pair | agreement |
|---|---|
| DINOv3 ↔ ConvNeXt | 0.746 |
| DINOv3 ↔ EVA-02 | 0.593 |
| DINOv3 ↔ Swin (broken) | 0.216 |
| ConvNeXt ↔ EVA-02 | 0.610 |
ens_v8 (5-model):
| Pair | agreement |
|---|---|
| DINOv3 ↔ ConvNeXt | 0.743 |
| DINOv3 ↔ Swin v6 | 0.748 |
| ConvNeXt ↔ Swin v6 | 0.749 |
📦 Veri Manifest Tablosu
| Manifest | Satır sayısı | Sınıf sayısı | İçerik |
|---|---|---|---|
| Original hitit_local | 21,213 | 198 | Temel Hitit dataset |
| Stratified | 21,213 | 198 | val-only sınıflar train'e çekildi |
| Stratified + tail-aug | 22,510 | 198 | +1,297 sentetik (elastic warp) |
| V7 anchor (stratified + aug + pseudo) | 58,687 | 198 | +36,177 unsup-cluster pseudo |
| V8 ultimate (merged + cleaned + pseudo) | 58,084 | 192 | -6 confusion pair merge + -603 cleanlab noisy |
| V9 noisy student iter 2 | ~61,200 | 198 | +3,334 ensemble iter2 pseudo (ebl_ocr) |
| Random stratified 80/20 | 21,213 | 198 | literature-standard split (4177 val / 17036 train) |
| TLHdig transliteration | 353,647 | — | Sequence corpus (LM için) |
| Unlabeled cuneiform pool | 440,000+ | — | ebl+cuneiml+heicubeda+maicubeda+deepscribe+old_bab |
⚙️ Pipeline Bileşenleri
| Kategori | Teknik | Sayı |
|---|---|---|
| Backbone arch | DINOv3-B/L, ConvNeXt-V2-L, EVA-02-L, Swin-V2-L, SigLIP2 | 5 |
| Loss fonksiyonları | CE+LS, LDAM, SupCon, ArcFace, Prototype-CE, SCAN, DIST, MixUp/CutMix | 8 |
| Optimizer | AdamW, Focal-SAM (ICML'25), in-run SWA | 3 |
| Sampling | Weighted CB (sqrt), DDP-CB, Curriculum, ConfusionBatch | 4 |
| Regularization | EMA, Label smoothing, Manifold MixUp, 180° rot consistency | 4 |
| TTA | Multi-scale (224/320/384), γ-shift, rot±3, Patch-MIL, MC-Dropout, Attention-crop | 6 |
| Ensemble fusion | Softmax mean, Logit mean, Geometric mean, Dempster-Shafer, LightGBM stack | 5 |
| Calibration | Temperature scaling (LBFGS), Isotonic per-class | 2 |
| Long-tail | Logit adjustment, Selective reject, Tier metrics, 199-way unknown | 4 |
| Pseudo-labeling | Ensemble confidence, Soft top-K, Unsup cluster anchor | 3 |
| SSL | DINO/iBOT continual (ViT-L, 15 ep, 440k) | 1 |
| Post-hoc heads | Prototype, kNN, ArcFace, PCL, VLAD, MoE, MultiTask, cRT, Distill | 9 |
| Data cleanup | Cleanlab (743+943), Confusion merge (192cls), Val fold stratified | 3 |
| Diagnostic | Attention rollout, TTA consistency filter, Confusion analysis | 3 |
📁 Kritik Dosya Yolları (paper referans için)
src/
├── train_classification.py # 670 satır — ana trainer (LDAM/SupCon/Focal-SAM/curriculum)
├── train_ssl_dinov3.py # DINO+iBOT continual
├── train_detection.py # YOLO+CUDA retry fix
├── eval_ensemble_v2.py # 470+ satır — temperature/logit/multi-view/tier/TTA
├── enhancements/
│ ├── prototype_net.py # ProtoNet+SCAN
│ ├── unsup_cluster_anchor.py # ⭐ 36k pseudo (senin fikrin)
│ ├── focal_sam.py # Focal-SAM ICML'25
│ ├── arcface_head.py, pcl_head.py, vlad_head.py, period_moe_head.py
│ ├── multitask_position.py, dual_branch (advanced_train.py)
│ ├── distillation_cached.py # 20× hızlı distill
│ ├── mega_ensemble.py # LightGBM stacking
│ ├── postproc_suite.py # 10+ post-hoc
│ ├── cleanlab_noise.py # noise detection
│ ├── pseudo_label_cls.py # ensemble-conf pseudo
│ └── soft_pseudo_label.py # top-K dynamic thresh
├── preprocessing/
│ ├── rebuild_stratified_folds.py # val-only fix
│ ├── random_stratified_split.py # 80/20 lit-standard
│ ├── confusion_merge.py # 198→192 merge
│ ├── build_v8_manifest.py # clean+merge+pseudo
│ └── tablet_loo_split.py # tablet-LOO CV
└── lm/
├── train_kenlm.py # 3-gram TLHdig LM
└── viterbi_rescore.py # Beam Viterbi
🏁 SONUÇ ÖZETİ (tek cümle)
DINOv3-L baseline 0.634 → v8 ultimate 0.708 EMA (+7.4) / ens_v8 optimized 0.657 (+2.3) / randsplit 0.794 (+16.0, lit-standard) / selective@τ=0.8: 0.943 (practical %90+).
Paper için ana mesajlar
- Unsupervised clustering + labeled anchoring → 36,177 pseudo-label (%19 veri artışı) → v7_anchor +4.9 puan
- Noisy student iter 2 → v9 (RUNNING, beklenen +2-3 puan)
- Random stratified split ile 0.794 (DeepScribe seviyesi)
- Selective accuracy @ τ=0.8 = %94.3 (rejection-based use case)
- 13 post-hoc head + LightGBM stacking → ensemble lift +1.3-1.5 puan
%90 Hedefi — gerçek tablo
| Metrik | Şu an | %90 için gereken |
|---|---|---|
| Pure top-1 (198 cls, tablet fold) | 0.657 | Val cleanup + random split = ~0.85-0.88 muhtemel |
| Pure top-1 (randsplit) | 0.794 | +ensemble ile 0.82-0.87, tek model zorlayıcı |
| Selective top-1 (τ=0.8, tablet fold) | 0.943 | ✅ Zaten %90+ |
| Selective top-1 (τ=0.7) | 0.921 | ✅ Zaten %90+ |
Sonuç: Pure top-1 %90 tablet fold'da zor; ama random split veya selective metric ile zaten hedefteyiz.