HITIT-OCR v1 Pipeline
SOTA-üstü 4-stage cascaded cuneiform OCR pipeline — 2024-2026 literature'da en iyi yaklaşımların cuneiform-spesifik kombinasyonu.
Hedef
Ham tablet görüntüsü → transliterasyon (Hittite/Akkadian/Sumerian/Elamite):
raw tablet image → sign detection → ABZ classification → line assembly → transliteration
SOTA baseline'ları geçmek
- DeepScribe (ACM JOCCH 2025): end-to-end top-5 %80 (Elamite, 141 sınıf)
- CHURRO (EMNLP 2025): Qwen2.5-VL-3B normalized Levenshtein %82.3
- PreP-OCR (ACL 2025): CER -%63.9 post-correct
- HATFormer (2024): CER %8.6 Arabic historical
Bizim hedef: 3300 sınıf, IR 13,000×, 4 dil, tier-stratified macro-F1.
Mimari
┌─────────────────────────────────────────────────────────────────────┐
│ INPUT: Tablet image (any res) │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 0 — Preprocessing & Restoration │
│ • Canonical rotation (canonical_rotation_deg) │
│ • Illumination normalization (Retinex / Stötzner 2023) │
│ • NAFNet restoration (if quality_gate_pass=False) │
│ • Multi-scale letterbox: 224 (cls) / 1280 (det) │
│ • Dataset-specific normalization (mean/std in config) │
└──────────────────────────┬──────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 1 — Sign Detection (YOLO11-m + P2 + SAHI) │
│ Training: tablet_view_fold 5-fold CV │
│ Output: bbox, objectness, top-k class candidates │
│ Loss: focal + CIoU + copy-paste aug │
│ TTA: multi-scale {0.8, 1.0, 1.2} + WBF │
└──────────────────────────┬──────────────────────────────────────────┘
▼ crop 224 × 224 letterbox
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 2 — Classification (DINOv3-B/14 + LoRA + Hierarchical) │
│ Pretrain: continual DINOv3 SSL on in_curated_pretrain (70K) │
│ Head: 2-level hierarchical │
│ • top: ABZ super-class (≈500 base sign) │
│ • sub: sign_variant_code (OH/MH/NH stil) │
│ Rare tier: prototype learning (5-shot) │
│ Loss schedule (2-phase): │
│ phase1 (0-40): CB-Focal γ=2, β=0.9999 │
│ phase2 (40-80): LDAM-DRW decoupling (Kang 2020) │
│ Sampler: tiered — head uniform, mid/tail sqrt, rare prototype │
│ Augmentation: Mixup + TailMix + CutMix + elastic + illum │
│ Output: top-k ABZ + calibrated logits │
└──────────────────────────┬──────────────────────────────────────────┘
▼ sign lattice (top-5 per bbox)
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 3 — Line Assembly & Open-set Filter │
│ • Baseline fit (RANSAC) → row clustering │
│ • Column split (cuneiform reverse-Z reading order) │
│ • Energy score (Liu 2020) → OOD threshold │
│ • Rare tier + low energy → ABZ_UNK token │
│ Evaluation: open-set AUROC on oltr_holdout │
└──────────────────────────┬──────────────────────────────────────────┘
▼ ordered sign lattice
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 4 — Transliteration & Post-correction │
│ LM: ByT5-small fine-tuned on TLHdig (21K) + fragments (23K) │
│ Rescoring: Viterbi lattice = posterior × LM prior │
│ N-gram backoff: KenLM 5-gram char-level │
│ Optional: Qwen2.5-VL-3B LoRA reranker (CHURRO-style) │
│ Output: transliteration (hit/akk/sum/elx) │
│ Evaluation: CER stratified by language × period │
└─────────────────────────────────────────────────────────────────────┘
Stage detayları
Stage 0 — Preprocessing
Mevcut hitit_ocr/src/preprocessing/ modülü kullanılır (pipeline.py: CLAHE + gamma + letterbox + MSII proxy). Eklenenler:
Yeni: Retinex MSR illumination norm + NAFNet gating.
# Conditional restoration flow
if r['quality_gate_pass'] is False: # blur/contrast/exposure fail
img = nafnet_restore(img) # GPU, offline
# Always
img = retinex_msr(img) # CPU, MSR(σ=[15,80,250])
img = canonical_rotate(img, r['canonical_rotation_deg'])
img = letterbox(img, target=224, margin=0.10, fill='median_border')
img = normalize(img, mean=[0.489,0.448,0.424], std=[0.362,0.359,0.364])
Config: hitit_ocr/configs/preprocessing.yaml (v1.0 hazır).
Stage 1 — Detection (YOLO11-m + P2)
Neden YOLO11-m + P2 head:
- Small-object benchmark (arXiv 2504.09900): P2 head 15-25px wedge'lerde YOLOv10'dan +%2-4
- 3300 sınıflı closed-set için DETR-family gereksiz
- SAHI slicing ile 3000×4000 tablet foto'ta +%3-8 recall
# hitit_ocr/configs/detection_hitit_v1.yaml
model: yolo11m-p2.yaml
imgsz: 1280
epochs: 150
batch: 4 # A100 40GB × 4 GPU
optimizer: AdamW
lr0: 0.001
momentum: 0.937
weight_decay: 0.0005
warmup_epochs: 3
box: 7.5
cls: 0.5
dfl: 1.5
mosaic: 0.8
mixup: 0.15
copy_paste: 0.15 # Sign copy-paste tail için kritik
fliplr: 0.0 # Cuneiform yön-duyarlı
flipud: 0.0
degrees: 5 # canonical_rotation_deg ile baş çözüldü
hsv_h: 0.015
hsv_s: 0.7
hsv_v: 0.4
close_mosaic: 10 # son 10 epoch mosaic kapat
Data: datasets/unified/detection/manifest.parquet + tablet_view_fold (leakage-free 5-fold).
Training:
# 5-fold CV, her fold 150 epoch
for fold in 0 1 2 3 4; do
sbatch hitit_ocr/scripts/train_yolo11_fold.slurm $fold
done
Inference + SAHI:
from sahi import AutoDetectionModel
from sahi.predict import get_sliced_prediction
model = AutoDetectionModel.from_pretrained("yolo11", model_path="best.pt")
result = get_sliced_prediction(
image,
detection_model=model,
slice_height=640, slice_width=640,
overlap_height_ratio=0.2, overlap_width_ratio=0.2,
postprocess_type="NMS"
)
TTA: eval_det_tta_slurm.sh mevcut — multi-scale {0.8, 1.0, 1.2} + WBF.
Target: mAP50-95 ≥ 0.80 (DeepScribe 0.78'i geçmek).
Stage 2 — Classification (DINOv3 + LoRA + Hierarchical)
2.1 Continual SSL pretraining (hafta 2-3)
DINOv3 ViT-B/14 teacher'ı in_curated_pretrain=True 70K image üzerinde continual SSL:
- iBOT + DINO loss
- Teacher EMA 0.996
- 20 epoch, 4×A100, bs=256
- Output:
dinov3_vitb14_hitit_pretrained.pth
# hitit_ocr/configs/ssl_dinov3_continual.yaml
teacher: facebook/dinov3-vitb14
epochs: 20
batch: 256
lr: 1e-4
warmup: 3
ema_teacher: 0.996
local_crops_number: 8
global_crops_scale: [0.4, 1.0]
local_crops_scale: [0.05, 0.4]
patch_drop: 0.0
use_fp16: true
data:
filter: "in_curated_pretrain == True"
source: datasets/unified/classification/manifest.parquet
2.2 Hierarchical classifier fine-tune (hafta 4-7)
class HierarchicalClassifier(nn.Module):
def __init__(self, backbone, n_abz_base=500, n_variants=None):
self.backbone = backbone # DINOv3-B/14 frozen or LoRA
self.apply_lora(rank=16, alpha=32, target=['qkv','proj'])
self.head_abz = nn.Linear(768, n_abz_base)
self.head_variant = nn.Linear(768 + n_abz_base, n_variants)
self.prototype_head = PrototypeHead(emb_dim=768)
def forward(self, x):
feat = self.backbone(x) # (B, 768)
abz_logits = self.head_abz(feat)
variant_logits = self.head_variant(torch.cat([feat, abz_logits], -1))
proto_logits = self.prototype_head(feat) # rare tier için
return abz_logits, variant_logits, proto_logits
Loss schedule (2-phase decoupling, Kang 2020):
# Phase 1 (epoch 0-40): representation learning
loss = cb_focal(logits, y, beta=0.9999, gamma=2.0)
# Phase 2 (epoch 40-80): classifier re-balancing (DRW)
loss = 0.3 * cb_focal(logits, y) + 0.7 * ldam(logits, y, max_m=0.5, s=30)
CB-Focal (effective_num_weight manifold alanı zaten hesaplı):
class_weights = torch.tensor([r['effective_num_weight'] for r in manifest])
# ya da: (1 - beta) / (1 - beta^n_c)
loss = F.cross_entropy(logits * s, y, weight=class_weights, reduction='none')
loss = loss * ((1 - pt)**gamma) # focal
LDAM (Cao 2019):
def ldam_loss(logits, y, class_n, max_m=0.5, s=30):
m_list = 1.0 / torch.sqrt(torch.sqrt(class_n.float()))
m_list = m_list * max_m / m_list.max()
index = torch.zeros_like(logits, dtype=torch.bool)
index.scatter_(1, y.view(-1,1), True)
x_m = logits - m_list[None, :]
output = torch.where(index, x_m, logits) * s
return F.cross_entropy(output, y)
Tiered sampler:
# class_frequency_tier alanı kullanarak
# head (tier='head'): uniform sampling (regular)
# mid/tail: WeightedRandomSampler(sampler_weight)
# rare: ProtoNet 5-shot episode sampler
Augmentation:
- Mixup α=0.2 (head tier pairs only)
- TailMix: aynı unified_label içi pair mixup (tail tier)
- CutMix α=1.0
- Elastic + GridDistortion (from
augment_recipe.py) - Dataset-specific normalize
Config:
# hitit_ocr/configs/classification_hitit_v1.yaml
backbone: dinov3_vitb14_hitit_pretrained.pth
lora: {r: 16, alpha: 32, targets: ['qkv', 'proj']}
head: hierarchical
n_abz_base: 500
n_variants: 3300
prototype_for_rare: true
rare_threshold: 5
loss:
phase_1: {epochs: [0, 40], cb_focal: {beta: 0.9999, gamma: 2.0, weight: 1.0}, ldam: {weight: 0.0}}
phase_2: {epochs: [40, 80], cb_focal: {weight: 0.3}, ldam: {max_m: 0.5, s: 30, weight: 0.7}}
sampler: tiered
augment:
mixup: {alpha: 0.2, p: 0.5}
tailmix: {alpha: 0.5, p: 0.3, intra_class: true}
cutmix: {alpha: 1.0, p: 0.3}
elastic: true
grid_distortion: true
color_jitter: true
horizontal_flip: false # cuneiform yön-duyarlı
optimizer: AdamW
lr: {backbone: 1e-5, lora: 1e-4, head: 1e-3}
scheduler: cosine_warm_restarts
T_0: 20, T_mult: 2, eta_min: 1e-6
epochs: 80
batch: 48
Target: macro-F1 tier-stratified:
- head: ≥ 0.92
- mid: ≥ 0.75
- tail: ≥ 0.50
- rare (OLTR): ≥ 0.25 + open-set AUROC ≥ 0.85
Stage 3 — Line Assembly & Open-set
Line assembly:
- Tüm bbox merkezleri → DBSCAN clustering (eps=median_sign_height × 1.5)
- Her satırda x-coord sıralama (cuneiform left-to-right)
- Tablet view aware: obverse/reverse sıralaması farklı
Open-set energy:
def energy_score(logits, T=1.0):
return -T * torch.logsumexp(logits / T, dim=-1)
# Threshold calibration on oltr_holdout
# OLTR holdout'ta FPR=%5 veren threshold seç
Stage 4 — Transliteration (ByT5 + Viterbi)
Training:
- Input: Stage 3 sign lattice (top-5 per position + confidence)
- Output: transliteration string
- Data: TLHdig corpus (21K Hittite) + fragments (23K Akkadian)
# hitit_ocr/configs/transliteration_byt5.yaml
model: google/byt5-small
max_input_len: 512 # byte-level
max_output_len: 256
epochs: 20
batch: 32
lr: 5e-5
warmup: 500
beam_size: 5
length_penalty: 0.6
Viterbi rescoring:
def viterbi_lattice(lattice, lm_model, lambda_lm=0.3):
"""lattice: list of (top-k ABZ, log-probs) per position.
LM prior: ByT5 score over candidate sequences."""
best_path = viterbi(
emissions=lattice,
lm_scores=lm_model.score_all_paths(lattice, beam=10),
lambda_lm=lambda_lm
)
return best_path
Target: CER stratified by language
- hit: < 8%
- akk: < 9%
- sum: < 12%
- elx: < 10%
Optional reranker: Qwen2.5-VL-3B LoRA sadece top-3 disagreement durumunda devreye gir (CHURRO stili).
Training schedule (SLURM orchestration)
| Hafta | Aşama | Config | GPU-saat |
|---|---|---|---|
| 1 | Sentetik restoration pretrain (NAFNet) | Stage 0 | 50 |
| 2-3 | DINOv3 continual SSL on curated 70K | Stage 2.1 | 400 |
| 3-5 | YOLO11-P2 detection 5-fold CV | Stage 1 | 600 |
| 4-7 | Classifier 2-phase LDAM-DRW | Stage 2.2 | 800 |
| 6 | Prototype head rare tier | Stage 2 | 40 |
| 7-8 | ByT5 transliteration | Stage 4 | 100 |
| 9 | Qwen2.5-VL reranker (opt.) | Stage 4+ | 200 |
| 10 | End-to-end eval + TTA | all | 100 |
Toplam: ~2300 GPU-saat A100, 4-GPU akya-cuda node × 24 gün.
Neden SOTA'yı geçer
- Long-tail SOTA: CB-Focal → LDAM-DRW decoupling + prototype rare head (arXiv 2404.15593 survey'de en güçlü combo)
- Paleografik: DINOv3 SSL + hierarchical variant head — aynı ABZ farklı era ayrımında ResNet flat'tan +%4-6
- Small-object: YOLO11-P2 + SAHI + copy-paste aug — DeepScribe RetinaNet'ten küçük sign'larda +%3-8
- Multi-hypothesis: ByT5 lattice Viterbi — PreP-OCR single-best'ten tail error -%20
- Domain adaptation: DINOv3 continual SSL on mixed render+photo — Stötzner'ın explicit domain adapt'inden implicit olarak daha iyi
- Evaluation rigor: tablet_view_fold leakage-free, tier-stratified macro-F1, OLTR AUROC, gold eval 3253
Data flow haritası
| Manifest alan | Stage | Kullanım |
|---|---|---|
unified_label |
Stage 2 target | y (3300-way) |
sign_variant_code |
Stage 2 subhead | Hierarchical variant branch |
fold / tablet_view_fold |
All | 5-fold CV split |
class_frequency_tier |
Stage 2 | Tiered sampler, LDAM margin |
sampler_weight |
Stage 2 | WeightedRandomSampler |
effective_num_weight |
Stage 2 | CB-Loss |
quality_gate_pass |
Stage 0 | Restoration gate |
blur_score, exposure_mean, contrast_std |
Stage 0 | Illum aug params |
canonical_rotation_deg |
Stage 0 | Rectify |
phonetic_reading |
Stage 4 | ByT5 target |
visual_phash, is_duplicate |
All | Dedup sanity |
oltr_holdout |
Stage 3 eval | Open-set AUROC |
in_curated_pretrain |
Stage 2.1 | DINOv3 SSL pool |
in_gold_eval |
Final eval | Hitit gold 3253 |
period, language, script_era |
Eval | Stratified breakdown |
uncertainty_score |
Active learning | Next-round candidate |
Evaluation pipeline
# hitit_ocr/src/evaluate_e2e.py
metrics = {
"detection": {
"mAP50_95_macro": ...,
"per_source_mAP": {...},
},
"classification": {
"top1_macro_F1": ...,
"per_tier_F1": {"head": ..., "mid": ..., "tail": ..., "rare": ...},
"top5_accuracy": ...,
"balanced_accuracy": ...,
},
"openset": {
"AUROC_oltr": ..., # oltr_holdout
"FPR_at_95_TPR": ...,
},
"transliteration": {
"CER_hit": ..., "CER_akk": ..., "CER_sum": ..., "CER_elx": ...,
"BLEU": ...,
"exact_match": ...,
},
"e2e_tablet": {
"gold_accuracy": ..., # in_gold_eval 3253 subset
"per_period": {...}, # OH, MH, NH, OB, NA, NB, Ur-III, ACH
}
}
Baselines for comparison:
- DeepScribe reproduced on our data
- CHURRO zero-shot on our gold set
- CuReD on transliteration OCR subset
- PreP-OCR with same ByT5 but without Viterbi