code/BEYOND_95_IMPLEMENTATION.md

# BEYOND_95 Implementation Status

5 quick-win strategy için kod + config + scaffolding tamamlandı.
Hepsi **opt-in, bağımsız, config-driven**.

## Implementation özet

| # | Strategy | Status | Files |
|---|---|---|---|
| 1 | End-to-end seq2seq paradigm | ✅ Code + 83 tablet pair | `src/seq2seq/` + `configs/seq2seq_hitit.yaml` |
| 2 | 3D illumination augmentation | ✅ Code + test (5 tablet × 4 angle) | `src/illum_aug/relighting.py` |
| 3 | TLHdig photo-transliteration | ✅ 19.5K tablet ID + email template | `src/tlhdig_integration/` |
| 4 | Image retrieval + exemplar | ✅ Code + template | `src/retrieval/exemplar_matching.py` |
| 5 | Active learning loop | ✅ BUAL + Label Studio/Prodigy | `src/active_learning/al_loop.py` |

## Kritik kazanç noktaları

### 1. Seq2seq paradigm (en büyük potansiyel +%3-7)

**Sonuç**: 83 Hitit tablet için image-transliteration pair hazır.
- `datasets/processed/seq2seq_pairs.jsonl`
- Avg 344 sign/tablet, toplam 28,556 sign
- ViT encoder + ByT5 decoder model template: `src/seq2seq/model.py`
- Config: `configs/seq2seq_hitit.yaml`

**Next**: 
- Training SLURM job (GPU gerekli)
- `python3 src/seq2seq/train.py --config configs/seq2seq_hitit.yaml`
- SumTablets baseline chrF 0.9755

### 2. 3D Illumination augmentation (+%2-4 paper-validated)

**Sonuç**: 2D Lambertian relighting pipeline (3D mesh gerekmeden).
- Depth-from-shading proxy + Sobel normal map
- Per tablet 8 azimuth angle → 8× augment
- Test edildi: 5 tablet × 4 angle = 20 image
- Tam run: 307 × 8 = 2456 augmented image

**Next**: 
- `python3 src/illum_aug/relighting.py --n-angles 8` (background)

### 3. TLHdig entegrasyon

**Sonuç**: 19,549 unique Hitit tablet ID tespit edildi.
- `datasets/processed/tlhdig_tablet_ids.jsonl` 
- Top tabletler: Privat 25, KUB 5.1+, StBoTB 1, Bo 86_299
- Email template: `datasets/processed/hpm_request_template.json`

**Next (kurumsal)**:
- Schwemer (Würzburg) ile contact
- CC-BY lisansı anlaşması
- HPM foto download pipeline aktivasyonu
- Tahmini 5K-20K tablet foto

### 4. Image retrieval + exemplar

**Sonuç**: FAISS-based ExemplarRetriever class.
- Gallery: hitit_local 21K crop
- Query → top-k neighbors + soft vote
- Leakage protection: same-tablet exemplars filtrelenir
- λ_exemplar=0.4 classifier fusion

**Next**:
- DINOv3-L ile 21K embedding (GPU job)
- Build gallery index
- Fuse with 198-class classifier at inference

### 5. Active learning loop

**Sonuç**: BUAL-based weekly iteration pipeline.
- BUAL score = α*entropy + β*disagreement + γ*open_score
- Diverse sampling (KMeans cluster-wise top)
- Prodigy + Label Studio export formats
- 50 sample/hafta → 6 ayda 2000+ yeni

**Next (kurumsal)**:
- Label Studio setup (ücretsiz) veya Prodigy (600€)
- Paleograph engagement (1 saat/hafta)
- Schedule weekly retrain

## Cumulative gain projection (dürüst)

```
v4 baseline (full enhancements):                   93-95%
+ Seq2seq paradigm shift:              +3-7%      → 96-98%  ⭐ en büyük
+ 3D illumination augment:             +2-4%      → 97-98%
+ Exemplar retrieval (rare class):     +1-3%      → 97.5-98.5%
+ Active learning (6 ay birikimli):    +1-3%      → 98-99%
+ TLHdig photo integration:            +1-3%      → 98-99%
```

**Theoretical ceiling**: **%98-99** (198 class Hitit).
**%99+ için**: yıllar boyunca sustained data collection + expert loop.

## Aktivasyon sıralaması

### Hemen başlatılabilir (bugün)
```bash
# Illumination augmentation (CPU, 30-60 dk)
./hitit_ocr/scripts/beyond95_orchestration.sh
```

### GPU job (SLURM)
```bash
# Exemplar gallery build + seq2seq training
sbatch hitit_ocr/scripts/pipeline/train_seq2seq.slurm      # TODO
sbatch hitit_ocr/scripts/pipeline/build_exemplar_gallery.slurm  # TODO
```

### Kurumsal aksiyon (haftalar/aylar)
1. **Schwemer'e email** (hpm_request_template.json) → TLHdig photo access
2. **Label Studio kurulum** → active learning
3. **Paleograph angajman** (ODTÜ/Ankara Üniversitesi)
4. **T.C. Kültür Bakanlığı başvurusu** → müze envanter

## Dosya haritası

```
hitit_ocr/
├── BEYOND_95.md                      # Strateji (13 yol)
├── BEYOND_95_IMPLEMENTATION.md       # BU — status
├── configs/
│   └── seq2seq_hitit.yaml             # Paradigm shift config
├── src/
│   ├── seq2seq/
│   │   ├── model.py                   # ViT + ByT5
│   │   └── build_pairs.py             # image-transliteration extractor
│   ├── illum_aug/
│   │   └── relighting.py              # 2D Lambertian multi-azimuth
│   ├── tlhdig_integration/
│   │   ├── scrape_photos.py           # XML scanner
│   │   └── hpm_web_scrape.py          # HPM URL generator
│   ├── retrieval/
│   │   └── exemplar_matching.py       # FAISS gallery
│   └── active_learning/
│       └── al_loop.py                 # BUAL + export
└── scripts/
    └── beyond95_orchestration.sh       # Master runner

datasets/processed/
├── seq2seq_pairs.jsonl                # 83 tablet
├── tlhdig_tablet_ids.jsonl            # 19,549 Hittite tablet ID
├── tlhdig_photo_pairs.jsonl           # 0 (TLHdig 0.2 yok)
├── illum_aug_map.json                 # augment log
├── hpm_request_template.json          # Schwemer contact
├── hpm_scrape_targets.jsonl           # (requires permission)
├── active_learning_recipe.json
└── tlhdig_download_recipe.json
```

## Bilinen sınırlar

- **TLHdig 0.2 fotoğraf yok**: TLHdig 1.0 (Kasım 2025) veya HPM web scrape gerek
- **Seq2seq training**: 83 Hitit tablet yeter değil, pretraining (SumTablets 91K) lazım
- **Exemplar retrieval**: DINOv3 embedding çıkarımı GPU job gerek
- **Active learning**: Paleograph angajman organizasyonu

## Referanslar

- SumTablets ACL ML4AL 2024 (chrF 97.55)
- Stötzner ICCVW 2023 (+%4 illumination aug)
- Kriege ICFHR 2021 (embedded attributes retrieval)
- Deng BUAL arXiv:2402.15198 (2024)
- Hethitologie-Portal Mainz (Schwemer, Würzburg)