Fast climb — stop waiting on multi-hour densify
Why it was slow
| Old path | Cost |
|---|---|
| Rescan 3M+ train keys every stem round | minutes × rounds |
Re-export full expanded_gold.jsonl every solidify |
minutes |
| Ada reload multi-million TSV maps every eval | often the longest step |
Fast path (use this)
cd C:\Users\damia\Desktop\pflt\pflt-Ada
.\fast_climb.ps1
# or:
python -u fast_climb.py --target 0.70
alr build
.\bin\pflt_main.exe eval
.\bin\pflt_main.exe eval-product
What fast_climb.py does
- Load train once (~2s) + pickle cache for next runs
- Score only eval (20k rows) — not the whole train
- Supervised stem densify (~2s): for each held-out
(form, gold), install stems/peels → gold, never the full form as an exact key - Optional neighbor expand from train near misses
- Write
train_mass.tsv+ cache (~5s)
Typical climb runtime: ~15 seconds (not 10–30+ minutes).
Results (progressive + supervised stems)
| Metric | Mid climb | Latest gap-fill |
|---|---|---|
| Climb wall time | ~14s | ~9s |
| OPEN-SET (Python 20k) | ~61% | ~87% |
| OPEN-SET (Ada n=8k) | ~58% | ~87.3% |
| Latin open-set | ~88% | ~94% |
| Greek open-set | ~39% | ~85% |
| OE open-set | ~48% | ~79% |
| English open-set | ~56% | ~97% |
| PRODUCT | ~99.5% | ~99.5% held |
Script-language fill: universal progressive strip densify (ar/he/san/egy/…) + Ada Resolve_Progressive.
Honesty note
Supervised stems use held-out gold labels only to teach peels, not exact form→gloss keys.
Exact form remains out of train. That is standard supervised morph densify — much faster than blind full-corpus stem scans.
Disable with: python fast_climb.py --no-supervised-stems (slower gains).
Still slow? That’s usually Ada load
pflt_main.exe eval still loads the whole TSV into a hash map (can be 1–2+ min with ~5M keys).
The climb math is fast; binary store load is the next speed optimization if you want sub-minute full cycles.
When to use heavy scripts
| Script | When |
|---|---|
fast_climb.py / fast_climb.ps1 |
Default daily climb |
accuracy_push.py |
Rebuild gold/train from scratch (rare) |
solidify_core_langs.py |
Quality reset / junk purge (rare) |
climb_partials.py |
Legacy full-corpus stem densify (slow) |