Beemer Claude Fable 5 commited on
Commit
8878510
·
1 Parent(s): be40f45

RUNBOOK: record the rejected multilingual-e5 trial with its numbers

Browse files
Files changed (1) hide show
  1. RUNBOOK.md +10 -4
RUNBOOK.md CHANGED
@@ -115,10 +115,16 @@ git push mirror --all
115
  detection) was considered and deliberately skipped: ~6 minutes of
116
  rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
117
  a decision before relying on it instead.
118
- - French support is phase 1 only (accent-folded tokens + FR→EN synonym
119
- bridges; the `french` eval slice tracks it). Phase 2 = multilingual
120
- embedder via the `CANLEX_EMB_REPO` env hooks; phase 3 = parallel French
121
- corpus.
 
 
 
 
 
 
122
  - Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
123
  (0.6 floor, shared 2-char prefix, 0.7 score discount) in
124
  `index._fuzzy_term`; the `typo` eval slice tracks it.
 
115
  detection) was considered and deliberately skipped: ~6 minutes of
116
  rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
117
  a decision before relying on it instead.
118
+ - French support is phase 1 (accent-folded tokens + FR→EN synonym bridges;
119
+ `french` eval slice: Hit@3 0.80). Phase 2 swapping to
120
+ multilingual-e5-small via the `CANLEX_EMB_REPO` env hooks was TRIED AND
121
+ REJECTED 2026-07-22: it regressed every slice (legacy MRR 0.87→0.82,
122
+ french Hit@3 0.80→0.60, since the synonym bridges already anglicize the
123
+ query) and broke confidence calibration (no-answer probes 0.67→0.84,
124
+ above the 0.72 hedge). Don't retry without also re-calibrating
125
+ HEDGE_THRESHOLD and adding untranslatable French queries to the eval.
126
+ Phase 3 (parallel French corpus from the bilingual Justice Laws XML)
127
+ remains the real path to full bilingual support.
128
  - Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
129
  (0.6 floor, shared 2-char prefix, 0.7 score discount) in
130
  `index._fuzzy_term`; the `typo` eval slice tracks it.