Beemer Claude Fable 5 commited on
Commit ·
8878510
1
Parent(s): be40f45
RUNBOOK: record the rejected multilingual-e5 trial with its numbers
Browse files- RUNBOOK.md +10 -4
RUNBOOK.md
CHANGED
|
@@ -115,10 +115,16 @@ git push mirror --all
|
|
| 115 |
detection) was considered and deliberately skipped: ~6 minutes of
|
| 116 |
rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
|
| 117 |
a decision before relying on it instead.
|
| 118 |
-
- French support is phase 1
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
- Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
|
| 123 |
(0.6 floor, shared 2-char prefix, 0.7 score discount) in
|
| 124 |
`index._fuzzy_term`; the `typo` eval slice tracks it.
|
|
|
|
| 115 |
detection) was considered and deliberately skipped: ~6 minutes of
|
| 116 |
rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
|
| 117 |
a decision before relying on it instead.
|
| 118 |
+
- French support is phase 1 (accent-folded tokens + FR→EN synonym bridges;
|
| 119 |
+
`french` eval slice: Hit@3 0.80). Phase 2 — swapping to
|
| 120 |
+
multilingual-e5-small via the `CANLEX_EMB_REPO` env hooks — was TRIED AND
|
| 121 |
+
REJECTED 2026-07-22: it regressed every slice (legacy MRR 0.87→0.82,
|
| 122 |
+
french Hit@3 0.80→0.60, since the synonym bridges already anglicize the
|
| 123 |
+
query) and broke confidence calibration (no-answer probes 0.67→0.84,
|
| 124 |
+
above the 0.72 hedge). Don't retry without also re-calibrating
|
| 125 |
+
HEDGE_THRESHOLD and adding untranslatable French queries to the eval.
|
| 126 |
+
Phase 3 (parallel French corpus from the bilingual Justice Laws XML)
|
| 127 |
+
remains the real path to full bilingual support.
|
| 128 |
- Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
|
| 129 |
(0.6 floor, shared 2-char prefix, 0.7 score discount) in
|
| 130 |
`index._fuzzy_term`; the `typo` eval slice tracks it.
|