Singaraj commited on
Commit
4d69e01
·
verified ·
1 Parent(s): 75ad30f

Precision fixes from claims audit

Browse files
Files changed (1) hide show
  1. README.md +8 -7
README.md CHANGED
@@ -97,7 +97,7 @@ negative set; Creole→English test ndcg@10 across seeds: **0.9653 ± 0.0002** (
97
  Every number in the tables above is reproducible from the
98
  [training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
99
  `scripts/evaluate.py --truncate-dim`). The Haitian-proximity and case-sensitivity figures under
100
- Limitations come from an adversarial audit of the released checkpoint.
101
 
102
  ## Training
103
 
@@ -113,7 +113,7 @@ Limitations come from an adversarial audit of the released checkpoint.
113
  checkpoint's contrastive stage trained on those 24,100 tuples (the stage-1 mining model itself was
114
  trained on all 35,064). Contrastive training uses `CachedMultipleNegativesRankingLoss` (batch 128,
115
  767 in-batch negatives per anchor) wrapped in `MatryoshkaLoss` (dims 768/512/256/128/64). 3 epochs,
116
- lr 2e-5, warmup 10%, fp16, seed 42, single T4 GPU (~30 min).
117
  - **Base model:** [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base)
118
  (278M parameters, MIT).
119
 
@@ -131,14 +131,15 @@ Limitations come from an adversarial audit of the released checkpoint.
131
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
132
  Creole close to Mauritian Creole: with same-meaning Haitian sentences injected into a FLORES-based
133
  corpus, mfe→eng accuracy@1 drops from 1.00 to 0.71 (the Haitian twin outranks the English
134
- translation). The fine-tune still discriminates the two creoles better than LaBSE does on the same
135
- trap (306/400 correct vs LaBSE's 170/400), and wrong-meaning Haitian text is never confused — but
136
- mixed mfe/hat corpora will degrade retrieval.
 
137
  - **Case sensitivity.** ALL-CAPS text embeds measurably differently from its lower-case form
138
  (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
139
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
140
- training run, so recipe selection had test visibility; an independent audit bounded the resulting
141
- optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering
142
  reserves the Creole side of every evaluation pair; English/French target texts are not reserved, and
143
  an audit found 1 of 999 benchmark passages also occurring in training as the translation of a
144
  different Creole sentence.
 
97
  Every number in the tables above is reproducible from the
98
  [training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
99
  `scripts/evaluate.py --truncate-dim`). The Haitian-proximity and case-sensitivity figures under
100
+ Limitations come from an internal adversarial audit of the released checkpoint.
101
 
102
  ## Training
103
 
 
113
  checkpoint's contrastive stage trained on those 24,100 tuples (the stage-1 mining model itself was
114
  trained on all 35,064). Contrastive training uses `CachedMultipleNegativesRankingLoss` (batch 128,
115
  767 in-batch negatives per anchor) wrapped in `MatryoshkaLoss` (dims 768/512/256/128/64). 3 epochs,
116
+ lr 2e-5, warmup 10%, fp16, seed 42, single T4 GPU (~30 min contrastive + ~11 min mining).
117
  - **Base model:** [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base)
118
  (278M parameters, MIT).
119
 
 
131
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
132
  Creole close to Mauritian Creole: with same-meaning Haitian sentences injected into a FLORES-based
133
  corpus, mfe→eng accuracy@1 drops from 1.00 to 0.71 (the Haitian twin outranks the English
134
+ translation) and LaBSE degrades less on this same trap (to 0.79). On a related eng→{mfe, hat}
135
+ discrimination test the fine-tune picks the correct Mauritian translation 306/400 times vs LaBSE's
136
+ 170/400, and wrong-meaning Haitian text is never confused — but mixed mfe/hat corpora will degrade
137
+ retrieval.
138
  - **Case sensitivity.** ALL-CAPS text embeds measurably differently from its lower-case form
139
  (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
140
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
141
+ training run, so recipe selection had test visibility; an internal adversarial audit bounded the
142
+ resulting optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering
143
  reserves the Creole side of every evaluation pair; English/French target texts are not reserved, and
144
  an audit found 1 of 999 benchmark passages also occurring in training as the translation of a
145
  different Creole sentence.