Singaraj commited on
Commit
fe1f289
·
verified ·
1 Parent(s): 55c2406

Haitian figures from the committed probe; STS-b and truncation limitations

Browse files
Files changed (1) hide show
  1. README.md +11 -6
README.md CHANGED
@@ -129,14 +129,19 @@ Limitations come from an internal adversarial audit of the released checkpoint.
129
  independent evaluation domain for `mfe` (FLORES+) is saturated at this corpus size — so the margin
130
  over LaBSE is demonstrated in-domain only.
131
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
132
- Creole close to Mauritian Creole: with same-meaning Haitian sentences injected into a FLORES-based
133
- corpus, mfe→eng accuracy@1 drops from 1.00 to 0.71 (the Haitian twin outranks the English
134
- translation) — and LaBSE degrades less on this same trap (to 0.79). On a related eng→{mfe, hat}
135
- discrimination test the fine-tune picks the correct Mauritian translation 306/400 times vs LaBSE's
136
- 170/400, and wrong-meaning Haitian text is never confused but mixed mfe/hat corpora will degrade
137
- retrieval.
138
  - **Case sensitivity.** ALL-CAPS text embeds measurably differently from its lower-case form
139
  (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
 
 
 
 
 
140
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
141
  training run, so recipe selection had test visibility; an internal adversarial audit bounded the
142
  resulting optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering
 
129
  independent evaluation domain for `mfe` (FLORES+) is saturated at this corpus size — so the margin
130
  over LaBSE is demonstrated in-domain only.
131
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
132
+ Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest
133
+ triplets (`scripts/probe_haitian.py`): with every same-meaning Haitian twin injected into the
134
+ corpus, mfe→eng accuracy@1 drops from 1.00 to 0.68 — and LaBSE resists this trap better (0.79).
135
+ Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or
136
+ its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012.
137
+ Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
138
  - **Case sensitivity.** ALL-CAPS text embeds measurably differently from its lower-case form
139
  (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
140
+ - **English-only regression.** Fine-tuning costs some pure-English semantic quality: STS-b test
141
+ Spearman ≈ 0.79 vs the base model's ≈ 0.85. Use a general model for English-only workloads; this
142
+ model is for Creole and Creole↔{English, French} work.
143
+ - **Long inputs are truncated** at the encoder's maximum sequence length; chunk long documents
144
+ before embedding.
145
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
146
  training run, so recipe selection had test visibility; an internal adversarial audit bounded the
147
  resulting optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering