rolf-mozilla/translations-en-fr-static

Translations en-fr static embeddings

Static sentence embeddings derived from the Mozilla Translations en-fr Marian student model's encoder embedding table (Wemb), for use with Firefox's static-embeddings ML backend.

Unlike the other models in this repo, these vectors are not distilled from a sentence encoder. They are the NMT encoder's token embeddings with SIF weighting baked in, plus a set of principal directions to project out at inference.

• dimensions: 384
• vocab: 32,000 (SentencePiece Unigram, shared en-fr vocab)
• source model: models/en-fr/retrain_hr_NLIxDbE1TBGyOTI-zwZagw/student/final.model.npz.best-chrf.npz
• source vocab: vocab.en.spm from the same run

Firefox configuration

staticEmbeddingsOptions: {
  subfolder: "models/mozilla/translations-en-fr-static",
  dtype: "fp8_e4m3",
  dimensions: 384,
  compression: true,
  commonComponents: 8,
}

commonComponents requires Firefox with common-component support in StaticEmbeddingsPipeline. Older builds ignore the option and will not fetch ccr.d384.npy, which yields the "no CCR" numbers below.

Files

file	size (zst)	notes
tokenizer.json.zst	297 KB	required
fp8_e4m3.d384.npy.zst	9.2 MB	recommended dtype
fp8_e5m2.d384.npy.zst	8.1 MB	smaller, slightly worse
fp16.d384.npy.zst	22 MB
fp32.d384.npy.zst	44 MB	reference
ccr.d384.npy.zst	22 KB	16 principal directions

Only the files matching the configured dtype are fetched at runtime.

ccr.d384.npy

A [16, 384] float32 array of orthonormal principal directions. The pipeline subtracts the projection onto the first commonComponents rows from each pooled embedding, before L2 normalization:

v' = v - sum_i <v, u_i> u_i

This is step (b) of SIF (Arora et al. 2017); a plain mean-pool implements only step (a). Averaging token vectors leaves a large shared component in every embedding, so unrelated texts look similar — mean cosine between unrelated documents is 0.56 before removal and 0.43 after.

Rows are ordered by decreasing variance explained: 7.9%, 6.7%, 2.9%, 2.5%, 2.1%, 1.8%, 1.7%, 1.4% (20.0% cumulative at k=4, 26.9% at k=8). More rows are shipped than the recommended commonComponents, so the value can be retuned without a re-export.

The directions were fit by replicating the runtime path exactly — tokenize with tokenizer.json, gather rows, unweighted mean, no normalization — over a bilingual corpus (5,000 EN + 5,000 FR history-style documents), then centering and taking the top right singular vectors. They are dimension-specific: a 384-d direction truncated to 128 is not the principal direction of the 128-d space.

Weighting

Per-piece SIF weights a / (a + p) with a = 3e-4 are baked into the stored vectors, because the pipeline pools with an unweighted mean. Baking is exact rather than an approximation: pooling gives (1/N) * sum(w_i * v_i) where true SIF gives sum(w_i * v_i) / sum(w_i), and those differ only by a positive scalar that L2 normalization removes.

Note a = 3e-4, not the 1e-3 used by TranslationModel::embed in the bergamot engine. Embeddings from that engine are therefore not interchangeable with these, and it does not apply common-component removal. Do not mix the two in one index.

Evaluation

Semantic history search, per-profile corpora (5 profiles x 1,000 docs, 500 queries), scored from these exact artifacts. Metric is nDCG@10.

dtype	FR, no CCR	FR, CCR k=8	EN, no CCR	EN, CCR k=8
fp32	0.6238	0.6589	0.7194	0.7655
fp16	0.6238	0.6589	0.7194	0.7655
fp8_e5m2	0.6253	0.6597	0.7183	0.7668
fp8_e4m3	0.6240	0.6603	0.7186	0.7666

Quantization is essentially free — fp8_e4m3 matches fp32 to within noise — so the 9.2 MB build is the one to use.

For context on the same task, multilingual-e5-small scores 0.8164 FR / 0.9556 EN and all-MiniLM-L6-v2 0.7057 FR / 0.9612 EN. This model is substantially weaker than a transformer encoder; its argument is cost, since it reuses weights the Translations component already ships.

Caveats

1. The French evaluation set is machine-translated from English browsing data, so it is translationese rather than native French browsing. English proper nouns survive translation, which flatters English-centric vocabularies.
2. a and the component count were selected on that same evaluation, so the reported gain is mildly optimistic. The optimum is a broad plateau (a in 1e-4..3e-4, k in 2..8), not a knife edge.
3. Components were fit on a bilingual pooled corpus. A French-only fit scores ~0.011 higher on French but loses ground on English; pooled was chosen because a browser cannot reliably know the language at index time.
4. The Wemb was not trained with Matryoshka loss, so only d=384 is published. Truncating these vectors is not principled.