lattice-retrieval

lattice-retrieval is a fast English static embedding model for retrieval. It is a single 30,522 × 1,024 token embedding table with mean pooling: no transformer layers, attention, or contextual token representations.

With a dedicated pure-Rust runtime it can embed all 6.4M articles in English Wikipedia in 7 minutes and 26 seconds (benchmarked on an 8-core Apple M2 MacBook Air).

The full details about training, data loading, evaluation, quantization, and the optimized runtime, are in the Lattice blog post.

Results

Lattice was pretrained on roughly 660M query/document pairs and then fine-tuned with hard negatives. Compared to the previously best performing static embedding model, this one is trained on roughly eight times more data and scores 0.0415 higher on Decontaminated BEIR NDCG@10.

Model	Pretraining pairs	Dimensions	Decontaminated BEIR NDCG@10
static-retrieval-mrl-en-v1	~80M	1,024	0.4334
lattice-retrieval (fp32)	~660M	1,024	0.4749

Slicing and quantization

Only the canonical fp32 model is published in this repository. The model was trained with Matryoshka objectives, which means it can be truncated to keep only the first 512, 256, 128, 64 or 32 dimensions.

To make lattice-retrieval even more compact, I ran experiments on int8, int4 and int2 post-training quantization schemes. The GitHub repo contains a CLI tool that:

• downloads the full fp32 model from this repository;
• quantizes and truncates the weights according to your configuration;
• saves the new artifact, which can then be used directly in your deployment.

This lets you choose the dimension and quantization scheme that fits your application. Here are a few representative combinations:

Dimensions	Quantization	Model weights	Decontaminated BEIR NDCG@10	Throughput
1,024	fp32	125.02 MB	0.4749	7.55M tokens/sec
1,024	int8 per-dim	31.26 MB	0.4747	8.43M tokens/sec
512	int8 per-dim	15.63 MB	0.4700	9.05M tokens/sec
512	int4 per-row	7.94 MB	0.4697	8.76M tokens/sec
256	int8 per-dim	7.82 MB	0.4624	9.38M tokens/sec
128	int4 per-dim	1.95 MB	0.4312	9.43M tokens/sec

Per-row quantization stores a separate scale for each token vector, while per-dim quantization stores a scale for each embedding dimension. Per-row usually preserves more quality; per-dim is simpler and somewhat faster.

Throughput was measured over the same 5,000-article corpus with 12 workers on an 8-core Apple M2, so it should be read as a comparison between variants rather than a universal hardware benchmark.

Usage

pip install -U "sentence-transformers>=5.4"

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("erikkaum/lattice-retrieval")
embeddings = model.encode(
    ["hello world", "static embeddings are fast"],
    normalize_embeddings=True,
)

You can truncate embeddings directly with Sentence Transformers:

embeddings_256 = model.encode(
    ["hello world"],
    normalize_embeddings=True,
    truncate_dim=256,
)

To run quantized and truncated versions of the model with a pure-Rust implementation, see the instructions in the lattice repository.

Limitations

This is a bag-of-token-vectors model. It does not contextualize tokens and is substantially less capable than a strong transformer retriever at word order, polysemy, compositional meaning, and fine semantic distinctions. It should not be presented as a drop-in quality replacement for a transformer model.

Acknowledgements

Training data, hard-negative fine-tuning, and the decontaminated evaluation come from LightOn's DenseOn/LateOn release. The model builds on Sentence Transformers' static-retrieval-mrl-en-v1.