tardellirs/colibri-embed-ptbr

🐦 Colibri — Brazilian-Portuguese Embeddings

A tiny embedding model that punches far above its weight.

Named after the colibri (hummingbird) — the smallest bird, yet it out-flies much larger ones.

~157M params · 768-dim · plain SentenceTransformer (no adapters) · runs on a cheap CPU VPS

---

Colibri is a compact Brazilian-Portuguese text-embedding model derived from google/embeddinggemma-300m. Despite being half the size, it matches or beats much larger multilingual embedders on MTEB(por) — including 7B and 27B models — and is designed small on purpose so it runs comfortably on a simple CPU VPS, with no GPU required.

Use cases: Portuguese semantic search, retrieval-augmented generation (RAG), retrieval, reranking, clustering, semantic textual similarity (STS) and classification — for Brazilian-Portuguese text across legal, tax, financial, medical, academic and general domains.

✨ Highlights

• 🏆 Beats bigger models on MTEB(por) — including embeddinggemma-300m, and 7B / 27B multilingual embedders.
• 🪶 Half the footprint of embeddinggemma-300m: ~607 MB vs ~1.2 GB, and less RAM.
• 💻 Runs on a $-few/month CPU VPS — benchmarked on 4 vCPU / 16 GB (no GPU).
• 🔌 Plain SentenceTransformer — no adapters, no LoRA at inference. Drop-in.
• 📐 Matryoshka dimensions (768 / 512 / 256 / 128) + an fp16 branch + a fast ONNX path.

---

🥇 Quality — MTEB(por)

Evaluated on MTEB(por) (22 native Brazilian-Portuguese tasks: retrieval, reranking, STS, classification, clustering, pair-classification). Score = mean over the 22 tasks.

Model	Params	MTEB(por)
🐦 Colibri (this model)	~157M	0.6501
google/embeddinggemma-300m	300M	0.6490
Linq-AI-Research/Linq-Embed-Mistral	7B	0.6473
openai/text-embedding-3-large	–	0.6449
intfloat/multilingual-e5-large-instruct	560M	0.6409
Salesforce/SFR-Embedding-2_R	7B	0.6397
Alibaba-NLP/gte-Qwen2-7B-instruct	7B	0.6392
microsoft/harrier-oss-v1-27b	27B	0.6390
BAAI/bge-m3	568M	0.6157
intfloat/multilingual-e5-large	560M	0.5909

A ~157M-parameter model outperforming 7B–27B multilingual embedders on Brazilian Portuguese.

Size ↔ quality frontier. Colibri sits on the open-model Pareto frontier for MTEB(por) — it dominates its own base embeddinggemma-300m (half the size, higher score) and matches models up to ~10× larger:

---

💻 Built for a simple CPU VPS

Colibri was designed to be small enough for cheap, GPU-less hosting. Benchmarked head-to-head against google/embeddinggemma-300m on a Verda CPU.4V.16G (4 vCPU, 16 GB RAM) — a typical small VPS — encoding Brazilian-Portuguese sentences:

Precision	Model	Model size	Peak RAM	Latency (p50)	Throughput
fp32	🐦 Colibri	607 MB	969 MB	76 ms	37 sent/s
fp32	embeddinggemma-300m	1211 MB	1253 MB	78 ms	36 sent/s
ONNX	🐦 Colibri	~600 MB	2.9 GB	33 ms	49 sent/s
ONNX	embeddinggemma-300m	~1200 MB	5.3 GB	38 ms	41 sent/s

Takeaways

• Half the size and less RAM than embeddinggemma-300m — fits a smaller, cheaper VPS.
• Same encode latency at equal precision (the vocabulary trim shrinks the token-embedding matrix, not the transformer compute) — you lose nothing in speed.
• The ONNX backend roughly doubles throughput (33 ms vs 76 ms) — best when latency matters; fp32 is the most RAM-frugal.

Benchmark hardware: Verda CPU.4V.16G. Numbers are indicative single-node CPU measurements.

---

🚀 Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("tardellirs/colibri-embed-ptbr")

query = "Como declarar imposto de renda de aluguel?"
docs = [
    "Rendimentos de aluguel devem ser informados na ficha de Rendimentos Tributáveis...",
    "O Pix é um meio de pagamento instantâneo criado pelo Banco Central...",
]

q = model.encode(query, prompt_name="query")      # "task: search result | query: "
d = model.encode(docs, prompt_name="document")    # "title: none | text: "
print(model.similarity(q, d))

Faster on CPU — ONNX backend (≈2× throughput, no code changes to your pipeline):

model = SentenceTransformer("tardellirs/colibri-embed-ptbr", backend="onnx")

Prompts follow EmbeddingGemma conventions (query / document / classification / clustering / similarity).

---

🎛️ Variants: precision & dimension

Precision — a half-precision branch (same quality, half the size; best for GPU/storage — on CPU it's not faster):

Variant	Load	Size
fp32 (default)	SentenceTransformer("tardellirs/colibri-embed-ptbr")	~607 MB
fp16	SentenceTransformer("tardellirs/colibri-embed-ptbr", revision="fp16")	~304 MB

Matryoshka dimensions — truncate the 768-dim output to 512 / 256 / 128 for smaller vectors (cheaper index, faster search) with a graceful quality trade-off:

model = SentenceTransformer("tardellirs/colibri-embed-ptbr", truncate_dim=256)

(The MTEB(por) score above is for the full 768-dim embeddings.)

---

🔧 How it was built

A three-stage pipeline: vocabulary trimming → knowledge distillation → model soup.

1. Vocabulary trimming. tardellirs/embeddinggemma-pt-br trims embeddinggemma-300m's 262k multilingual vocabulary down to a ~64k Brazilian-Portuguese vocabulary, cutting the model from 300M to **157M effective parameters** with negligible quality loss. Code: github.com/tardellirs/embedding-vocab-trimmer.

2. Knowledge distillation. Relational (similarity-preserving) KD from two complementary teachers — Qwen/Qwen3-Embedding-4B (strong at clustering) and Qwen/Qwen3-Embedding-8B (strong at retrieval / reranking). The student learns to reproduce the average of the two teachers' pairwise-similarity matrices (dimension-agnostic; preserves STS). Trained on a diverse ~100k-passage corpus of native Brazilian Portuguese spanning legal, tax, financial, medical, academic, technical, encyclopedic and general-web domains:

Domain	Sources
General web	HuggingFaceFW/fineweb-2 (pt)
Legal	jurisprudencias_stj, leis_ordinarias_1988_2024
Tax	codigo_tributario
Financial regulation	bbrc, bacen2024
Academic	Brazilian scientific abstracts, corpus-carolina
Medical	Brazilian clinical / health texts
Technical	Stack Overflow em Português
Encyclopedic	Wikipedia-PT

Evaluation integrity: the model is trained only on training / non-evaluation splits — every MTEB(por) evaluation example is held out, so the scores reflect generalization, not memorization.

3. Model soup + merge. Distillation checkpoints were combined (model soup / weight averaging) and linearly merged with the trimmed base (θ = 0.35·base + 0.65·soup), with the mixing weight chosen on held-out validation. Both operations yield ordinary weights, so the published model is a single standalone encoder.

---

🔗 Links

• Base: google/embeddinggemma-300m · Trimmed base: tardellirs/embeddinggemma-pt-br
• Trimming code: github.com/tardellirs/embedding-vocab-trimmer
• Distillation teachers: Qwen/Qwen3-Embedding-4B · Qwen/Qwen3-Embedding-8B
• Benchmark: MTEB(por) · CPU benchmark hardware: Verda

---

🙏 Acknowledgments

We gratefully acknowledge Verda for the GPU compute credits that supported this work.

Built with vocabulary trimming · relational knowledge distillation · model soup.