eoinedge/zephyrproject-docs-embeddings

Zephyr RTOS documentation embeddings

all-MiniLM-L6-v2 fine-tuned on the Zephyr RTOS documentation, for retrieval over that corpus.

• Comparison Space: https://huggingface.co/spaces/eoinedge/zephyrproject-embeddings-compare
• Index and pipeline: https://huggingface.co/eoinedge/zephyrproject
• Source: https://github.com/eoinjordan/zephyrproject-rag

What it is for

The base model is trained on general web text, where "binding" means a contract and "west" is a direction. This copy is adapted to Zephyr RTOS's vocabulary so those senses separate.

Use it as a drop-in replacement for the base model when embedding Zephyr RTOS documentation:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("eoinedge/zephyrproject-docs-embeddings")
vectors = model.encode(passages, normalize_embeddings=True)

Training

Base	sentence-transformers/all-MiniLM-L6-v2
Pairs	6,079 mined from the docs' own heading/body structure
Loss	MultipleNegativesRankingLoss (in-batch negatives)
Epochs	1
Batch size	32
Held-out pairs	400

No hand-labelled data and no synthetic questions from a generator — a section heading is a natural query for the body beneath it, and it is already written by the documentation authors.

Results, and how to read them

Metric	Base	Tuned
recall@5 on held-out pairs	0.63	0.86

That number flatters the model, and you should treat it with suspicion. The evaluation measures the same heading→body relationship the model was trained on, so it partly measures whether training converged rather than whether retrieval improved.

Spot-checking real questions gives a more mixed picture: some clearly improve, several are unchanged, and at least one regressed. Real questions are not phrased like section headings, which is exactly the gap this evaluation does not cover.

The comparison Space runs both models on the same query so you can judge for yourself rather than trusting the headline.

Limitations

• One epoch, one seed, no hyperparameter search.
• Evaluation is aligned with the training objective, as above.
• Trained on Zephyr RTOS documentation only — it will not help on other corpora and may be worse than the base model there.
• This adapts the retriever, not a generator. It does not teach a coding model anything about Zephyr RTOS; it changes which passages that model is handed.

Licence

Apache-2.0, matching the base model.