| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| tags:
|
| - zephyr
|
| - sentence-transformers
|
| - feature-extraction
|
| - sentence-similarity
|
| - retrieval
|
| - rag
|
| base_model: sentence-transformers/all-MiniLM-L6-v2
|
| pipeline_tag: sentence-similarity
|
| library_name: sentence-transformers
|
| ---
|
|
|
| # Zephyr RTOS documentation embeddings
|
|
|
| `all-MiniLM-L6-v2` fine-tuned on the Zephyr RTOS documentation, for retrieval over
|
| that corpus.
|
|
|
| - **Comparison Space:** https://huggingface.co/spaces/eoinedge/zephyrproject-embeddings-compare
|
| - **Index and pipeline:** https://huggingface.co/eoinedge/zephyrproject
|
| - **Source:** https://github.com/eoinjordan/zephyrproject-rag
|
|
|
| ## What it is for
|
|
|
| The base model is trained on general web text, where "binding" means a contract and "west" is a direction. This copy is
|
| adapted to Zephyr RTOS's vocabulary so those senses separate.
|
|
|
| Use it as a drop-in replacement for the base model when embedding Zephyr RTOS
|
| documentation:
|
|
|
| ```python
|
| from sentence_transformers import SentenceTransformer
|
|
|
| model = SentenceTransformer("eoinedge/zephyrproject-docs-embeddings")
|
| vectors = model.encode(passages, normalize_embeddings=True)
|
| ```
|
|
|
| ## Training
|
|
|
| | | |
|
| |---|---|
|
| | Base | `sentence-transformers/all-MiniLM-L6-v2` |
|
| | Pairs | 6,079 mined from the docs' own heading/body structure |
|
| | Loss | `MultipleNegativesRankingLoss` (in-batch negatives) |
|
| | Epochs | 1 |
|
| | Batch size | 32 |
|
| | Held-out pairs | 400 |
|
|
|
| No hand-labelled data and no synthetic questions from a generator — a section
|
| heading is a natural query for the body beneath it, and it is already written by
|
| the documentation authors.
|
|
|
| ## Results, and how to read them
|
|
|
| | Metric | Base | Tuned |
|
| |---|---|---|
|
| | recall@5 on held-out pairs | 0.63 | **0.86** |
|
|
|
| **That number flatters the model, and you should treat it with suspicion.** The
|
| evaluation measures the same heading→body relationship the model was trained on,
|
| so it partly measures whether training converged rather than whether retrieval
|
| improved.
|
|
|
| Spot-checking real questions gives a more mixed picture: some clearly improve,
|
| several are unchanged, and at least one regressed. Real questions are not phrased
|
| like section headings, which is exactly the gap this evaluation does not cover.
|
|
|
| The comparison Space runs both models on the same query so you can judge for
|
| yourself rather than trusting the headline.
|
|
|
| ## Limitations
|
|
|
| - One epoch, one seed, no hyperparameter search.
|
| - Evaluation is aligned with the training objective, as above.
|
| - Trained on Zephyr RTOS documentation only — it will not help on other corpora and
|
| may be worse than the base model there.
|
| - This adapts the **retriever**, not a generator. It does not teach a coding
|
| model anything about Zephyr RTOS; it changes which passages that model is handed.
|
|
|
| ## Licence
|
|
|
| Apache-2.0, matching the base model.
|
|
|
