eoinedge/ros2-docs-embeddings

ROS 2 documentation embeddings

all-MiniLM-L6-v2 fine-tuned on the ROS 2 documentation, for retrieval over that corpus.

• Comparison Space: https://huggingface.co/spaces/eoinedge/ros2-embeddings-compare
• Index and pipeline: https://huggingface.co/eoinedge/ros2
• Source: https://github.com/eoinjordan/ros2-coding-assistant

What it is for

The base model is trained on general web text, where "node" means a graph vertex and "action" means a UI event. This copy is adapted to ROS 2's vocabulary so those senses separate.

Use it as a drop-in replacement for the base model when embedding ROS 2 documentation:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("eoinedge/ros2-docs-embeddings")
vectors = model.encode(passages, normalize_embeddings=True)

Training

Base	sentence-transformers/all-MiniLM-L6-v2
Pairs	4,507 mined from the docs' own heading/body structure
Loss	MultipleNegativesRankingLoss (in-batch negatives)
Epochs	1
Batch size	32
Held-out pairs	400

No hand-labelled data and no synthetic questions from a generator — a section heading is a natural query for the body beneath it, and it is already written by the documentation authors.

Results, and how to read them

Metric	Base	Tuned
recall@5 on held-out pairs	0.565	0.8225

That number flatters the model, and you should treat it with suspicion. The evaluation measures the same heading→body relationship the model was trained on, so it partly measures whether training converged rather than whether retrieval improved.

Spot-checking real questions gives a more mixed picture: some clearly improve, several are unchanged, and at least one regressed. Real questions are not phrased like section headings, which is exactly the gap this evaluation does not cover.

The comparison Space runs both models on the same query so you can judge for yourself rather than trusting the headline.

Limitations

• One epoch, one seed, no hyperparameter search.
• Evaluation is aligned with the training objective, as above.
• Trained on ROS 2 documentation only — it will not help on other corpora and may be worse than the base model there.
• This adapts the retriever, not a generator. It does not teach a coding model anything about ROS 2; it changes which passages that model is handed.

Licence

Apache-2.0, matching the base model.