README.md

metadata

title: Rust Docs Assistant
emoji: 🦀
colorFrom: red
colorTo: gray
sdk: gradio
sdk_version: 6.22.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Ask about Rust, answered from the official docs with links
models:
  - Qwen/Qwen3-Embedding-0.6B

Rust Docs Assistant 🦀

What the app is

A question answering app over the official Rust documentation. Type a question into a search box, the app searches five official Rust books, and a language model writes the answer from the passages it found. Every answer cites the sections it drew on, and links back to them on doc.rust-lang.org.

Rust's documentation is thorough but spread across books with different purposes: the Book teaches, the Reference specifies, Rust by Example demonstrates, the Rustonomicon covers unsafe code, and the async book covers async. This app searches all five at once and shows where each part of the answer came from, so the user can visit them for further reading.

The model is prompted always answer from the documentation and to never fall back to its own knowledge. It should also surface when the excerpts do not cover the question.

Features, and the ones set as default

Feature	Default	Can the user change it
LLM provider	OpenAI	Yes, dropdown
Model	gpt-5-mini	Yes, dropdown
Retrieval method	Hybrid search	No, fixed in rag/config.py
Passages returned per search	8	No
Searches allowed per question	3	No
Book to search	Chosen by the model	No

The three providers each offer a cheaper default model and a stronger one. The defaults are gpt-5-mini, gemini-2.5-flash, and claude-haiku-4-5; the stronger options are gpt-5, gemini-2.5-pro, and claude-sonnet-4-5.

Retrieval is not a fixed step that runs before the model is called. Instead, the model is given a search tool that takes a query and, optionally, a book to restrict the search to. The model decides when to call it, what to search for, and whether one search is enough. This allows a multi-part question to be answered with one search per part, and it allows the model to perform follow-up queries if it could not extract the answer from its previous queries.

Answers stream as they are written, and each search appears in the conversation as it happens, so the reader can see what was searched for before the answer arrives.

Other functionalities implemented

1. Hybrid search — BM25 and vector retrieval run over the same corpus and are merged by reciprocal rank fusion.
2. Metadata filtering — a search can be restricted to one of the five books.
3. Query routing — the model chooses that restriction itself, as an argument to the search tool.
4. Function calling — retrieval is a tool the model calls.
5. Streaming responses — answers stream, and each search is shown as it is performed.
6. RAG evaluation — the evaluation dataset, the generation script, and the scoring script are in eval/, and the results are in this README.

A cross-encoder re-ranker is also implemented and was evaluated, but it is not used by the app. The reason is given in the evaluation section below.

Data collection

The corpus is built from the source repositories of five official Rust books, each pinned to a specific commit so the corpus can be rebuilt exactly.

Source	Repository	Commit	Licence	Chunks
The Rust Programming Language	rust-lang/book	917544888a55	MIT OR Apache-2.0	1,048
The Rust Reference	rust-lang/reference	bec6b5e6631b	MIT OR Apache-2.0	926
Rust by Example	rust-lang/rust-by-example	15308f3e9518	MIT OR Apache-2.0	456
The Rustonomicon	rust-lang/nomicon	5012a37c682b	MIT OR Apache-2.0	258
Asynchronous Programming in Rust	rust-lang/async-book	43891cedf954	MIT	182

507 pages in total, of which 504 produce content; the remaining three are index pages in the Reference that contain only links. The result is 2,870 chunks and roughly 760k tokens.

Collection runs in four scripts, kept separate because they have different costs. Fetching is network-bound and rarely rerun, while parsing and indexing are local and were iterated on constantly.

Fetching (ingest/fetch_sources.py) clones each repository at its pinned commit and runs mdBook's markdown backend over it. Rendering with mdBook rather than reading the raw source files is preffered because the source is not what the published site shows: code listings are pulled in from separate files by include directives, and only the build resolves them. The result is 507 markdown pages plus a manifest recording each page's title, path, and published URL.

Parsing (ingest/parse_books.py) turns those pages into chunks. It scans each page a line at a time and keeps a window of the text it has seen, closing the window when adding the next block would take it past 400 tokens. A code fence is treated as a single block, so a code example is never split across two chunks even when that pushes a chunk over the target. Sizes are counted with the embedding model's own tokenizer rather than by characters.

Three kinds of build-time markup are resolved during parsing, because indexing them would put text in the corpus that would add noise:

• Lines beginning with # inside a Rust code fence are scaffolding that rustdoc hides from the rendered page, and they are dropped. The parser then scans the finished corpus and fails if any survived, so a change in the books' conventions is caught rather than silently indexed.
• The Reference marks each rule with an identifier such as r[items.fn]. These are stripped from the indexed text.
• Link targets are removed while the link text is kept, so a chunk reads as prose rather than as a list of URLs.

Each chunk carries the book it belongs to, the chapter and heading trail it sits under, whether it contains code and in what language, and the URL of the section it came from. The book field is what metadata filtering searches on, and the URL is what answers cite.

Indexing (ingest/build_index.py) embeds every chunk with Qwen/Qwen3-Embedding-0.6B and writes them to a Chroma collection. The heading trail is written into the embedded text as a breadcrumb, so that a passage carries the context of where it sits rather than being embedded as loose prose. Retrieval was measurably worse when it was absent.

Publishing (ingest/publish_index.py) uploads the built collection to a Hugging Face dataset repository. The app downloads it at startup rather than building it, so a Space does not spend several minutes embedding on every restart.

Evaluation

How the dataset was generated

194 question and passage pairs, generated by sampling chunks from the corpus and asking gpt-5-mini for questions each one answers. The chunk a question was generated from is that question's correct answer.

The main problem with generating questions this way is that a model shown a passage writes a question that quotes it, which measures word matching rather than search. Three things were implemented to reduce this:

1. The generator is shown six real Stack Overflow question titles in a few-shot prompt, so that it writes in the register of a search box rather than an exam paper.
2. It drafts four candidate questions per passage, and the one sharing the fewest words with the passage is kept.
3. A second pass judges each candidate and discards any question the passage does not specifically answer, which prevents the second step from selecting questions so generic that another page would answer them better.

The resulting questions are around nine words long, which is the length of something a user is likely to type. Sampling is seeded, so the same 200 chunks are drawn every time; 194 of them produced a usable question.

Results

The metrics used are: Hit Rate, the share of questions where a correct passage appeared in the eight results, and MRR, which also accounts for how high it ranked.

Every question is scored twice, because scoring only the passage a question was generated from penalises the retriever unfairly. A chapter is split across several passages and a neighbouring one often answers the question just as well. The first pair of columns counts only the exact passage. The second pair counts any passage under the same heading, which is the level an answer's citation links to.

Retrieval method	Hit Rate (exact)	MRR (exact)	Hit Rate (section)	MRR (section)
Vector search only	0.66	0.42	0.73	0.50
BM25 only	0.62	0.39	0.70	0.48
Hybrid search — what the app uses	0.73	0.44	0.77	0.53
Hybrid search with re-ranking	0.68	0.41	0.78	0.49

Based on the table above, combining the two search methods seems to perform better than using either on its own.

Re-ranking reaches the right section marginally more often, but ranks the exact passage lower when it does. The effect of using re-ranking for this corpus does not improve results enough to justify using it, especially given that it doubles the latency of retrieval from 50ms to 100ms on average. Based on these findings the app always combines embedding retrieval and keyword search with re-ranking omitted.

Run uv run python -m eval.run_eval to reproduce the table.

Known limitations of the evaluation, and how they could be improved

The main limitation is that neither pair of columns fully captures how often the search succeeds. The test dataset assumes that every question has exactly one passage across all books that could answer it, when in reality these books can cover and potentially answer the same question in equally valid ways. A search that returns a better page than the one the question was generated from is scored as a failure.

One possible way to improve this would be to use a pool of graded judgments: run several retrievers, collect the union of what they return, and have an assessor rate how relevant each candidate is instead of marking one right and the rest wrong. For example, this is how TREC builds its judgments. However, it is limited by the fact that a retriever that finds a good passage no other system surfaced is still scored as wrong, because unjudged means non-relevant (Buckley et al.).

So ideally the ratings need to come from human labelling. It has been found that asking an LLM to do the labelling makes the evaluation circular — rating how relevant a passage is to a query is itself the retrieval task — and caps the measured score at the judge's own ability (Soboroff, NIST).

Finally, another limitation is caused by the way in which the eval dataset was generated. While it is easier to ask an LLM to generate question and answer pairs from the original data, this causes an inherent bias in which the LLM has already seen the answer, so it will tend to fit its question to match it. While measures were taken above to reduce this bias, it would still not reach the same quality as curating an eval dataset based on real questions asked by users of this app, or on other forums like Stack Overflow.

API keys

The app needs one key, for whichever provider you choose. Paste it into the field in the app.

• OpenAI API key, or
• Google Gemini API key, or
• Anthropic API key

No key is stored in this repository. The key you paste is held for the length of a single request and is never logged or written to disk.

Cost estimation

Searching costs nothing. Embedding the query, BM25, and the fusion of the two all run on the machine hosting the app, using an open source embedding model. The only billed work is writing the answer.

A turn is dominated by the passages the model reads. Measured on OpenAI over three questions spanning one and two searches, a turn averages 5,239 input and 736 output tokens. The figures below apply those token counts to each provider's published prices.

Model	$/1M in	$/1M out	Per question
gpt-5-mini (default)	0.25	2.00	$0.0028
gemini-2.5-flash (default)	0.30	2.50	$0.0034
claude-haiku-4-5 (default)	1.00	5.00	$0.0089
gpt-5	1.25	10.00	$0.0139
gemini-2.5-pro	1.25	10.00	$0.0139
claude-sonnet-4-5	3.00	15.00	$0.0268

Twenty questions on any default model costs under $0.20, and every feature of the app is visible within an ordinary question: the searches, the book scoping, and the citations all happen inside a turn rather than adding to it.

None of the data collection scripts are reachable from the app, so no expensive pipeline can be run against a visitor's key.

Running it locally

Requires Python 3.12.

uv venv --python 3.12
uv pip install -r requirements.txt

cp .env.example .env     # only needed for the offline scripts
uv run python app.py

The app downloads the prebuilt index from the Hugging Face Hub on first run.

Rebuilding the knowledge base

Fetching needs the mdbook binary on PATH.

uv pip install -r requirements-ingest.txt

uv run python -m ingest.fetch_sources    # -> data/sources/, data/manifest.json
uv run python -m ingest.parse_books      # -> data/chunks.jsonl
uv run python -m ingest.build_index      # -> data/chroma/
uv run python -m ingest.publish_index    # -> Hugging Face dataset repo

The parsing patterns are documented by ingest/test_parse_books.py, which pins what each one matches against real lines from the books, so a change in their markup fails a test rather than quietly dropping text:

uv pip install -r requirements-dev.txt
uv run pytest

Fetching caches to disk and skips books already rendered. Chunk sizes are measured with the embedding model's own tokenizer, so changing the embedding model re-chunks the corpus, and the index and the evaluation dataset have to be rebuilt with it. Publishing needs HF_INDEX_REPO and a write-scoped HF_TOKEN; use --dry-run to see what would be uploaded.

Layout

app.py           Gradio UI, the Hugging Face Space entrypoint
rag/
  config.py      model ids, paths, book specs, chunking and retrieval settings
  providers.py   (provider, key, model) -> LlamaIndex LLM
  index.py       embedding model, Chroma collection, retriever
  types.py       shapes of manifest.json and chunks.jsonl
  prompts.py     system prompt, excerpt rendering, citations, retrieval trace
  pipeline.py    retrieve -> assemble prompt -> stream
ingest/          offline scripts that fetch, parse, index, and publish
eval/            evaluation dataset, generation script, and scoring script
data/            rendered markdown, chunks, and the built index (not in git)