docs/README.md

metadata

title: TorchDocsAgent — documentation index
kind: index

TorchDocsAgent — documentation

This folder is the reasoning behind the code: why each part exists, the logic it follows, and the tool choices it rests on. It has two layers.

• Cross-cutting design docs explain the system as a whole — the product shape, the content pipeline, the agent contract, deployment.
• Per-package references (docs/<package>/README.md) mirror the code tree one-to-one: each documents a single top-level package — its boundary, its flow, the decisions and their rejected alternatives, and a file-by-file map.

Start with the design docs for the shape of the whole; drop into a package reference when you're about to touch that package.

The system in one picture

                         the docs.pytorch.org site
                                   │
   ingest/   discover → crawl → chunk           builds the _corpus/ snapshot
                                   │
   index/    embed → Neon (pgvector + tsvector)  storage + hybrid retrieval
                                   │  pointers + hydrated sections
   agent/    guard → route → grounded | loop → llm → Answer   the answering brain
                                   │
   app/      Gradio Space: stream the reasoning, render the answer + citations
                                   │
                              the user

   scripts/  the CLI + CI surface that builds, evaluates, and operates all of the above
   eval/     the measurement layer that keeps every layer honest

Data flows down the middle; scripts/ and eval/ wrap the column as tooling. The knowledge boundary is firm: the docs site is the corpus, source code is never indexed (it is referred out), and every answer is grounded in retrieved sections with clickable citations.

Per-package references

Package	What it is	Reference
ingest/	The crawl → snapshot → chunk pipeline that turns the live docs site into a heading-chunked corpus. Touches no DB.	docs/ingest/
index/	The Neon/pgvector storage + retrieval layer: embeddings, hybrid search, DB-served hydration, and the self-healing freshness pass.	docs/index/
agent/	The answer-generation brain: guard → route → grounded/loop → LLM dispatch → validated Answer.	docs/agent/
app/	The Gradio web app (the Hugging Face Space): serving, concurrency, and the live grey reasoning trace.	docs/app/
eval/	The measurement layer: static answer checks, retrieval metrics, and LLM-as-judge — wired into CI and the live answer path.	docs/eval/
scripts/	The command-line + CI surface: build the index, generate synthetic data, evaluate, calibrate, and smoke-test.	docs/scripts/

Cross-cutting design docs

Document	What it covers
design-content-and-agent-flow.md	The architecture bible: corpus scope, the ingestion pipeline, the three agent tools, the session flow, LangChain-vs-LangGraph, and how stored pointers map to live links. Start here.
retrieval-gaps-and-improvements.md	Known retrieval weaknesses and the improvement backlog (reranking, multilingual embedder, judge model, …).
loop-vs-langgraph.md	The measured comparison of the manual tool loop (agent/loop.py) against its LangGraph twin (agent/graph.py).
deploy-hf-spaces.md	The Hugging Face Spaces deployment walkthrough — secrets, the sync workflow, and the post-deploy smoke test.

See also PLAN.md for the milestone roadmap and the binding decisions that constrain execution.

Suggested reading order

1. design-content-and-agent-flow.md — the whole shape in one read.
2. docs/ingest/ → docs/index/ — how the corpus is built and retrieved (data flows up from here).
3. docs/agent/ — how a question becomes a grounded answer.
4. docs/app/ — how it is served and streamed to the user.
5. docs/eval/ and docs/scripts/ — how it is measured and operated.

Conventions

• Each package reference has YAML frontmatter (title, kind: reference, package), then: boundary → flow → design decisions & rationale → tool choices → file-by-file → related docs.
• Docs describe the code as written, not an idealized plan. Where a plan item is unbuilt or a decision was superseded, the doc says so.
• These are OKF-style knowledge documents (markdown + frontmatter), the same convention the corpus chunks and the other docs/*.md reports use.