Hugging Face's logo

Spaces:
AdarshDS
/
ThesisBackend
Build error

App Files Community
ThesisBackend / docs /THESIS_SYSTEM_DESIGN_AND_MCP.md
AdarshRajDS
Add thesis system design and MCP architecture document.
e418df7
preview code
|
Raw
History Blame Contribute Delete
28.4 kB

HFU Multimodal Anatomy Chatbot — System Design, Evolution, and MCP Architecture

Document type: Thesis technical design record
Scope: Planning rationale, failed/iterative approaches, final architecture, and Model Context Protocol (MCP) integration
Repository: ThesisBackend (FastAPI + Next.js + anatomy_mcp/)

1. Executive summary

This project delivers a multimodal anatomy learning assistant with two deliberately separated capabilities:

Mode User intent Knowledge source Output
RAG Chatbot Answer textbook questions from uploaded PDFs Chroma + BM25 + optional CLIP figure index Grounded text, ≤3 citations, optional figures
MCP 3D Anatomy Export and inspect a named body structure Z-Anatomy Startup.blend + geometry-proven catalog GLB + annotation JSON + Three.js viewer

The thesis contribution is not “one model that does everything,” but a dual-pipeline architecture where document grounding and 3D geometry export remain evaluated independently. The final 3D path implements the official Model Context Protocol (stdio transport, ClientSession.call_tool, FastMCP tool server) rather than ad-hoc Python function calls or prompt-only LLM hallucination.

2. Original research goals

When the project started, the goals were:

  1. Grounded Q&A — Students ask anatomy questions; answers must cite uploaded course PDFs, not unconstrained LLM knowledge.
  2. Multimodal support — Textbook figures (diagrams, histology, radiology stills) should appear when relevant, not only text chunks.
  3. 3D visualization — When a structure is named, the system should produce inspectable 3D geometry with labels, not a static screenshot alone.
  4. Reproducible evaluation — RAG quality (citation accuracy, abstention) and 3D export success (GLB URL, viewer load) must be measurable separately.
  5. Local-first deployment — Run on a Windows developer machine: LM Studio, Blender 5.x, Z-Anatomy, optional MinIO.

These goals drove every architectural decision below.

3. Initial approach (v0) and its flaws

3.1 What we built first

The first iteration was a monolithic FastAPI + single LLM stack:

User question → Chroma similarity search → top-k chunks → LLM prompt → answer

Supporting pieces:

  • PDF upload → text chunking → MiniLM embeddings → Chroma persist directory
  • Optional local /outputs folder for extracted images
  • A separate experimental path: prompt Blender to procedurally generate geometry (src/mcp/blender_generator.py, BlenderMCP.generate_3d_brain)

3.2 Flaws discovered in practice

Flaw Symptom Root cause
Weak retrieval Correct answer in PDF but wrong passage retrieved Dense-only search; no BM25 for exact terms (e.g. “mitral valve”, “NMJ”)
Citation sprawl 10+ reference cards; duplicates from same page No deduplication, no max-source cap, no evidence grading
Ungrounded synthesis Plausible anatomy not in corpus LLM answered from parametric knowledge when retrieval was empty
Figure blindness Questions about labeled diagrams missed images Text-only index; figures stored but not retrieved cross-modally
Fragile image URLs Broken images after restart or HF deploy Local paths only; no object storage abstraction
Fake 3D success UI showed “Exported liver” with no GLB LLM text claimed export; no tool verification
Wrong 3D source Procedural “brain” mesh unrelated to Z-Anatomy blender_generator.py creates synthetic geometry, not curriculum-accurate labels
Name-only matching “kidney” export failed or picked wrong collection Raw label index lists scene names, not exportable geometry
Parallel Blender corruption Random export failures on Windows Multiple Blender subprocesses writing concurrently
Coupled evaluation Could not tell if RAG or 3D failed Single chat endpoint mixed document answers and export URLs

These flaws motivated the phased redesign documented in Sections 4–7.

4. Approaches we tried (chronological evolution)

4.1 Overview timeline 

timeline
    title Architecture evolution
    section Retrieval
        v0 Dense Chroma only : Weak on exact terms
        v1 Hybrid BM25 + dense + RRF : Better keyword + semantic balance
        v2 Evidence grading + max 3 sources : Thesis-grade grounding
    section Multimodal
        v0 Text chunks only : Missed figures
        v1 CLIP multimodal Chroma : Figure retrieval added
        v2 PDF pipeline + MinIO/Supabase : Durable public image URLs
    section 3D visualization
        v0 Procedural Blender script : Not Z-Anatomy accurate
        v1 Remote render worker (asset PNG) : RAG sidebar preview only
        v2 Direct Python anatomy calls : Worked but not MCP-standard
        v3 True MCP stdio + catalog fast path : Final production design
    section LLM
        v0 Single local model : LM Studio only
        v1 Split providers : Groq for RAG speed, LM Studio as MCP tool host

4.2 Approach A — Dense vector RAG only

Design: VectorStoreFactory + similarity_search(k=...).

Why we tried it: Fastest path to a demo; LangChain/Chroma defaults.

Result: Failed on anatomy terminology where embedding similarity diverges from exact nomenclature (Latin names, hyphenation, “left vs right”).

Replacement: HybridRetriever in src/retrieval/hybrid_retriever.py — BM25 + dense + phrase matching fused with Reciprocal Rank Fusion (RRF).

4.3 Approach B — Multimodal RAG (CLIP + text)

Design: MultimodalRAG combines:

  • Text corpus (MiniLM Chroma, same as ingestion)
  • Multimodal retriever (MultimodalRetriever + CLIPEmbedding)
  • Up to 3 unique text passages (_MAX_UNIQUE_TEXT_PASSAGES = 3)

Why we tried it: Anatomy learning is inherently visual; students ask about figures explicitly.

Improvements over A:

  • Retrieves figure metadata and serves images from /outputs or presigned storage
  • Dedupes by PDF stem + page + text fingerprint

Remaining gap: Still no 3D export; images are 2D textbook figures only.

4.4 Approach C — Strict grounding & thesis evaluation pipeline

Design: src/multimodal/thesis_rag_eval.py — parallel experiment track:

  • Baseline — LLM without retrieval (measures hallucination rate)
  • Strict RAG — answer only from numbered passages
  • Coherent synthesis — stricter prompt with citation IDs
  • Blind judge — compares baseline vs strict

Why we tried it: Quantify how much retrieval helps for thesis evaluation (POST /rag/experiment/ask).

Lesson: Production POST /rag/ask and experiment pipeline stay separate so tuning does not break live UI.

4.5 Approach D — Procedural Blender generation (BlenderMCP)

Design: src/mcp/blender_mcp.py + blender_generator.py

LLM prompt → blender --background --python blender_generator.py → synthetic GLB

Why we tried it: Quick 3D “something” without licensing a full anatomy asset library.

Flaws:

  • Geometry is not Z-Anatomy; labels do not match course material
  • No annotation JSON sidecar
  • POST /blender/generate-brain-3d is a research stub, not curriculum export

Status: Kept for optional brain demo; not the MCP catalog path.

4.6 Approach E — Remote Blender render worker (RAG adjunct)

Design: app/services/blender_service.py → render_related_anatomy()

POST /rag/ask → detect keyword (brain, heart, spine…) → POST {BLENDER_SERVER_URL}/render-asset → PNG URL

Why we tried it: Attach a related 3D preview to RAG answers without blocking on full GLB export latency.

Characteristics:

  • Fixed asset map (ASSET_MAPPING: brain, heart, hand, lung, spine)
  • Worker uploads to object storage; backend stores no local render files
  • Returns render_3d_url in RAG response (optional, best-effort)

Flaws:

  • Only ~5 pre-authored assets; no arbitrary “left femur” from user query
  • Docker blender service in docker-compose.yml is a different worker from Z-Anatomy MCP export
  • Must not be confused with MCP panel exports in evaluation

Status: Optional RAG enhancement; orthogonal to MCP.

4.7 Approach F — Label index without exportable catalog

Design: Scan Z-Anatomy once → z_anatomy_index.json (all collection/object names in scene).

Why we tried it: First automated bridge from natural language to Blender object names.

Flaws:

  • Index proves name exists, not that mesh is exportable
  • Broad queries (kidney, iris) hit ambiguous or empty collections
  • Exports sometimes succeeded with huge selections (entire limb regions)

Replacement: build_exportable_catalog.py → exportable_catalog.json — entries validated by geometry probe in Blender.

4.8 Approach G — Direct in-process Python calls (pre-MCP)

Design: Import anatomy_mcp/server.py functions directly from FastAPI handlers.

Why we tried it: Fastest integration before MCP SDK maturity on Windows.

Flaws:

  • Not interoperable with LM Studio, Claude Desktop, or MCP Inspector
  • No standard tool discovery / schema contract
  • LLM could not participate in multi-step tool loops using a portable protocol
  • Harder to sandbox Blender side effects

Replacement: True MCP stdio server + MCPBridge client (Approach H).

4.9 Approach H — True MCP with dual execution paths (FINAL)

Design: See Section 6–7. This is the production 3D architecture.

4.10 Storage approaches tried

Approach Mechanism When used Flaw / fix
Local outputs/ FastAPI static mount Earliest dev URLs break on multi-replica / HF Spaces
MinIO S3-compatible Docker (docker-compose.yml) Local full stack Requires Docker; good for dev
Supabase Storage STORAGE_PROVIDER=supabase Production / HF Needs service role key; auto-selected when env set

Image pipeline steps are observable via GET /debug/storage and upload response pipeline array.

4.11 LLM provider approaches

Role Initial Final
RAG synthesis LM Studio local Groq (llama-3.1-8b-instant) for latency
MCP tool host N/A LM Studio OpenAI-compatible API at :1234/v1
Evidence grading / judge Groq Groq (thesis eval)

Rationale for split: RAG needs fast repeated calls; MCP tool loop needs a model with reliable function calling locally without cloud cost for Blender waits.

5. Final architecture (what we reached)

5.1 System context 

flowchart TB
    subgraph User["User / Evaluator"]
        U1[Browser]
    end

    subgraph Frontend["Next.js Frontend :3000"]
        Chat[Main Chat UI]
        Upload[Upload Panel]
        MCPPanel[MCP 3D Panel + iframe viewer]
    end

    subgraph Backend["FastAPI Backend :8000"]
        RAGRoute["/rag/ask"]
        UploadRoute["/upload-pdf/"]
        MCPRoute["/anatomy/ask"]
        RAGSvc[rag_service]
        RAGEngine[MultimodalRAG]
        MCPChat[anatomy_mcp_chat]
        MCPBridge[MCPBridge stdio client]
    end

    subgraph RAGData["Document knowledge"]
        Chroma[(Chroma text index)]
        BM25[(BM25 pickle)]
        MMChroma[(Multimodal Chroma)]
        MinIO[(MinIO / Supabase / local outputs)]
    end

    subgraph MCPData["3D anatomy knowledge"]
        Catalog[(exportable_catalog.json)]
        Blend[Z-Anatomy Startup.blend]
        Exports[/anatomy-exports GLB + JSON/]
        Viewer[/anatomy-viewer Three.js/]
    end

    subgraph External["External runtimes"]
        Groq[Groq API — RAG LLM]
        LMStudio[LM Studio — MCP host LLM]
        Blender[Blender 5.x headless]
    end

    subgraph MCPServer["anatomy_mcp/server.py — stdio MCP"]
        SearchTool[search_anatomy_catalog]
        ExportPart[export_anatomy_part]
        ExportPkg[export_anatomy_package]
    end

    U1 --> Chat & Upload & MCPPanel
    Chat -->|POST question| RAGRoute
    Upload -->|POST PDF| UploadRoute
    MCPPanel -->|POST message| MCPRoute

    RAGRoute --> RAGSvc --> RAGEngine
    RAGEngine --> Chroma & BM25 & MMChroma
    RAGEngine --> Groq
    UploadRoute --> Chroma & MMChroma & MinIO
    RAGEngine --> MinIO
    RAGSvc -->|optional| RenderWorker[Remote render worker]

    MCPRoute --> MCPChat --> MCPBridge
    MCPChat -->|fast path or tool loop| LMStudio
    MCPBridge <-->|stdio JSON-RPC| MCPServer
    SearchTool --> Catalog
    ExportPart & ExportPkg --> Blender
    Blender --> Blend
    ExportPart & ExportPkg --> Exports
    MCPPanel -->|iframe src=viewer_url| Viewer
    Viewer --> Exports

5.2 Separation of concerns (evaluation rule) 

flowchart LR
    subgraph RAGPath["Mode A — RAG Chat"]
        Q1[Textbook question]
        A1[Grounded answer + PDF citations]
    end

    subgraph MCPPath["Mode B — MCP 3D"]
        Q2[Structure name e.g. left femur]
        A2[GLB + annotations + viewer]
    end

    Q1 -.->|must NOT| A2
    Q2 -.->|must NOT| A1

    style RAGPath fill:#e8f4fc,stroke:#2563eb
    style MCPPath fill:#ecfdf5,stroke:#059669

Thesis rule: Never score a RAG answer on GLB export success; never score an MCP export on PDF citation quality.

6. MCP deep dive — protocol, components, and guarantees

6.1 What “proper MCP” means in this project

The implementation satisfies the MCP contract at three layers:

Layer Component Responsibility
Tool server anatomy_mcp/server.py FastMCP registers tools; runs Blender; returns structured JSON
Transport stdio via mcp.client.stdio Parent process spawns server; no HTTP port on MCP server
Host orchestration app/services/anatomy_mcp_chat.py LM Studio chooses tools; backend executes via ClientSession.call_tool

Non-MCP patterns explicitly rejected:

  • LLM printing http://.../anatomy.glb without calling a tool
  • Direct Python import of export functions from FastAPI routes
  • Hard-coded export in frontend

6.2 MCP layer diagram 

flowchart TB
    subgraph Presentation["Presentation layer"]
        UI[AnatomyMcpPanel.js]
        Viewer[Three.js viewer iframe]
    end

    subgraph API["FastAPI REST layer"]
        Ask["POST /anatomy/ask"]
        Health["GET /anatomy/health"]
        Static["/anatomy-exports /anatomy-viewer"]
    end

    subgraph Orchestration["MCP host orchestration"]
        Agent[run_lmstudio_mcp_agent]
        FastPath[_try_catalog_fast_path]
        ToolLoop[LM Studio tool loop max 6 rounds]
    end

    subgraph Client["MCP client — MCPBridge"]
        Session[ClientSession]
        ListTools[list_tools → OpenAI schema]
        CallTool[call_tool]
        StdioTransport[stdio_client]
    end

    subgraph Server["MCP server — FastMCP"]
        T1[search_anatomy_catalog]
        T2[export_anatomy_part]
        T3[export_anatomy_package]
        Lock[_BLENDER_EXPORT_LOCK]
        Resolver[exportable_catalog resolver]
    end

    subgraph Execution["Execution layer"]
        BlenderCLI[Blender subprocess]
        ZAnatomy[Startup.blend]
        Cache[exports/cache v4 schema]
    end

    UI --> Ask
    Ask --> Agent
    Agent --> FastPath
    Agent --> ToolLoop
    FastPath --> CallTool
    ToolLoop --> ListTools
    ToolLoop --> CallTool
    CallTool --> StdioTransport --> Session
    Session --> T1 & T2 & T3
    T2 & T3 --> Lock --> BlenderCLI --> ZAnatomy
    T2 & T3 --> Cache
    Static --> Viewer
    UI --> Viewer

6.3 MCP tool catalog

Tool Input Output When used
search_anatomy_catalog query, limit Ranked catalog entries, suggestions, ambiguity errors Every export path starts here or equivalent resolution
export_anatomy_part part_query, include_preview model_url, annotations_url, viewer_url, metadata Single structure (e.g. Femur.l, Liver)
export_anatomy_package part_query, options Study package with subparts for large regions Brain, thalamus; fallback after part export timeout

Safety boundaries (from anatomy_mcp/README.md):

  • No arbitrary .blend paths from user input
  • No arbitrary Python injection
  • No full-scene export by default
  • Serialized Blender lock — one export at a time on Windows

6.4 Catalog resolution pipeline 

flowchart TD
    Input[User message e.g. left femur]
    Validate{Query valid?}
    Vague{Vague query?}
    Normalize[catalog_query_from_user_message]
    Search[search_anatomy_catalog]
    Match{Exact exportable match?}
    Ambiguous{Multiple matches?}
    Export[export_anatomy_part]
    Package[export_anatomy_package]
    Success[anatomy_export status ok]
    Clarify[Clarification message]
    Suggest[Return suggestion labels]

    Input --> Validate
    Validate -->|invalid chars / too long| Clarify
    Validate --> Vague
    Vague -->|show me the organ| Clarify
    Vague --> Normalize --> Search
    Search --> Match
    Match -->|yes Femur.l| Export
    Match -->|no| Ambiguous
    Ambiguous -->|kidney| Suggest
    Export -->|timeout| Package
    Export --> Success
    Package --> Success

Key data artifact: exportable_catalog.json — only entries with proven mesh geometry. This replaced naive z_anatomy_index.json matching.

6.5 MCP sequence — fast path (plain queries)

Used when input matches looks_like_plain_anatomy_query (≤80 chars, alphanumeric + space/dot/dash).

sequenceDiagram
    autonumber
    actor User
    participant UI as AnatomyMcpPanel
    participant API as POST /anatomy/ask
    participant Chat as anatomy_mcp_chat
    participant Bridge as MCPBridge
    participant Srv as anatomy_mcp/server.py
    participant Cat as exportable_catalog.json
    participant B as Blender
    participant V as anatomy-viewer iframe

    User->>UI: "left femur"
    UI->>API: { message, language }
    API->>Chat: run_lmstudio_mcp_agent()

    Note over Chat: Fast path — no LM Studio round-trip

    Chat->>Bridge: call_tool search_anatomy_catalog
    Bridge->>Srv: MCP CallTool
    Srv->>Cat: token + lateral resolution
    Srv-->>Chat: Femur.l match

    Chat->>Bridge: call_tool export_anatomy_part
    Bridge->>Srv: MCP CallTool
    Srv->>B: subprocess export_part.py
    B-->>Srv: anatomy.glb + annotations.json
    Srv-->>Chat: model_url, viewer_url

    Chat->>Chat: structured_to_anatomy_export
    Note over Chat: Fail if model_url missing — no fake success

    Chat-->>API: answer + anatomy_export + mcp_tools_used
    API-->>UI: JSON 200
    UI->>V: iframe src = viewer_url
    V-->>User: labeled 3D mesh

Why fast path exists: Blender exports take 30–120s. Skipping LLM orchestration for "Femur.l"-style queries reduces failure modes and token cost.

6.6 MCP sequence — LLM tool loop (complex queries) 

sequenceDiagram
    autonumber
    participant Chat as anatomy_mcp_chat
    participant LM as LM Studio LLM
    participant Bridge as MCPBridge
    participant Srv as MCP server

    Chat->>Bridge: get_openai_tools()
    Bridge->>Srv: list_tools
    Srv-->>Bridge: tool schemas
    Bridge-->>Chat: OpenAI function definitions

    loop Up to MAX_TOOL_ROUNDS = 6
        Chat->>LM: messages + tools + tool_choice auto
        LM-->>Chat: assistant message + tool_calls
        alt tool_calls present
            Chat->>Bridge: call_tool(name, args)
            Bridge->>Srv: MCP CallTool
            Srv-->>Bridge: structured JSON result
            Bridge-->>Chat: compact_tool_result_for_llm
            Chat->>Chat: append tool result to messages
        else no tool_calls
            Chat->>Chat: verify export or require tools
        end
    end

    Chat->>Chat: structured_to_anatomy_export
    alt model_url present
        Chat-->>Chat: status ok + mcp_tools_used logged
    else export error
        Chat-->>Chat: status error — UI shows failure, no empty iframe
    end

Host prompt constraints (build_mcp_system_prompt in app/i18n/locale.py):

  • Must call export_anatomy_part or export_anatomy_package before claiming success
  • Must not echo raw package manifest URLs to user
  • Must ask clarification on vague queries
  • Retry package export after part export timeout

6.7 How the final approach ensures MCP integrity

Guarantee Mechanism Code reference
Real tool discovery session.list_tools() → OpenAI schemas MCPBridge.get_openai_tools()
Real tool execution session.call_tool(name, arguments) MCPBridge.call_tool()
Stdio isolation Separate Python process for server.py StdioServerParameters
Structured tool results structuredContent + text fallback parsing extract_structured_from_tool_payload()
No hallucinated exports structured_to_anatomy_export returns null without model_url anatomy_mcp_client.py
UI honesty Frontend checks anatomy_export.status === "ok" before iframe AnatomyMcpPanel.js
Audit trail Response includes mcp_tools_used, mcp_tool_steps _agent_result()
URL rewriting Dev server :8123 → public API base rewrite_local_urls()
Concurrency safety threading.RLock around Blender anatomy_mcp/server.py
Health probe GET /anatomy/health checks Blender, catalog, MCP import anatomy_mcp_health() 
flowchart TD
    Claim[LLM says Exported liver]
    Tools{mcp_tools_used includes export tool?}
    URL{model_url HTTP 200?}
    OK[Show viewer + GLB links]
    Fail[Show error panel]

    Claim --> Tools
    Tools -->|no| Fail
    Tools -->|yes| URL
    URL -->|no| Fail
    URL -->|yes| OK

7. Final approach — what we implemented in the last iteration

7.1 Backend wiring (app/main.py)

Routers and static mounts required for MCP UI:

  • app.include_router(anatomy_mcp.router) → /anatomy/*
  • app.mount("/anatomy-exports", ...) → GLB, packages, annotations
  • app.mount("/anatomy-viewer", ...) → Three.js viewer assets

7.2 Settings contract (src/config/settings.py)

Environment-driven configuration for MCP:

  • PUBLIC_API_BASE — URL prefix for exported assets
  • BLENDER_BIN, Z_ANATOMY_BLEND — passed into MCP server env
  • LLM_API_BASE — LM Studio for tool host
  • ANATOMY_MCP_ENABLED — feature gate

7.3 Frontend MCP panel

frontend/app/components/AnatomyMcpPanel.js:

  • Calls POST /anatomy/ask (not /rag/ask)
  • Renders AnatomyExportPanel with viewer link, GLB download
  • Salvage path parses URLs from verbose LLM dumps (legacy compatibility)
  • i18n EN/DE strings for errors including “restart backend if 404”

7.4 Caching & performance

  • Export cache schema v4 under anatomy_mcp/exports/cache/
  • Repeated Femur.l queries hit cache → near-instant URLs
  • Package exports for large structures avoid timeout on single-part mesh

7.5 RAG path (unchanged responsibility)

POST /rag/ask still uses MultimodalRAG + optional render_3d_url from remote worker — does not invoke MCP tools by default. This preserves evaluation separation.

8. RAG pipeline reference (Mode A detail) 

sequenceDiagram
    autonumber
    actor User
    participant UI as Next.js Chat
    participant API as POST /rag/ask
    participant RAG as MultimodalRAG
    participant IDX as Chroma + BM25 + Hybrid
    participant LLM as Groq

    User->>UI: Question + optional PDF context
    UI->>API: { question, allow_world_knowledge, language }
    API->>RAG: ask()

    RAG->>RAG: classify + typo correct + rewrite
    RAG->>IDX: hybrid retrieve
    RAG->>RAG: rerank + dedupe passages
    RAG->>LLM: evidence grading A/B/C
    RAG->>LLM: grounded synthesis
    RAG->>RAG: normalize citations, max 3 sources
    RAG-->>API: answer, sources, images, grounding
    API-->>UI: AskResponse

9. Comparison matrix (for thesis discussion section)

Dimension Initial v0 Final system
Retrieval Dense only Hybrid BM25 + dense + RRF + phrases
Grounding Prompt-only Evidence grading + abstention + max 3 sources
Figures Ignored CLIP multimodal index + storage URLs
3D source Procedural script Z-Anatomy catalog + Blender export
3D protocol Direct Python / fake LLM text MCP stdio + tool audit trail
LLM Single local Groq (RAG) + LM Studio (MCP host)
Storage Local folder MinIO / Supabase abstraction
Evaluation Mixed Separated RAG vs MCP metrics
Export correctness Name in index Geometry-proven catalog
Parallelism Unsafe Blender export lock

10. Known limitations & future work

  1. LM Studio dependency for complex MCP queries — Fast path avoids it; natural-language disambiguation still needs tool-capable local model.
  2. Windows-only Blender path — Linux/Mac require different Blender binary paths.
  3. Export latency — Large packages (brain) may exceed user patience; cache mitigates repeat queries.
  4. RAG render worker — Only five asset keys; not a general anatomy renderer.
  5. MCP + RAG fusion — Deliberately not merged; future work could link citation text to MCP viewer via shared entity IDs without mixing retrieval corpora.

11. Verification checklist

RAG (Mode A) 

curl -X POST http://127.0.0.1:8000/rag/ask `
  -H "Content-Type: application/json" `
  -d '{"question":"What is the function of the mitral valve?","allow_world_knowledge":false,"language":"en"}'

Pass: answer cites PDF sources; no GLB URLs.

MCP (Mode B) 

curl -X POST http://127.0.0.1:8000/anatomy/ask `
  -H "Content-Type: application/json" `
  -d '{"message":"left femur","language":"en"}'

Pass: anatomy_export.status == "ok", mcp_tools_used includes export_anatomy_part, viewer URL loads.

MCP health 

Invoke-RestMethod http://127.0.0.1:8000/anatomy/health

Pass: ready: true, mcp_stdio_ok: true, blender_exists: true, exportable_catalog_exists: true.

12. Key file index

Path Role
app/main.py FastAPI entry, routers, static mounts
app/services/rag_service.py RAG + optional render worker
app/services/anatomy_mcp_chat.py MCP host agent (fast path + tool loop)
app/services/anatomy_mcp_client.py MCPBridge stdio client
app/services/anatomy_mcp_service.py Health checks, env configuration
anatomy_mcp/server.py FastMCP tool server
anatomy_mcp/label_index/exportable_catalog.json Geometry-validated catalog
anatomy_mcp/viewer/ Three.js annotation viewer
src/multimodal/multimodal_rag_chain.py MultimodalRAG core
src/retrieval/hybrid_retriever.py BM25 + dense hybrid
frontend/app/components/AnatomyMcpPanel.js MCP UI
docs/ARCHITECTURE_FOR_DIAGRAMS.md Diagram source pack

13. Conclusion

The project evolved from a single-path RAG demo into a dual-mode architecture grounded in evaluation needs:

  • Documents are handled by retrieval-augmented generation with strict citation limits.
  • Geometry is handled by a standards-based MCP toolchain connecting an LLM host, a stdio tool server, Blender, and a web viewer.

The final MCP design is not “MCP-themed naming” — it implements discoverable tools, stdio transport, structured tool results, and verifiable export URLs. The fast path optimizes common classroom queries without bypassing MCP (call_tool is still used); the LLM tool loop handles ambiguous natural language while respecting the same server contract.

This document should be cited alongside docs/ARCHITECTURE_FOR_DIAGRAMS.md when generating thesis figures: use blue styling for RAG paths and green styling for MCP paths in all diagrams.

Generated from repository state and architecture records. Update when new approaches are tried or MCP tools change.