Spaces:
Running
Production RAG Pipeline — Project Structure
Project 02 | Version 1.1 Revised May 2026 — Aligned with v1.1 Technical Review
Revision Notes
This structure supersedes the original Project_Structure.md. Changes were made
to align with the architectural decisions resolved in v1.1 and the findings of
the pre-Phase 1 corpus audit. Key additions: src/storage/, src/validation/,
data/ground_truth/, data/processed/filtered/, data/processed/chunks/,
and tests/red_team/.
/src
ingestion/
Document parsing, structure analysis, chunking, and metadata extraction. Corresponds to Phase 1 of the implementation plan.
ingestion/
parser.py # Type-specific handlers: PDF (PyMuPDF), DOCX (python-docx),
# XLSX (openpyxl), Images (unstructured + OCR).
# Do NOT use a single generic parser.
structure_analyzer.py # Classifies every block: heading, paragraph, table,
# code_block, list_item. Outputs a tagged document tree.
chunker.py # Structure-aware chunking: 256–512 tokens, 50-token overlap.
# Hard rules: never split table rows or code blocks.
# Includes minimum character threshold filter (< 100 chars)
# per corpus audit finding (zero-text / low-density pages).
metadata_pipeline.py # Per-chunk metadata: source_file, section_heading,
# chunk_index, date, department, version, summary,
# keywords, hypothetical_questions (HyDE via LLM).
# Temporal metadata: regex-based year extraction from
# filename takes priority over internal PDF metadata
# per corpus audit finding.
Audit note: The corpus contains documents ranging from 7 to 408 pages with high table density in financial files (up to 3 tables/page). Use
unstructuredfor table-to-Markdown conversion and batch processing for large files to prevent memory spikes during embedding generation.
storage/
Infrastructure initialization for all three storage backends. Corresponds to Phase 2 of the implementation plan.
storage/
qdrant_client.py # Qdrant (local Docker) setup for dense vector storage.
# Stores chunk text + full metadata payload.
bm25_index.py # rank_bm25 in-memory index (Decision 1: confirmed).
# Feasible for corpus under ~80K chunks on 16 GB RAM.
# Migration trigger: >80K chunks OR Context Recall < 0.70
# → revisit Qdrant sparse vectors (SPLADE).
neon_db.py # Neon/Postgres + pgvector schema.
# Required fields: version, department, date.
# Supports Neon branching for isolated testing of new
# embedding models or chunking strategies.
retrieval/
Hybrid retrieval pipeline and cross-encoder reranking. Corresponds to Phase 3 of the implementation plan.
retrieval/
hybrid_search.py # Qdrant dense retrieval (top 20) + BM25 sparse (top 20).
rrf_fusion.py # Reciprocal Rank Fusion (k=60). Preferred over weighted
# averaging — requires no tuning.
reranker.py # cross-encoder/ms-marco-MiniLM-L-6-v2 via sentence-
# transformers (replaces Cohere Rerank — zero cost).
# Alternative: cross-encoder/ms-marco-electra-base for
# higher accuracy at additional latency cost.
# Target: ≤ 120ms for full retrieval + reranking.
Hardware note: Local cross-encoder inference is slower than a paid API call. Profile on target hardware early. Reduce top-N candidate count if the 120ms budget is exceeded before adjusting anything else.
reasoning/
LangGraph stateful graph: planner, conditional router, and agent nodes. Corresponds to Phase 4 of the implementation plan.
reasoning/
state.py # RAGState TypedDict: query, sub_tasks, retrieved_chunks,
# agent_outputs, final_answer, validation_passed.
planner.py # Decomposes multi-step queries into executable sub-tasks.
router.py # Classifies sub-task type and routes to correct agent.
agents/
retrieval_agent.py # Retrieval Agent node.
summarization_agent.py # Summarization Agent node.
calculation_agent.py # Calculation Agent node.
graph.py # Full LangGraph StateGraph wiring: entry point, sequential
# edges, conditional edges (Decision 2: sequential execution
# confirmed). Parallelism reserved for Phase 4 optimization
# only if profiling reveals measurable bottlenecks.
validation/
LLM-as-Judge validation nodes. Corresponds to Phase 5 of the implementation plan.
validation/
gatekeeper.py # Verifies response addresses original query. PASS/FAIL + reason.
auditor.py # Grounding check: every claim must be traceable to a retrieved
# chunk. Uses hardened prompt (Decision 3: same model confirmed).
# Prompt must include: "For each claim, identify the exact chunk
# that supports it. If a claim cannot be traced to a specific
# chunk, mark it UNGROUNDED."
strategist.py # Contextual coherence evaluation. PASS/FAIL + reason.
schemas.py # Shared output schema: { verdict: PASS/FAIL, reason: string }.
# All nodes must return this structure. Decisions are logged
# and auditable.
Validation LLM: Same model as reasoning engine (Decision 3). Optional upgrade: load a second quantized model (e.g., Mistral-7B-Instruct Q4_K_M via llama.cpp) for the Auditor role only if 16 GB RAM permits. Two 4-bit quantized 7B models can co-exist within 16 GB.
evaluation/
RAGAS integration, latency and cost tracking, and ground truth validation. Corresponds to Phase 6 of the implementation plan.
evaluation/
ragas_runner.py # Runs RAGAS across all 4 iterations. Records
# Faithfulness, Answer Relevancy, Context Precision,
# Context Recall. Outputs delta scores per iteration.
latency_tracker.py # Per-query logging: retrieval_latency_ms,
# rerank_latency_ms, generation_latency_ms,
# total_latency_ms, validation_passed, cost_usd.
# Alert threshold: total_latency_ms > 280ms.
validate_ground_truth.py # Ground truth schema validation script.
# See ground_truth_plan.md for full specification.
utils/
Shared infrastructure. No business logic.
utils/
logging.py # Structured logging across all subsystems.
env.py # Environment variable management.
helpers.py # Shared helper functions.
/data
data/
raw/ # Original corpus: 27 PDF files + 6 DOCX files.
# Read-only. Never modify files in this directory.
processed/
filtered/ # Post junk-node removal. Pages below 100-character
# threshold are excluded here per corpus audit finding.
chunks/ # Validated structure-aware chunks from Phase 1.
# These are the chunks used for chunk ID assignment
# in the ground truth dataset (Track B).
ground_truth/ # Ground truth QA dataset. Construction begins in Phase 1
# (Track A) and completes after chunking is validated
# (Track B). See ground_truth_plan.md.
ground_truth.json # Primary dataset. Schema: question_id, question,
# ground_truth_answer, ground_truth_chunk_ids,
# source_document, domain_tag.
ground_truth.csv # Optional CSV mirror for spreadsheet-based annotation.
metadata/ # Temporal metadata outputs and other extraction artifacts.
/config
config/
settings.yaml # Model names, chunk size (256–512 tokens), overlap (50 tokens),
# RRF k-value (60), latency thresholds (p95: 280ms,
# retrieval: 120ms), RAGAS alert threshold (CP < 0.75),
# top-N retrieval counts, API endpoint configurations.
/tests
tests/
unit/ # Per-module unit tests for each subsystem.
integration/ # Round-trip and cross-subsystem tests.
# Includes: single-document ingest → Qdrant + Postgres
# round-trip (Phase 2 milestone), 50-query retrieval
# validation (Phase 3 milestone).
red_team/ # Adversarial test suites (Phase 7).
prompt_injection/ # Attempts to override system prompt and core instructions.
info_evasion/ # Attempts to leak data across access levels.
bias_probing/ # Demographic, political, religious query framing.
# Every test case must be logged — undocumented tests
# do not count toward the Phase 7 milestone.
/docs
docs/
audit_pre_phase.md # Pre-Phase 1 corpus audit (complete). Covers 25/33 documents.
# Key findings: zero-text pages, high table density in
# financial files, document scale 7–408 pages, filename-
# based temporal metadata.
ground_truth_plan.md # Ground truth construction workflow (Track A + Track B).
PROJECT_STATUS.md # Project status, decisions log, and active blockers.
# Replaces GEMINI.md — audit GEMINI.md for Google-specific
# tooling assumptions before renaming.
Root Files
main.py # CLI or API entry point to run the pipeline.
.env # Environment variables (keys, paths). Never commit.
requirements.txt # Python dependency manifest.
pyproject.toml # Build system configuration (alternative to requirements.txt).
GEMINI.md # Gemini CLI configuration. Do not rename or modify structure.
Hardware Reference
| Environment | Specification | Role |
|---|---|---|
| Local machine | No GPU, 16 GB RAM | Qdrant, BM25, cross-encoder inference, quantized LLM |
| Cloud GPU (free) | Kaggle T4/P100 or Colab T4 | Embedding generation only — one-time ingestion job |
| Workflow | Embed on cloud → serialize → load into local Qdrant | Eliminates GPU dependency at query time |
Before Phase 2: Run
htoporfree -hon a clean boot to establish actual RAM headroom. Qdrant + cross-encoder + Python runtime can approach 6–8 GB combined.
Pre-Phase 1 Gate Checklist
All items below must be complete before Phase 1 implementation code begins.
| # | Item | Status |
|---|---|---|
| 1 | YouTube reference video watched and plan updated | ✅ Complete |
| 2 | GEMINI.md audited and renamed to PROJECT_STATUS.md | ✅ Complete |
| 3 | BM25 path decided (rank_bm25 — Decision 1) | ✅ Closed |
| 4 | Agent execution model decided (Sequential — Decision 2) | ✅ Closed |
| 5 | Validation LLM decided (Same model + hardened prompts — Decision 3) | ✅ Closed |
| 6 | Project structure revised and aligned with v1.1 | ✅ Complete |
| 7 | Ground truth schema defined, file initialized (Track A, Step 1) | ☐ |
| 8 | Ground truth questions and answers written (Track A, Step 2) | ☐ |
| 9 | Validation script written and passing on empty dataset (Track A, Step 3) | ☐ |
Items 7–9 are the final gate. Phase 1 implementation begins once they are checked.
Document prepared May 2026 | Version 1.1 | Project 02 Confidential — Internal Use Only *