Spaces:
Sleeping
Sleeping
Document Ingestion Pipeline - Detailed Explanation
Date: March 26, 2026
System: RBAC-Enforced RAG with Groq + SentenceTransformer
Overview
The ingestion pipeline transforms raw documents into queryable chunks with embeddings and RBAC metadata. It consists of 7 stages designed to preserve document hierarchy while enabling secure, semantic search.
RAW DOCUMENT → PARSE → POST-PROCESS → EXTRACT HIERARCHY → CHUNK → EMBED → STORE
(PDF) (Docling) (ResultPostprocessor) (Tree walk) (512tok) (384dim) (Qdrant)
Stage 1: Document Parsing (Docling)
Input
- File format: PDF, DOCX, Markdown, TXT
- File location: Provided via API endpoint
/admin/ingest - Maximum size: 100MB (configurable)
Process
┌──────────────────────────────────────────────────────┐
│ DOCLING PARSER INITIALIZATION │
│ │
│ DocumentConverter() │
│ ├─ PDF handler: pdfplumber │
│ ├─ DOCX handler: python-docx │
│ ├─ Markdown handler: markdown parser │
│ └─ Auto-detects format from extension │
└──────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────┐
│ Validate file │
│ ✓ Exists │
│ ✓ Readable │
│ ✓ Size within limits │
└────────┬───────────────────────┘
│
▼
┌────────────────────────────────┐
│ converter.convert(file_path) │
│ │
│ Returns: ConversionResult │
│ Field: .document │
│ Type: DoclingDocument │
└────────┬───────────────────────┘
│
▼
┌────────────────────────────────┐
│ Extract document structure │
│ ✓ Heading levels │
│ ✓ Table of contents │
│ ✓ Section breaks │
│ ✓ Inline formatting │
│ ✓ Tables & lists │
└────────────────────────────────┘
Output
DoclingDocument {
blocks: [ # Structured content blocks
Header, # # Heading 1
Paragraph, # Body text
List, # Bullet/numbered lists
Table, # Tabular data
...
],
metadata: {
title,
author,
created_date,
}
}
Code Location
File: ingestion/docling_parser.py:parse_document()
result = self.converter.convert(path)
return {
"document": result.document,
"text": result.document.export_to_markdown(),
}
Stage 2: Post-Processing (Hierarchy Preservation)
Purpose
Maintain hierarchical structure after parsing. Some documents lose structure during parsing — post-processing restores it.
Process
PARSED DOCLING DOCUMENT
│
▼
┌──────────────────────────────────────────────────┐
│ ResultPostprocessor(result) │
│ │
│ Analyzes document structure: │
│ ✓ Identifies header levels (H1, H2, H3...) │
│ ✓ Groups content by section │
│ ✓ Preserves nesting relationships │
│ ✓ Maintains reading order │
│ ✓ Reconstructs table of contents │
└──────────┬───────────────────────────────────────┘
│
▼
.process() ← Returns processed result
│
├─ ✅ Success: Return structured document
│ (hierarchy preserved)
│
└─ ❌ Fail: Use raw document as fallback
(graceful degradation)
Key Features
| Feature | Benefit |
|---|---|
| Header Level Detection | Understand document structure |
| Nesting Preservation | Maintain parent-child relationships |
| Reading Order | Correct text flow especially with multi-column |
| Table/List Handling | Keep tabular data intact |
Code Location
File: ingestion/docling_parser.py:parse_document() (Lines 38-47)
# Post-process result to maintain hierarchical structure
try:
result_postprocessor = ResultPostprocessor(result)
result = result_postprocessor.process()
logger.debug(f"Applied post-processing to {path.name}")
except Exception as e:
logger.warning(f"Post-processing failed, using raw result: {str(e)}")
# Continue with raw result if post-processing fails
Stage 3: Hierarchy Extraction
Purpose
Build a tree representation of the document structure for chunking.
Process
POST-PROCESSED DOCUMENT
│
▼
walk_document_tree()
│
├─ Recursive depth-first traversal
│
└─ Extract at each level:
├─ Element type (Header, Paragraph, List, Table)
├─ Content text
├─ Hierarchy depth
├─ Parent element ID
└─ Parent section title
HIERARCHY STRUCTURE:
Depth 0: Document root
│
Depth 1: ├─ # Introduction
│ │
Depth 2: │ ├─ ## Background
│ │ │
Depth 3: │ │ ├─ ### Key Concepts
│ │ └─ ### Related Work
│ │
│ └─ ## Methodology
│
Depth 1: └─ # Results
│
Depth 2: └─ ## Findings
Generated Hierarchy Data
[
(0, {
"id": "doc_001_root",
"type": "Document",
"text": "Overall content...",
"depth": 0,
}, None), # parent_id = None (root)
(1, {
"id": "doc_001_h1_intro",
"type": "Header",
"text": "Introduction",
"depth": 1,
"is_heading": True,
}, "doc_001_root"), # parent = root
(2, {
"id": "doc_001_h2_bg",
"type": "Header",
"text": "Background",
"depth": 2,
"is_heading": True,
}, "doc_001_h1_intro"), # parent = intro
(2, {
"id": "doc_001_p_bg_content",
"type": "Paragraph",
"text": "The background explains...",
"depth": 2,
}, "doc_001_h2_bg"), # parent = background section
]
Code Location
File: ingestion/docling_parser.py:extract_hierarchy()
Method: _walk_document_tree() (recursive)
Stage 4: Hierarchical Chunking
Purpose
Break document into semantic chunks while preserving context and hierarchy.
Process
Step 1: Split into Paragraphs
CLEANED DOCUMENT TEXT
│
▼
Split by:
├─ Markdown headers (#, ##, ###, etc.)
├─ Double newlines (paragraph breaks)
└─ Logical section boundaries
Output: List of paragraphs
Step 2: Build Section Summaries
PARAGRAPHS
│
▼
Group by hierarchy level:
├─ Section 1 (H1: Introduction)
│ │
│ ├─ Subsection 1.1 (H2: Background)
│ │ ├─ Content paragraph
│ │ ├─ Content paragraph
│ │ └─ Content paragraph
│ │
│ └─ Subsection 1.2 (Methodology)
│ └─ [paragraphs]
│
└─ Section 2 (H1: Results)
└─ [paragraphs]
Result: Section metadata for context injection
Step 3: Tokenize & Split
EACH PARAGRAPH/SECTION
│
▼
Count tokens
│
┌───┴──────┐
│ │
< 512 tok ≥ 512 tok
│ │
├─ Keep └─ Split recursively
│
└─ One chunk
Step 4: Apply Overlap
CHUNKS:
Chunk 1: [Tok 0-512]
││
│└─ Overlap region (20%)
│
Chunk 2: [Tok 410-922] ← Starts at 410 (20% overlap)
││
│└─ Overlap region (20%)
│
Chunk 3: [Tok 738-1250] ← Starts at 738 (20% overlap)
Benefits:
✓ Context continuity
✓ Semantic coherence
✓ Prevents mid-sentence cuts
✓ Enables cross-chunk relationships
Configuration
# From config.py
CHUNKING_CONFIG = {
"max_leaf_chunk_tokens": 512, # Max tokens per chunk
"overlap_tokens": 102, # ~20% for 512-tok chunks
"min_chunk_tokens": 50, # Skip tiny chunks
"chunk_type_detection": True, # Detect paragraph types
}
Output: Chunk Objects
Chunk {
id: "finance_policy_chunk_001",
text: "The financial policy states...",
source_document: "finance_policy.pdf",
collection: "finance",
access_roles: ["finance", "c_level"],
# Hierarchy context
section_title: "Financial Policies",
subsection_title: "Investment Guidelines",
depth: 2,
parent_chunk_id: "finance_policy_chunk_000",
parent_summary: "This section covers key policy areas...",
# Content type
chunk_type: "paragraph",
page_number: 5,
}
Code Location
File: ingestion/hierarchical_chunker.py:chunk_document()
Key Methods:
_split_into_paragraphs()— Split by structure_split_paragraph_into_chunks()— Tokenize_build_section_summaries()— Create context
Stage 5: Add RBAC Metadata
Purpose
Attach role-based access control information to chunks.
Process
CHUNK FROM STAGE 4
│
▼
SET ACCESS CONTROL
┌────────────────────────────────┐
│ collection → access_roles map │
│ │
│ Collection: "finance" │
│ Maps to roles: │
│ ├─ "finance" (direct access) │
│ └─ "c_level" (executive) │
└────────┬───────────────────────┘
│
▼
ADD METADATA FILTERS
┌────────────────────────────────┐
│ { │
│ "collection": "finance", │
│ "access_roles": [ │
│ "finance", │
│ "c_level" │
│ ], │
│ "section_title": "...", │
│ "chunk_type": "paragraph", │
│ "source_doc": "policy.pdf" │
│ } │
└────────┬───────────────────────┘
│
▼
GENERATE UNIQUE ID
finance_policy_001 (deterministic hash)
RBAC Role-to-Collection Mapping
ROLE_COLLECTION_ACCESS = {
"employee": ["general"],
"finance": ["general", "finance"],
"engineering": ["general", "engineering"],
"marketing": ["general", "marketing"],
"c_level": ["general", "finance", "engineering", "marketing", "hr"],
}
Enforcement: When searching, filter by:
Qdrant filter: {
"access_roles": {"any": [user_role]}
}
Only chunks marked as accessible by the user's role will be returned.
Stage 6: Generate Embeddings
Purpose
Convert chunk text to semantic vectors for similarity search.
Process
CHUNKS WITH METADATA
│
▼
FOR EACH CHUNK:
├─ chunk.text
│
▼
SentenceTransformer(
model="all-MiniLM-L6-v2"
)
│
├─ Input: Text string
│ (max ~512 tokens, already chunk size)
│
├─ Processing:
│ 1. Tokenize (subwords)
│ 2. Embed with transformer
│ 3. Pool: extract [CLS] token
│ 4. Normalize: L2 normalization
│
▼
OUTPUT: 384-dimensional vector
[0.234, -0.156, 0.892, ..., 0.123]
(384 float values)
PERFORMANCE:
┌─────────────────────────────────────┐
│ Latency: ~10ms per chunk │
│ Model size: ~80MB (on disk) │
│ Memory: ~200MB (loaded) │
│ Cost: FREE (local) │
│ Alternative: OpenAI (optional) │
│ - Cost: $0.02/1M tokens │
│ - Latency: 100ms per chunk │
│ - Dimensions: 1536 (larger) │
└─────────────────────────────────────┘
Why SentenceTransformer:
✓ Local inference (no API calls)
✓ Fast (10x faster than API)
✓ Free (no per-token cost)
✓ Privacy (no data sent to OpenAI)
✓ Offline capable (works without internet)
✓ Proven for semantic search (384 dims sufficient)
Vector + Metadata Package
PointStruct {
id: 12345,
vector: [0.234, -0.156, ..., 0.123], # 384 floats
payload: {
"chunk_text": "The policy...",
"source_document": "finance_policy.pdf",
"collection": "finance",
"access_roles": ["finance", "c_level"],
"section_title": "Investment Guidelines",
"chunk_type": "paragraph",
"depth": 2,
}
}
Code Location
File: vector_store.py:embed_chunks()
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(
[chunk.text for chunk in chunks]
) # Returns: List[List[float]] (384-dim vectors)
Stage 7: Store in Qdrant Vector Database
Purpose
Index vectors and metadata for fast semantic search with RBAC filtering.
Process
EMBEDDINGS + METADATA
│
▼
┌──────────────────────────────────────┐
│ QDRANT COLLECTION SETUP │
│ │
│ collection_name: "document_chunks"│
│ vector_size: 384 │
│ distance_metric: cosine │
│ indexing_config: HNSW │
└──────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ ADD POINTS TO INDEX │
│ │
│ for each chunk: │
│ ├─ Point ID (sequential) │
│ ├─ Vector (384 floats) │
│ ├─ Metadata payload │
│ │ ├─ access_roles: [...] │
│ │ ├─ collection: "finance" │
│ │ └─ ... (other fields) │
│ │ │
│ └─ Insert into index │
└──────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ BUILD VECTOR INDEX │
│ │
│ Algorithm: HNSW │
│ (Hierarchical Navigable Small World)
│ │
│ Benefits: │
│ ✓ Fast approximate search │
│ ✓ Memory efficient │
│ ✓ Scales to millions of vectors │
│ ✓ Sub-millisecond queries │
└──────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ READY FOR SEARCH │
│ │
│ Search query: │
│ ├─ Embed query (SentenceTransformer)
│ ├─ Find similar vectors (HNSW) │
│ ├─ Filter by access_roles │
│ └─ Return top-k chunks │
└──────────────────────────────────────┘
Qdrant Storage Structure
Collection: document_chunks
Vector Config:
size: 384
distance: cosine
hnsw:
m: 16 # Connections per node
ef_construct: 200
ef: 100
Points:
- id: 1
vector: [0.234, -0.156, ..., 0.123]
payload:
chunk_text: "..."
source_document: "annual_budget_report.docx"
collection: "finance"
access_roles: ["finance", "c_level"]
section_title: "Investment Guidelines"
chunk_type: "paragraph"
depth: 2
page_number: 5
- id: 2
vector: [0.445, 0.678, ..., -0.234]
payload:
# ... similar structure
Query-Time Search
USER QUERY (e.g., "What's the financial policy?")
│
▼
EMBED QUERY
model.encode("What's the financial policy?")
│
▼
[0.123, 0.456, ..., 0.789] (384 floats)
│
▼
QDRANT SEARCH
{
"vector": [0.123, 0.456, ..., 0.789],
"limit": 5,
"filter": {
"access_roles": {
"any": ["finance"] # User role
}
}
}
│
▼
RESULTS (top-5 by similarity):
[
{id: 1, score: 0.92, payload: {...}},
{id: 5, score: 0.87, payload: {...}},
{id: 12, score: 0.81, payload: {...}},
{id: 8, score: 0.79, payload: {...}},
{id: 15, score: 0.76, payload: {...}},
]
Code Location
File: vector_store.py:store_chunks()
client = QdrantClient(":memory:") # or cloud URL
client.upsert(
collection_name="document_chunks",
points=[
PointStruct(
id=chunk_id,
vector=embedding,
payload=chunk_metadata,
)
for chunk_id, embedding in zip(chunk_ids, embeddings)
]
)
End-to-End Data Flow Example
Scenario: Ingesting a Finance PDF
1. UPLOAD STAGE
File: /uploads/annual_budget_report.docx
Size: 2.5 MB
Type: DOCX
2. PARSE STAGE (Docling)
✓ Converted to DoclingDocument
✓ Extracted: 250 paragraphs, 15 tables, 8 sections
✓ Hierarchy: 3 levels deep (H1, H2, H3)
3. POST-PROCESS STAGE
✓ ResultPostprocessor applied
✓ Hierarchy preserved
✓ Headers recognized: H1 (3), H2 (8), H3 (15)
4. EXTRACT HIERARCHY
✓ Tree built: 26 nodes
✓ Parent-child relationships: 23
✓ Depth levels: 0-3
5. CHUNK STAGE
✓ Split into 50 paragraphs
✓ Applied overlap: 20%
✓ Created chunks:
- avg_size: 256 tokens
- count: 47 chunks
- min_size: 50 tokens
- max_size: 512 tokens
6. METADATA STAGE
✓ Collection: "finance"
✓ Access roles: ["finance", "c_level"]
✓ Unique IDs: financial_policy_2024_001, ..., _047
7. EMBEDDING STAGE
✓ Model: all-MiniLM-L6-v2
✓ Encoded 47 chunks
✓ Total time: ~470ms (10ms per chunk)
✓ Vectors: 47 × 384 float array
8. STORE STAGE
✓ Created Qdrant points
✓ Added to collection: document_chunks
✓ Indexed for search
✓ Ready for queries
FINAL RESULT:
✅ 47 searchable chunks
✅ Full hierarchy preserved
✅ RBAC enforced at search time
✅ Latency: ~500ms (parsing + chunking + embedding)
✅ Cost: FREE (all local operations)
Configuration & Tuning
Chunking Parameters
CHUNKING_CONFIG = {
"max_leaf_chunk_tokens": 512,
"overlap_tokens": 102, # 20% of 512
"min_chunk_tokens": 50,
"chunk_type_detection": True,
}
Impact:
- Larger chunks (512+): Better context, fewer chunks, higher latency
- Smaller chunks (<256): More chunks, better granularity, may split sentences
- Higher overlap (30%): Better context preservation, more redundancy
- Lower overlap (10%): Fewer chunks, may lose context at boundaries
Embedding Model Selection
| Model | Dimensions | Speed | Cost | Use Case |
|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | 10ms | FREE | ✅ Default (balanced) |
| all-mpnet-base | 768 | 20ms | FREE | Slower but more accurate |
| all-MiniLM-L12-v2 | 384 | 15ms | FREE | Better accuracy than L6 |
| OpenAI embedding | 1536 | 100ms | $0.02/M | Deprecated (costly) |
Qdrant Configuration
QDRANT_CONFIG = {
"vector_size": 384,
"distance": "cosine", # Semantic similarity
"hnsw": {
"m": 16, # Connections per node
"ef_construct": 200,
"ef": 100,
}
}
# Memory mode (dev/testing)
client = QdrantClient(":memory:")
# Persistent (production)
client = QdrantClient("./qdrant_storage")
# Cloud / Production (Persistent & Managed)
# Mandatory for free-tier hosting (Hugging Face Spaces) to persist data
client = QdrantClient(
url=os.getenv("QDRANT_URL"),
api_key=os.getenv("QDRANT_API_KEY"),
prefer_grpc=True
)
Performance Metrics
Ingestion Times (per 100 chunks)
| Stage | Time | % of Total |
|---|---|---|
| Parse (PDF) | 200ms | 15% |
| Post-process | 50ms | 4% |
| Extract hierarchy | 30ms | 2% |
| Chunk | 100mm | 7% |
| Metadata | 20ms | 1% |
| Embed | 1000ms | 71% |
| Store (Qdrant) | 30ms | 2% |
| TOTAL | 1430ms | 100% |
Bottleneck: Embedding generation (SentenceTransformer)
Optimization: Batch encode all chunks at once (vs. one-by-one)
Storage Size (per 100 chunks)
| Component | Size |
|---|---|
| Raw text | 50 KB |
| Metadata | 5 KB |
| Embeddings (384 × 100 floats) | 150 KB |
| Qdrant index overhead | 50 KB |
| TOTAL | ~255 KB |
For 10,000 chunks: ~25 MB (easily fits in memory)
Error Handling & Graceful Degradation
Stage-by-Stage Resilience
Parse Error
├─ Corrupted PDF
├─ Unsupported format
└─ → Log & skip file
Post-process Error
├─ Hierarchy extraction fails
└─ → Use raw document (degraded)
Chunking Error
├─ Text encoding fails
└─ → Use whole text as one chunk
Embedding Error
├─ SentenceTransformer fails
└─ → Log & skip (user alerted)
Storage Error
├─ Qdrant unavailable
└─ → Queue for later ingestion
(persist to disk)
Fallback Strategy
# If post-processing fails
try:
result = ResultPostprocessor(result).process()
except Exception:
result = raw_result # Use raw parse
# If embedding fails
try:
embeddings = model.encode(chunks)
except Exception:
embeddings = dummy_embeddings # Use fallback
log_error()
# If Qdrant store fails
try:
client.upsert(...)
except Exception:
save_to_pending_queue()
schedule_retry()
Security: RBAC at Ingestion Time
Access Control Metadata
Every chunk stores the roles that can access it:
chunk.access_roles = ["finance", "c_level"]
Multi-Layer Enforcement
| Layer | When | How |
|---|---|---|
| Ingestion | Document added | Assign to collection with roles |
| Retrieval | Search query | Filter by user role |
| Database | Vector search | Qdrant filter by access_roles |
| Response | Return results | Only approved chunks |
Example
# Finance department adds confidential budget document
chunk = Chunk(
collection="finance",
access_roles=["finance", "c_level"], # Only these roles
)
# Later: Employee searches
# User role = "employee"
# Qdrant filter: access_roles contains "employee"?
# → NO → 0 results (cannot see this chunk)
# Later: CFO searches
# User role = "c_level"
# Qdrant filter: access_roles contains "c_level"?
# → YES → Document returned
Summary
Ingestion Pipeline: 7 Stages
- Parse — Docling converts file to structured document
- Post-Process — ResultPostprocessor maintains hierarchy
- Extract — Walk tree, build parent-child relationships
- Chunk — Split into ~512-token chunks with 20% overlap
- Metadata — Add RBAC roles and collection info
- Embed — SentenceTransformer generates 384-dim vectors
- Store — Qdrant indexes vectors + metadata for search
Key Features
✅ Preserves document hierarchy
✅ Enforces RBAC at chunk level
✅ Fast local embeddings (10x better than OpenAI API)
✅ Graceful error handling (fallbacks at each stage)
✅ Efficient storage (~250KB per 100 chunks)
✅ Production-ready with proper logging
Performance
- Latency: ~1.4 seconds per 100 chunks
- Cost: FREE (all local operations)
- Throughput: ~50-70 chunks/second (limited by embedding)
- Storage: ~2.5 MB per 10,000 chunks
Next: Use ingested chunks for semantic retrieval in RAG pipeline!