docs/MAIN_DOCUMENT_GUIDE.md

Main Document Feature Guide

Overview

The Main Document feature provides guaranteed context availability for your RAG chatbot by ensuring critical profile information is always included in the LLM prompt, positioned before VectorDB retrieval results.

How It Works

1. Architecture

Query → [Main Document] → [VectorDB Retrieval] → [Combined Context] → LLM → Response
           ↑ PRIORITY         ↑ SUPPLEMENTARY

2. Loading Process

1. File Detection: Auto-detects format from extension
2. Content Loading: Uses appropriate loader (PDF, DOCX, HTML, MD, TXT)
3. Token Counting: Calculates tokens using tiktoken
4. Size Management:
   • If ≤ 10k tokens → Use as-is
   • If > 10k tokens → LLM summarization to ~8k tokens
5. Caching: Stores in memory, reloads only on file change

3. Prompt Structure

[System Prompt]

=== ESSENTIAL PROFILE INFORMATION ===
(This information is always available and takes priority)

[Main Document Content - 10k tokens max]

=== ADDITIONAL CONTEXT FROM DOCUMENTS ===

[VectorDB Retrieved Chunks]

Question: [User Question]

Answer:

Configuration

Basic Setup

main_document:
  enabled: true
  path: "data/documents/main_profile.md"
  max_tokens: 10000

Advanced Options

main_document:
  enabled: true
  path: "data/documents/main_profile.md"
  max_tokens: 10000
  position: "before"  # before/after VectorDB context

  # Summarization
  summarize_if_exceeds: true
  summarization_target_tokens: 8000
  summarization_prompt: |
    Custom summarization instructions...

  # Caching
  cache_enabled: true
  cache_check_interval: 60  # seconds

  # Error handling
  fail_silently: true
  fallback_to_vectordb_only: true

Best Practices

Content Structure

DO Include:

• ✅ Full name and professional title
• ✅ Contact information (email, LinkedIn, etc.)
• ✅ Current role and responsibilities
• ✅ Core technical skills (with proficiency levels)
• ✅ Major projects with metrics/achievements
• ✅ Education and certifications
• ✅ Career summary/objective

DON'T Include:

• ❌ Sensitive personal information
• ❌ Redundant lengthy descriptions
• ❌ Information better suited for retrieval (blog posts, detailed docs)
• ❌ Frequently changing information

Format Recommendations

Format	Best For	Notes
Markdown (.md)	Structured profiles	Readable, easy to edit, recommended
Text (.txt)	Simple profiles	Plain text, fastest to load
PDF (.pdf)	Existing resumes	Auto-parsed, may need cleanup
Word (.docx)	Existing documents	Good formatting preservation
HTML (.html)	LinkedIn exports	Auto-parsed, clean content

Token Management

Token Budget Example (llama3.2:3b with 8192 context window):

Component	Tokens	Percentage
Main Document	7,000	85%
VectorDB Context	500	6%
Output Generation	512	6%
Safety Buffer	180	3%
Total	8,192	100%

Optimization Tips:

1. Keep main doc under 7,000 tokens for optimal retrieval space
2. Use bullet points instead of verbose paragraphs
3. Remove redundant information
4. Let LLM summarize if necessary

Troubleshooting

Issue: Main document not loading

Check:

# Verify file exists
ls -lh data/documents/main_profile.md

# Check logs
tail -f app.log | grep "main_document"

Solution:

• Verify path in config.yaml is correct
• Check file permissions
• Ensure format is supported

Issue: Content truncated/summarized unexpectedly

Diagnosis:

• Check token count in logs: "Main document loaded: X tokens"
• If X > 10,000, content will be summarized

Solution:

• Reduce content length
• Increase max_tokens in config
• Disable summarization: summarize_if_exceeds: false

Issue: Responses don't seem to use main document

Check:

# In Python console:
from src.rag_pipeline import get_rag_pipeline

pipeline = get_rag_pipeline()
info = pipeline.get_main_document_info()
print(info)

Verify:

• enabled: true in info
• loaded: true in info
• tokens > 0 in info

API Reference

MainDocumentLoader Class

from src.main_document_loader import get_main_document_loader

loader = get_main_document_loader()

# Load document
content = loader.load_main_document()

# Count tokens
tokens = loader.count_tokens("your text here")

# Truncate to token limit
truncated = loader.truncate_to_tokens(content, max_tokens=5000)

# Invalidate cache (force reload)
loader.invalidate_cache()

RAGPipeline Methods

from src.rag_pipeline import get_rag_pipeline

pipeline = get_rag_pipeline()

# Get main document info
info = pipeline.get_main_document_info()
# Returns: {'enabled': bool, 'loaded': bool, 'tokens': int, 'path': str, ...}

# Reload main document at runtime
success = pipeline.reload_main_document()

# Get token budget breakdown
budget = pipeline._calculate_context_budget()

Examples

Example 1: Markdown Main Profile

# Jane Doe

**Title**: Senior Data Scientist | ML Engineer
**Email**: jane.doe@example.com
**LinkedIn**: linkedin.com/in/janedoe
**Location**: San Francisco, CA

## Professional Summary
Data Scientist with 8+ years building production ML systems. Specialized in NLP, recommendation systems, and MLOps.

## Core Skills
- **Languages**: Python, SQL, R
- **ML/DL**: PyTorch, TensorFlow, scikit-learn, Hugging Face
- **MLOps**: Docker, Kubernetes, MLflow, AWS SageMaker
- **Data**: Spark, Airflow, PostgreSQL, Redis

## Experience

### Tech Corp - Senior Data Scientist (2020-Present)
- Built recommendation engine serving 5M+ users, improving CTR by 23%
- Deployed 12 ML models to production using CI/CD pipelines
- Led team of 4 data scientists on personalization initiatives

### StartupXYZ - Data Scientist (2017-2020)
- Developed NLP pipeline for customer support automation (85% accuracy)
- Reduced model training costs by 40% through infrastructure optimization

## Education
- **M.S. Computer Science**, Stanford University (2017)
- **B.S. Mathematics**, UC Berkeley (2015)

## Certifications
- AWS Certified Machine Learning - Specialty
- Google Professional Data Engineer

Example 2: Minimal Text Profile

John Smith - AI Research Engineer
Email: john.smith@email.com
GitHub: github.com/johnsmith

5+ years in deep learning and computer vision.

Key Skills: PyTorch, TensorFlow, OpenCV, CUDA, Python, C++

Notable Projects:
- Real-time object detection system (45 FPS, 92% mAP)
- Federated learning framework for healthcare ML
- Open-source contributor: torchvision, detectron2

Education: Ph.D. Computer Science, MIT (2019)

Migration Guide

From VectorDB-Only to Main Document

Before:

• All content in data/documents/ processed equally
• Retrieval quality determines context

After:

1. Identify critical information that must always be available
2. Create main document with this essential content
3. Enable feature in config.yaml
4. Keep other documents in data/documents/ for supplementary info
5. Test queries to verify main document is used

You don't need to remove existing documents - main document supplements, not replaces, VectorDB retrieval!

Advanced: Token Budget Calculation

The system automatically calculates and manages token budgets:

budget = {
    "model_context_window": 8192,      # Total available
    "main_doc_tokens": 7000,           # Main document
    "max_output_tokens": 512,          # LLM generation
    "buffer_tokens": 500,              # Safety buffer
    "available_for_retrieval": 180,    # Remaining for VectorDB
    "total_input_budget": 7680,        # Total input space
}

Warnings:

• If main doc > 50% of context window, system logs warning
• Adjust max_tokens or enable summarization to optimize

Performance Considerations

Memory Usage

• Cached content: ~1-5 MB per main document
• Token counting: Negligible (< 10ms)
• Summarization: 2-10 seconds (only if needed)

Loading Times

• First load: 50-200ms (depends on format)
• Cached loads: < 1ms
• Cache validation: < 5ms (file hash check)

Recommendations

• Enable caching for production (default: true)
• Set cache_check_interval to 300+ for rarely changing docs
• Use markdown or text for fastest loading

---

For more help, see ARCHITECTURE.md or README.md.