Spaces:

chintu4
/

documentation-crawler-rag

Sleeping

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is async/await?",
    "urls": ["https://docs.python.org/3/library/asyncio.html"]
  }'

Option C: Use Python Directly

from src.app_enhanced import answer_question

answer = answer_question(
    "How do I use decorators?",
    urls=["https://docs.python.org"]
)
print(answer)

📚 Adding Data Sources

Method 1: PDF Files

Create my_docs/ folder (already exists)
Add your PDF files there
The system will automatically load them on next query

Method 2: Website URLs

Paste URLs in the Gradio dashboard, OR
Pass urls parameter to answer_question(), OR
Use the /query API endpoint with urls field

Method 3: Crawl a Website

from src.crawler import DocumentationCrawler
from src.app_enhanced import index_crawler_results

# Crawl a documentation site
crawler = DocumentationCrawler(
    base_url="https://docs.python.org/3",
    max_depth=3,
    max_pages=100
)

documents = crawler.crawl()
index_crawler_results(documents)

# Now query
from src.app_enhanced import answer_question
answer = answer_question("What's the difference between list and tuple?")
print(answer)

🔧 API Endpoints

All endpoints require Ollama running with llama3 model.

`GET /health`

Check if the API is running

curl http://localhost:8000/health

`POST /query`

Ask a question (complete response)

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is a context manager?", "urls": ["https://realpython.com"]}'

`POST /query/stream`

Ask a question (streamed response)

curl -X POST http://localhost:8000/query/stream \
  -H "Content-Type: application/json" \
  -d '{"question": "Explain generators"}'

`POST /crawl/prepare`

Crawl a website (returns status)

curl -X POST http://localhost:8000/crawl/prepare \
  -H "Content-Type: application/json" \
  -d '{
    "base_url": "https://docs.python.org/3",
    "max_depth": 2,
    "max_pages": 50
  }'

`POST /index/from-crawl`

Crawl and automatically index

curl -X POST http://localhost:8000/index/from-crawl \
  -H "Content-Type: application/json" \
  -d '{
    "base_url": "https://docs.python.org/3",
    "max_depth": 2
  }'

📁 Project Structure

fd/
├── src/                      # Core application modules
│   ├── app_enhanced.py       # Enhanced RAG with crawler integration ⭐
│   ├── app_hf.py             # Hugging Face inference RAG app
│   ├── api.py                # FastAPI server with streaming ⭐
│   ├── crawler.py            # Web crawler module ⭐
│   ├── config.py             # Configuration helpers
│   ├── util.py               # Utility helpers
│   ├── __init__.py           # Package entrypoint
├── ui/                       # UI and dashboard scripts
│   ├── gradio-dashboard.py    # User-friendly UI ⭐
│   ├── spaces_app.py         # Spaces-compatible UI
│   └── gradio_chatbot_info.txt
├── tests/                    # Test suite
│   └── test_system.py
├── notebooks/                # Project notebooks
│   └── project_code.ipynb
├── my_docs/                  # PDF storage
├── chroma_db/                # Persisted vector DB storage
├── requirements.txt          # Dependencies
├── IMPLEMENTATION_GUIDE.md   # Detailed technical guide
└── README.md                 # This file

⭐ = New/Enhanced components

🧪 Testing

Quick Test

python tests/test_system.py

Test with Crawler

python tests/test_system.py --crawl

Test API

# Terminal 1: Start API
python -m src.api

# Terminal 2: Test it
curl http://localhost:8000/health

⚙️ Configuration

Crawler Parameters

DocumentationCrawler(
    base_url="https://...",
    max_depth=3,           # How deep to crawl
    delay=0.5,             # Politeness delay (seconds)
    timeout=10,            # Request timeout
    max_pages=100          # Stop after this many pages
)

Text Splitting

Edit in app_enhanced.py:

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # Characters per chunk
    chunk_overlap=200       # Overlap between chunks
)

Embedding Model

Edit in app_enhanced.py:

hf_embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
    # Other options:
    # - "sentence-transformers/all-mpnet-base-v2"
    # - "BAAI/bge-small-en"
)

LLM Model

Edit in app_enhanced.py:

llm = ChatOllama(
    model="llama3",     # Change to "llama2" or other available models
    temperature=0       # 0 = deterministic, 1 = creative
)

🐛 Troubleshooting

Error: "Ollama connection error"

Ensure Ollama service is running: ollama serve
Check model exists: ollama list
Pull if needed: ollama pull llama3

Error: "No documents found"

Add PDFs to my_docs/ folder, OR
Provide URLs via API/dashboard, OR
Run crawler first to crawl a website

Slow First Query

First query loads embeddings (~30-60 seconds)
Subsequent queries are cached and faster (~1-5 seconds)

Timeout During Crawl

Increase delay: delay=1.0
Reduce pages: max_pages=50
Check target site is accessible

Memory Issues

ChromaDB runs locally (low overhead)
Embeddings model cached after first load
For large crawls, reduce max_pages

📊 Performance

Typical latencies (after warmup):

Embedding generation: 5-10ms per chunk
Vector search: 50-100ms
LLM inference: 1-5 seconds
Total query time: 1-10 seconds

📖 Documentation

For more details:

✨ Features

✅ Recursive web crawling with rate limiting
✅ Local PDF loading
✅ Code-aware text chunking
✅ Local embeddings (no API key needed)
✅ Semantic search
✅ Streaming responses
✅ FastAPI with Swagger docs
✅ Gradio user interface
✅ Multi-source data integration
✅ Graceful error handling

🎯 Next Steps

Test with real docs: Try crawling your target website
Evaluate quality: Check answer accuracy and relevance
Fine-tune: Adjust chunk size, depth, embedding model
Deploy: Use production server (Gunicorn + Uvicorn)
Monitor: Log queries and feedback for improvement

Happy documenting! 🚀