# Quick Start Guide: Intelligent Documentation Crawler & RAG Assistant

## 🚀 Getting Started (5 minutes)

### Step 1: Prerequisites
- Python 3.10+ (already installed: Python 3.11)
- Ollama running locally with `llama3` model
- All dependencies installed (see `requiements.txt`)

### Step 2: Start Ollama (in a separate terminal)
```bash
ollama serve
```

Then in another terminal, ensure the model is pulled:
```bash
ollama pull llama3
```

### Step 3: Run the Dashboard
```bash
python ui/gradio-dashboard.py
```

Open: `http://localhost:7860`

### Step 4: Use It!

**Option A: Use the Gradio UI**
1. Paste URLs in the right sidebar (one per line)
2. Type your question in the chat
3. Get instant answers from the crawled/uploaded documents

**Option B: Use the FastAPI Server**
```bash
python -m src.api
```

Then query via curl:
```bash
curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is async/await?",
    "urls": ["https://docs.python.org/3/library/asyncio.html"]
  }'
```

**Option C: Use Python Directly**
```python
from src.app_enhanced import answer_question

answer = answer_question(
    "How do I use decorators?",
    urls=["https://docs.python.org"]
)
print(answer)
```

---

## 📚 Adding Data Sources

### Method 1: PDF Files
1. Create `my_docs/` folder (already exists)
2. Add your PDF files there
3. The system will automatically load them on next query

### Method 2: Website URLs
1. Paste URLs in the Gradio dashboard, OR
2. Pass `urls` parameter to `answer_question()`, OR
3. Use the `/query` API endpoint with `urls` field

### Method 3: Crawl a Website
```python
from src.crawler import DocumentationCrawler
from src.app_enhanced import index_crawler_results

# Crawl a documentation site
crawler = DocumentationCrawler(
    base_url="https://docs.python.org/3",
    max_depth=3,
    max_pages=100
)

documents = crawler.crawl()
index_crawler_results(documents)

# Now query
from src.app_enhanced import answer_question
answer = answer_question("What's the difference between list and tuple?")
print(answer)
```

---

## 🔧 API Endpoints

All endpoints require Ollama running with `llama3` model.

### `GET /health`
Check if the API is running
```bash
curl http://localhost:8000/health
```

### `POST /query`
Ask a question (complete response)
```bash
curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is a context manager?", "urls": ["https://realpython.com"]}'
```

### `POST /query/stream`
Ask a question (streamed response)
```bash
curl -X POST http://localhost:8000/query/stream \
  -H "Content-Type: application/json" \
  -d '{"question": "Explain generators"}'
```

### `POST /crawl/prepare`
Crawl a website (returns status)
```bash
curl -X POST http://localhost:8000/crawl/prepare \
  -H "Content-Type: application/json" \
  -d '{
    "base_url": "https://docs.python.org/3",
    "max_depth": 2,
    "max_pages": 50
  }'
```

### `POST /index/from-crawl`
Crawl and automatically index
```bash
curl -X POST http://localhost:8000/index/from-crawl \
  -H "Content-Type: application/json" \
  -d '{
    "base_url": "https://docs.python.org/3",
    "max_depth": 2
  }'
```

---

## 📁 Project Structure

```
fd/
├── src/                      # Core application modules
│   ├── app_enhanced.py       # Enhanced RAG with crawler integration ⭐
│   ├── app_hf.py             # Hugging Face inference RAG app
│   ├── api.py                # FastAPI server with streaming ⭐
│   ├── crawler.py            # Web crawler module ⭐
│   ├── config.py             # Configuration helpers
│   ├── util.py               # Utility helpers
│   ├── __init__.py           # Package entrypoint
├── ui/                       # UI and dashboard scripts
│   ├── gradio-dashboard.py    # User-friendly UI ⭐
│   ├── spaces_app.py         # Spaces-compatible UI
│   └── gradio_chatbot_info.txt
├── tests/                    # Test suite
│   └── test_system.py
├── notebooks/                # Project notebooks
│   └── project_code.ipynb
├── my_docs/                  # PDF storage
├── chroma_db/                # Persisted vector DB storage
├── requirements.txt          # Dependencies
├── IMPLEMENTATION_GUIDE.md   # Detailed technical guide
└── README.md                 # This file
```

⭐ = New/Enhanced components

---

## 🧪 Testing

### Quick Test
```bash
python tests/test_system.py
```

### Test with Crawler
```bash
python tests/test_system.py --crawl
```

### Test API
```bash
# Terminal 1: Start API
python -m src.api

# Terminal 2: Test it
curl http://localhost:8000/health
```

---

## ⚙️ Configuration

### Crawler Parameters
```python
DocumentationCrawler(
    base_url="https://...",
    max_depth=3,           # How deep to crawl
    delay=0.5,             # Politeness delay (seconds)
    timeout=10,            # Request timeout
    max_pages=100          # Stop after this many pages
)
```

### Text Splitting
Edit in `app_enhanced.py`:
```python
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # Characters per chunk
    chunk_overlap=200       # Overlap between chunks
)
```

### Embedding Model
Edit in `app_enhanced.py`:
```python
hf_embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
    # Other options:
    # - "sentence-transformers/all-mpnet-base-v2"
    # - "BAAI/bge-small-en"
)
```

### LLM Model
Edit in `app_enhanced.py`:
```python
llm = ChatOllama(
    model="llama3",     # Change to "llama2" or other available models
    temperature=0       # 0 = deterministic, 1 = creative
)
```

---

## 🐛 Troubleshooting

### Error: "Ollama connection error"
- Ensure Ollama service is running: `ollama serve`
- Check model exists: `ollama list`
- Pull if needed: `ollama pull llama3`

### Error: "No documents found"
- Add PDFs to `my_docs/` folder, OR
- Provide URLs via API/dashboard, OR
- Run crawler first to crawl a website

### Slow First Query
- First query loads embeddings (~30-60 seconds)
- Subsequent queries are cached and faster (~1-5 seconds)

### Timeout During Crawl
- Increase delay: `delay=1.0`
- Reduce pages: `max_pages=50`
- Check target site is accessible

### Memory Issues
- ChromaDB runs locally (low overhead)
- Embeddings model cached after first load
- For large crawls, reduce `max_pages`

---

## 📊 Performance

Typical latencies (after warmup):
- **Embedding generation**: 5-10ms per chunk
- **Vector search**: 50-100ms
- **LLM inference**: 1-5 seconds
- **Total query time**: 1-10 seconds

---

## 📖 Documentation

For more details:
- [Full Implementation Guide](IMPLEMENTATION_GUIDE.md)
- [SRS Specification](srs.md)
- [API Swagger Docs](http://localhost:8000/docs) (when API running)

---

## ✨ Features

✅ Recursive web crawling with rate limiting  
✅ Local PDF loading  
✅ Code-aware text chunking  
✅ Local embeddings (no API key needed)  
✅ Semantic search  
✅ Streaming responses  
✅ FastAPI with Swagger docs  
✅ Gradio user interface  
✅ Multi-source data integration  
✅ Graceful error handling  

---

## 🎯 Next Steps

1. **Test with real docs**: Try crawling your target website
2. **Evaluate quality**: Check answer accuracy and relevance
3. **Fine-tune**: Adjust chunk size, depth, embedding model
4. **Deploy**: Use production server (Gunicorn + Uvicorn)
5. **Monitor**: Log queries and feedback for improvement

---

**Happy documenting! 🚀**