docs/LATENCY_LOG.md

# Latency Tracking Log — Project 02

This log tracks the end-to-end performance of the RAG pipeline. Metrics are recorded at the exit of each phase and after major architectural changes.

**Target p95 Budget:** 280ms

| Date | Phase | Description | Retrieval (ms) | Rerank (ms) | LLM Gen (ms) | Total (ms) | Status |
|---|---|---|---|---|---|---|---|
| 2026-05-07 | Phase 3 Exit | Pre-Remediation Baseline | ~120 | ~2,450 | N/A | ~2,570 | ❌ OVER BUDGET |
| 2026-05-07 | Phase 3 Exit | Post-Remediation (PyTorch) | 188.5 | 2,269.2 | N/A | 2,457.7 | ❌ OVER BUDGET |
| 2026-05-08 | Phase 3 Exit | Post-Remediation (ONNX - Est.) | 188.5 | ~500.0 | TBD | ~688.5 | ⚠️ WARM |
| 2026-05-08 | Phase 3 Exit | 8B Model CPU Baseline | 152.0 | ~500.0 | ~47,568 | ~48,220 | ❌ 3-MIN BUDGET |
| 2026-05-13 | Phase 4/5 | Groq API Integration | ~150 | ~500 | ~2,500 | ~3,150 | ✅ COMPLIANT |
| 2026-05-12 | Phase 6 Eval | Ollama Fallback (Llama-3 8B) | 152.0 | ~500.0 | > 180,000 | > 180,000 | ❌ CPU BOTTLENECK |

---

## Component Benchmarks (Averages)

### LLM Inference (Ollama - Llama-3-8B-Instruct - CPU)
*Profiled 2026-05-12. Hardware: 16GB RAM, No GPU.*

| Prompt Type | TTFT (ms) | TPS (tokens/s) | Total Time (ms) | Status |
|---|---|---|---|---|
| Routing (short) | 10,333 | 0.4 | 11,287 | Baseline |
| Summary (med) | 21,392 | 1.8 | 51,510 | Baseline |
| Reasoning (long) | 15,448 | 2.5 | 79,908 | Baseline |
| Cold Start (test) | ~170,000 | < 0.1 | 176,910 | ⚠️ CRITICAL SLOWDOWN |

---

## Strategic Risk: Hardware Latency Ceiling
The **Llama-3 8B model** on local CPU is too slow for sequential "Planner -> Agent -> Validator" logic under original RAG targets.

**Update 2026-05-12:**
Phase 6 evaluation confirmed that unsetting API keys triggers the **Ollama fallback** correctly. However, a single reasoning query (5+ LLM nodes) exceeds 15 minutes on local CPU, making a full 68-query evaluation run prohibitive without GPU acceleration or a significantly smaller model (e.g., Llama-3.2 1B).
**Decision Log:**
1. **Revised Target:** Updated `total_p95_ms` in `settings.yaml` to **180,000ms** (3 minutes).
2. **Inference Mode:** Path B (Streaming) will be implemented in Phase 8 to mitigate UX impact. Phase 4 will focus on **Accuracy/Correctness** via sequential nodes.
3. **Retrieval Optimization:** Moved `ThreadPoolExecutor` to persistent class attribute in `HybridRetriever` to reduce ~128ms spawn overhead.

### Retrieval Backends
* Qdrant (Local): ~40ms
* BM25 (In-memory): ~15ms
* RRF Fusion: ~5ms
* Overhead/Wait: ~128ms (Parallel execution latency)