SVARASETU performance
These are measured results, not estimated or hard-coded values.
Run: 2026-08-22 on Windows 11, Python 3.13.3, 16 logical CPUs, 15.63 GB RAM
Indexes: native 367,832 · sentence-window 44,489 · semantic 8,828 · total 421,149
Window: transcript to extractive answer. Sarvam STT and optional Gemini are outside the Task 2 clock.
Task 2 SLA (py -3.13 -m svarasetu.measure.sla 20)
| Layer | n | P50 | P70 | P99 | P100 | Budget | Pass |
|---|---|---|---|---|---|---|---|
| Exact cache lookup | 20 | 0.043 | 0.046 | 0.056 | 0.057 | 0.1 ms | yes |
| FAISS HNSW only | 20 | 0.439 | 0.567 | 0.951 | 0.955 | 0.1 ms P50 stretch | no |
| Embed + FAISS | 20 | 7.68 | 8.06 | 9.44 | 9.46 | 50 ms | yes |
| Full RAG cache-on | 20 | 0.243 | 0.283 | 0.350 | 0.352 | 200 ms | yes |
| Full RAG cache-off | 20 | 11.235 | 12.022 | 12.855 | 12.860 | 200 ms | yes |
The full path includes dynamic IDF lexical rescue over the loaded corpus. This prevents high dense scores from overriding missing subject evidence. Unsupported questions decline rather than receiving a static answer.
Guardrails: unsafe 2/2 blocked; off-topic/intent 2/2 blocked.
Retrieval and answer quality (py -3.13 -m svarasetu.measure.quality_eval)
| Metric | Value |
|---|---|
| Recall@1 | 0.4033 |
| Recall@5 | 0.9167 |
| Recall@10 | 0.9867 |
| Precision@5 | 0.1833 |
| MRR | 0.6096 |
The strict multilingual answer gate passed 6/6: correctness 1.0, groundedness 1.0, refusal correctness 1.0, and hallucination rate 0.0. It covers grounded English/Hindi/Marathi answers plus unsupported, unsafe, and creative-query refusals. The evaluator exits nonzero if quality gates fail.
Chunking sample
| Strategy | Mean chunks | Hashed chunks |
|---|---|---|
| passage-native | 1.00 | 12 |
| sentence-window | 5.17 | 62 |
| semantic | 1.33 | 16 |
| sliding | 1.08 | 13 |
| adaptive | 5.00 | 60 |
Live voice
py -3.13 -m svarasetu.measure.live_voice passed 15/15 checks with real Sarvam calls on 2026-08-21:
- TTS: English, Hindi, Marathi
- STT → RAG: English and Hindi
- RAG-core during live API checks: 59–135 ms
- observed Sarvam STT: about 5.7–8.3 seconds (reported separately)
- cached TTS: 8.96 ms
The deployed 2026-08-22 smoke test also passed a complete Bulbul v3 TTS → Saaras v3 STT → grounded RAG round trip. Browser voice mode uses voice-activity detection to submit after natural silence, pauses its microphone during playback, and resumes listening automatically.
Current machine-readable artifacts are in svarasetu/measure/reports/svarasetu_sla.json and svarasetu_quality_eval.json.