| # SVARASETU performance |
|
|
| These are measured results, not estimated or hard-coded values. |
|
|
| **Run:** 2026-08-22 on Windows 11, Python 3.13.3, 16 logical CPUs, 15.63 GB RAM |
|
|
| **Indexes:** native 367,832 · sentence-window 44,489 · semantic 8,828 · total 421,149 |
|
|
| **Window:** transcript to extractive answer. Sarvam STT and optional Gemini are outside the Task 2 clock. |
|
|
| ## Task 2 SLA (`py -3.13 -m svarasetu.measure.sla 20`) |
|
|
| | Layer | n | P50 | P70 | P99 | P100 | Budget | Pass | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | |
| | Exact cache lookup | 20 | 0.043 | 0.046 | 0.056 | 0.057 | 0.1 ms | yes | |
| | FAISS HNSW only | 20 | 0.439 | 0.567 | 0.951 | 0.955 | 0.1 ms P50 stretch | no | |
| | Embed + FAISS | 20 | 7.68 | 8.06 | 9.44 | 9.46 | 50 ms | yes | |
| | Full RAG cache-on | 20 | 0.243 | 0.283 | 0.350 | 0.352 | 200 ms | yes | |
| | **Full RAG cache-off** | 20 | **11.235** | **12.022** | **12.855** | **12.860** | 200 ms | **yes** | |
|
|
| The full path includes dynamic IDF lexical rescue over the loaded corpus. This prevents high dense scores from overriding missing subject evidence. Unsupported questions decline rather than receiving a static answer. |
|
|
| Guardrails: unsafe 2/2 blocked; off-topic/intent 2/2 blocked. |
|
|
| ## Retrieval and answer quality (`py -3.13 -m svarasetu.measure.quality_eval`) |
| |
| | Metric | Value | |
| | --- | ---: | |
| | Recall@1 | 0.4033 | |
| | Recall@5 | 0.9167 | |
| | Recall@10 | 0.9867 | |
| | Precision@5 | 0.1833 | |
| | MRR | 0.6096 | |
| |
| The strict multilingual answer gate passed 6/6: correctness 1.0, groundedness 1.0, refusal correctness 1.0, and hallucination rate 0.0. It covers grounded English/Hindi/Marathi answers plus unsupported, unsafe, and creative-query refusals. The evaluator exits nonzero if quality gates fail. |
| |
| ## Chunking sample |
| |
| | Strategy | Mean chunks | Hashed chunks | |
| | --- | ---: | ---: | |
| | passage-native | 1.00 | 12 | |
| | sentence-window | 5.17 | 62 | |
| | semantic | 1.33 | 16 | |
| | sliding | 1.08 | 13 | |
| | adaptive | 5.00 | 60 | |
| |
| ## Live voice |
| |
| `py -3.13 -m svarasetu.measure.live_voice` passed 15/15 checks with real Sarvam calls on 2026-08-21: |
|
|
| - TTS: English, Hindi, Marathi |
| - STT → RAG: English and Hindi |
| - RAG-core during live API checks: 59–135 ms |
| - observed Sarvam STT: about 5.7–8.3 seconds (reported separately) |
| - cached TTS: 8.96 ms |
|
|
| The deployed 2026-08-22 smoke test also passed a complete Bulbul v3 TTS → Saaras v3 STT → grounded RAG round trip. Browser voice mode uses voice-activity detection to submit after natural silence, pauses its microphone during playback, and resumes listening automatically. |
|
|
| Current machine-readable artifacts are in `svarasetu/measure/reports/svarasetu_sla.json` and `svarasetu_quality_eval.json`. |
|
|
