docs/demo_talking_points.md

Demo Talking Points

One-Minute Summary

This project builds a Turkish legal RAG system that answers legal questions only from retrieved source passages and displays citations. The strongest live demo configuration is BM25 retrieval with an extractive grounded answer generator.

What To Show

1. Open the browser demo.
2. Ask one of the prepared questions.
3. Point to the answer and its Kaynak: line.
4. Show the retrieved sources below the answer.
5. Explain that the answer is not free-form guessing; it is grounded in the displayed source.

Safe Demo Questions

Kasten oldurme sucu nedir?
Adil yargilanma hakki nasil guvence altina alinir?
Evlilik birligi temelinden sarsilirsa ne olur?

Key Metrics

Metric	Result
BM25 Recall@10	0.975
QA Token F1	0.799
Top-5 Source Hit	0.908
Citation Accuracy	0.813
NLI Judge Faithfulness	0.858

If Asked About Fine-Tuning

• Embedding fine-tuning was run on CPU with all 2,059 triples.
• It did not improve dense retrieval, so BM25 stayed as the live demo retriever.
• Reranker fine-tuning was run on CPU with all 6,752 pairs.
• It improved strongly over the pretrained reranker, but still did not beat BM25 on the full benchmark.
• FLAN-T5 SFT smoke training was run with llm.jsonl.
• The browser and CLI demos now support an optional local_hf answer mode using the fine-tuned FLAN-T5 smoke checkpoint.
• The small CPU model is shown as an optional LLM path, while the default extractive mode remains the recommended legal demo because it is more citation-reliable.

Best Closing Line

The main result is not just a demo, but a measured RAG pipeline: we compared retrieval methods, ran fine-tuning experiments, measured citation and faithfulness, analyzed errors, and selected the safest configuration for legal QA.