README.md

---
title: Nlp Project Turkish Legal Rag
emoji: ⚖️
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---

# Turkish Legal RAG - CENG493

This repository contains a step-by-step implementation of a Turkish legal question answering system with Retrieval-Augmented Generation (RAG).

## Instructor Quick Run

Validate the included custom-data example:

```bash
python scripts/validate_custom_data.py --data-dir sample_custom_data --require-benchmark
```

Run the reliable default demo:

```bash
python scripts/demo_app.py --data-dir data
```

The browser demo also includes a `Custom Document Test` panel. An instructor can upload a `.txt`, `.md`, `.csv`, `.json`, `.jsonl`, `.docx`, or `.pdf` file, ask a question over that uploaded file, and inspect the retrieved chunks/citations in the same page.

Run a submission-style Base RAG vs Fine-tuned RAG comparison after training or providing a fine-tuned reranker checkpoint:

```bash
python scripts/train_cross_encoder_reranker.py --data-dir data --epochs 1 --batch-size 4 --max-length 128 --output-dir outputs/models/legal_cross_encoder_reranker_full_cpu_128

python scripts/run_base_vs_finetuned_eval.py --data-dir data --output-dir outputs/submission_eval_bm25_vs_finetuned_reranker --base-retriever bm25 --finetuned-retriever bm25 --finetuned-reranker-model outputs/models/legal_cross_encoder_reranker_full_cpu_128
```

The fine-tuned checkpoints under `outputs/models/` are generated artifacts and are not required for the default demo. If they are not present after cloning, recreate them with the training commands above or use the documented base-model runs.

## Data

Expected dataset directory:

```text
Datasets_Ceng493_legal_rag/
  corpus.jsonl
  embedding.jsonl
  gold_benchmark.json
  llm.jsonl
  rag_eval.json
  reranker.jsonl
```

## Step 1: Baseline Retrieval

Dense retrieval:

```bash
python scripts/evaluate_retrieval.py --retriever dense --top-k 10
```

BM25 retrieval:

```bash
python scripts/evaluate_retrieval.py --retriever bm25 --top-k 10
```

Hybrid retrieval:

```bash
python scripts/evaluate_retrieval.py --retriever hybrid --top-k 10
```

BM25 + cross-encoder reranking:

```bash
python scripts/evaluate_reranker.py --candidate-k 50 --top-k 10
```

Optional cross-encoder fine-tuning:

```bash
python scripts/train_cross_encoder_reranker.py --epochs 1 --batch-size 8
python scripts/evaluate_reranker.py --reranker-model outputs/models/legal_cross_encoder_reranker
```

On the local CPU-only machine, full reranker fine-tuning was run with `batch-size 4` and `max-length 128`. It improved strongly over the pretrained reranker, but direct BM25 ranking remained best on the full retrieval benchmark. See `docs/step2_reranker_results.md`.

Optional embedding fine-tuning:

```bash
python scripts/train_embedding_model.py --epochs 1 --batch-size 16
python scripts/evaluate_retrieval.py --retriever dense --embedding-model outputs/models/legal_embedding_triplet
```

On the local CPU-only machine, full triplet fine-tuning was also run with `batch-size 4` and `max-seq-length 256`. The fine-tuned dense model scored lower than the base dense model, so the demo keeps BM25 as the primary retriever. See `docs/step3_embedding_tuning.md`.

Optional seq2seq generator SFT smoke:

```bash
python scripts/train_seq2seq_generator.py --limit 512 --eval-size 64 --epochs 1 --batch-size 2 --grad-accum 8
```

The CPU smoke run verifies the `llm.jsonl` training path, but the final demo uses the extractive generator because it is more reliable for citations.

## Step 2: Baseline RAG Answers

This creates source-grounded extractive baseline answers from the retrieved context.

```bash
python scripts/run_baseline_rag.py --retriever hybrid --top-k 5 --limit 20
```

Richer QA evaluation with citation and faithfulness proxy metrics:

```bash
python scripts/evaluate_qa.py --retriever bm25 --generation-mode extractive --top-k 5
python scripts/analyze_qa_errors.py --input outputs/qa_eval.json
```

Outputs are written under `outputs/`.

## Live Demo

For the instructor demo, use the dependency-light browser demo:

```bash
python scripts/demo_app.py --data-dir data
```

On Windows, the same demo can be launched with:

```bash
run_demo.bat
```

Then open:

```text
http://127.0.0.1:7860
```

Reliable demo questions for the included small corpus:

```text
Kasten oldurme sucu nedir?
Adil yargilanma hakki nasil guvence altina alinir?
Evlilik birligi temelinden sarsilirsa ne olur?
```

The demo shows:

- the generated grounded answer,
- the retrieved source passages,
- the citation/source label,
- and the headline evaluation metrics.

Terminal-only fallback:

```bash
python scripts/demo_cli.py --data-dir data --question "Kasten oldurme sucu nedir?"
```

Optional local generative LLM demo with the CPU fine-tuned FLAN-T5 smoke checkpoint:

```bash
python scripts/demo_app.py --data-dir data --answer-mode local_hf --generation-model outputs/models/flan_t5_legal_sft_smoke_512
```

If the generated checkpoint is not present after cloning, recreate it with:

```bash
python scripts/train_seq2seq_generator.py --data-dir data --limit 512 --eval-size 64 --epochs 1 --batch-size 2 --grad-accum 8 --output-dir outputs/models/flan_t5_legal_sft_smoke_512
```

The default demo does not require this checkpoint.

CLI version of the same generative path:

```bash
python scripts/demo_cli.py --data-dir data --question "Kasten oldurme sucu nedir?" --answer-mode local_hf --generation-model outputs/models/flan_t5_legal_sft_smoke_512
```

Fast local health check before the demo:

```bash
python scripts/smoke_test.py --data-dir data
```

## Judge-Based Faithfulness

After QA evaluation, run the local multilingual judge:

```bash
python scripts/evaluate_llm_judge.py --input outputs/qa_eval_extractive_bm25_full.json --provider nli --output outputs/nli_judge_faithfulness_full.json
```

The reported run judged 206 of 240 answers as supported by their retrieved source, for a judge faithfulness score of 0.858.

## Final Deliverables

Generated submission files are under `deliverables/`:

- `Turkish_Legal_RAG_Final_Report.docx`
- `Turkish_Legal_RAG_Presentation.pptx`

Supporting report and presentation source notes are under `docs/`.

For exact commands and measured local outputs, see `docs/reproducibility_evidence.md`.

For before/after component comparisons, see `docs/controlled_ablation_summary.md`.

For a map from project claims to implementation files, see `docs/code_evidence_map.md`.

For instructor-provided custom documents and custom benchmarks, see `docs/submission_custom_evaluation_guide.md`.

For the latest Base RAG vs Fine-tuned RAG checks aligned with the submission note, see `docs/submission_evaluation_results.md`.

Validate a custom dataset:

```bash
python scripts/validate_custom_data.py --data-dir sample_custom_data --require-benchmark
```

Run base vs fine-tuned comparison on the same benchmark:

```bash
python scripts/run_base_vs_finetuned_eval.py --data-dir sample_custom_data --output-dir outputs/sample_submission_eval
```