erukude/rag-audit-judge

RAG Audit Judge

RAG Audit Judge is a small local evaluator for retrieval-augmented generation (RAG) answers. It is trained to inspect a question, supplied retrieval context, and candidate answer, then return exactly one JSON audit object.

Expected output shape:

{
  "groundedness_score": 0.82,
  "hallucination_risk": "low",
  "citation_quality": "medium",
  "issues": [],
  "recommended_fix": ""
}

Included Artifacts

• lora-1b/: PEFT LoRA adapter trained from unsloth/Llama-3.2-1B-Instruct-bnb-4bit
• lora-3b/: PEFT LoRA adapter trained from unsloth/Llama-3.2-3B-Instruct-bnb-4bit
• train_data.json: 108 supervised examples for RAG answer auditing
• fine_tune.ipynb: Colab/CUDA notebook used to train and export adapters
• Modelfile: Ollama Modelfile for the 3B GGUF LoRA adapter

The referenced rag-audit-judge-1b.gguf and rag-audit-judge-3b.gguf files are not included in this local repo yet. Generate them from the LoRA adapter folders before using the Ollama Modelfile.

Intended Use

Provide the model with:

• a user question
• retrieved context entries with stable source IDs
• the candidate answer to audit

The judge should evaluate only the supplied context, not outside knowledge. It is intended for development-time or local automation checks around RAG answer quality, citation coverage, unsupported claims, overconfidence, and sensitive information exposure.

Example input:

Question: Can I return a headset?

Context:
- id: refund-policy
  text: Headsets can be returned within 30 days if they are unopened.

Candidate answer:
An unopened headset can be returned within 30 days, and a $5 restocking fee always applies [refund-policy].

Example output:

{
  "groundedness_score": 0.55,
  "hallucination_risk": "medium",
  "citation_quality": "medium",
  "issues": [
    "The $5 restocking fee is not supported by the supplied context."
  ],
  "recommended_fix": "Remove the restocking-fee claim or provide context that supports it."
}

Training Details

The adapters were trained with Unsloth and TRL SFT on JSON-formatted examples. The dataset contains 108 examples with target audits covering low, medium, and high hallucination-risk labels and citation quality labels.

LoRA configuration:

• rank: 32
• alpha: 64
• dropout: 0
• target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• sequence length: 3072
• epochs: 5
• batch size per device: 2
• gradient accumulation steps: 4
• learning rate: 2e-4
• PEFT: 0.19.1

Loading With Transformers/PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "unsloth/Llama-3.2-3B-Instruct-bnb-4bit"
adapter_path = "lora-3b"

tokenizer = AutoTokenizer.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(base_model, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_path)

Use lora-1b with the 1B base model if you need a smaller local adapter.

Ollama

After converting a LoRA adapter to GGUF, update or use Modelfile:

ollama create rag-audit-judge -f Modelfile
ollama run rag-audit-judge

For llama.cpp conversion, the notes file contains the command pattern used locally:

python llama.cpp/convert_lora_to_gguf.py \
  ./rag-audit-judge/lora-3b \
  --outfile ./rag-audit-judge/rag-audit-judge-3b.gguf \
  --outtype f16 \
  --base-model-id unsloth/Llama-3.2-3B-Instruct-bnb-4bit