evaluation_report.md

# ORCA6 Evaluation Report

Version: v0.1-bootstrap

## Current Evaluation Scope

The current verified evaluation covers retrieval, grounded-answer behavior, the
first Qwen3 14B LoRA smoke adapter, and a grounded-data fit-check adapter.

The Qwen3 adapter evaluations are intentionally limited: they use tiny training
splits and should be treated as pipeline proofs, not publishable quality results.

## Retrieval Metrics

Latest smoke-test results:

```text
queries: 20
match_modes: any=14 all=6
calibrated_pass@1: 14/20 = 70.00%
calibrated_pass@3: 20/20 = 100.00%
calibrated_pass@5: 20/20 = 100.00%
hit@1: 20/20 = 100.00%
hit@3: 20/20 = 100.00%
hit@5: 20/20 = 100.00%
all_expected@3: 19/20 = 95.00%
all_expected@5: 20/20 = 100.00%
all_expected@10: 20/20 = 100.00%
```

Re-check on 2026-06-24 matched these recorded metrics exactly.

## Model Evaluation Plan

After fine-tuning:

1. Run the retrieval smoke eval to confirm RAG context quality.
2. Run 30 held-out orchestration prompts.
3. Score each answer on correctness, relevance, actionability, safety, and novelty.
4. Compare base model, grounded RAG answer, and fine-tuned model answer.
5. Record failure modes: hallucinated tool capability, missing tradeoffs, over-complex architecture, weak safety guidance.

## Qwen3 14B Smoke Adapter Evaluation

Artifact:

```text
qwen3_14b_orca_smoke/
```

Generated outputs:

```text
evals/qwen3_14b_orca_smoke_outputs.jsonl
```

Prompt set:

```text
evals/orca_model_eval_prompts.jsonl
```

Result:

- 8 held-out ORCA prompts generated successfully.
- Adapter inference works with the local base model at `/tmp/orca6-qwen3-14b-download`.
- Outputs generally follow the expected shape: summary, architecture, tradeoffs, safety/failure modes, and next steps.
- Quality is not release-ready. The smoke adapter still makes confident unsupported claims and sometimes invents source-style citations or specific integration details.
- This confirms the training/inference path works, but the next improvement needs better grounded SFT data and stricter eval, not more epochs on the same tiny auto-graded set.

Immediate next model-data actions:

1. Replace or augment auto-graded preference data with human-checked examples.
2. Generate SFT examples that explicitly include retrieved evidence and require source-grounded answers.
   - Status: first grounded batch generated with `scripts/build_grounded_sft.py`.
   - Standalone artifact: `data/grounded_sft_eval_prompts.jsonl` (8 rows).
   - Merged train artifact: `data/training_sft_plus_grounded.jsonl` (35 rows).
3. Add hard-negative eval prompts for unsupported tool claims.
4. Re-run `scripts/evaluate_adapter_outputs.py` after the next adapter.

## Qwen3 14B Grounded Smoke Adapter Evaluation

Artifact:

```text
qwen3_14b_orca_grounded_smoke/
```

Generated outputs:

```text
evals/qwen3_14b_orca_grounded_smoke_outputs.jsonl
```

Training data:

```text
data/training_sft_plus_grounded.jsonl
```

Fit-check command used:

```bash
python scripts/finetune_qwen_unsloth.py \
  --model_name /tmp/orca6-qwen3-14b-download \
  --data_path data/training_sft_plus_grounded.jsonl \
  --validation_data_path data/validation_sft.jsonl \
  --output_dir qwen3_14b_orca_grounded_smoke \
  --epochs 2 \
  --batch_size 1 \
  --seq_length 512 \
  --eval_steps 5 \
  --save_strategy no \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_dropout 0 \
  --gradient_accumulation_steps 2
```

Result:

- Training completed on the RTX 3090 after reducing the fit-check profile to 512 tokens and LoRA rank 16.
- Adapter size: about 257 MB.
- Final metrics: `eval_loss=1.211`, `train_loss=1.592`, 36 total steps, 35 train / 3 validation examples.
- Earlier attempts at 2048/r64 and 1024/r32 hit CUDA OOM with the longer grounded examples.
- `SFTConfig.max_length` is now set from `--seq_length`; without this, TRL tokenized to its default 1024-token length even when the model was loaded at 512.
- Free-form held-out outputs still invent source IDs and links because the eval prompts do not include retrieved evidence packets.

Conclusion:

The grounded-data adapter is a successful hardware/training fit check and has
better validation loss than the first Qwen3 smoke, but it does not solve
groundedness in free-form evaluation. The next quality step is to evaluate with
explicit source-packet prompts and add hard negatives that reward refusing
unsupported citations.

## Grounded Eval Harness

Prompt set:

```text
evals/orca_grounded_eval_prompts.jsonl
```

Outputs:

```text
evals/qwen3_14b_orca_grounded_eval_outputs.jsonl
```

Report:

```text
evals/qwen3_14b_orca_grounded_eval_report.json
```

Result:

- Overall deterministic pass rate: 11/12 = 91.7%.
- Source-packet citation discipline: 8/8 passed.
- Hard negatives: 3/4 passed.
- Remaining failure: when the evidence packet was empty and the request asked for autonomous payment approval / irreversible bank transfers, the model invented LangGraph/CrewAI citations instead of refusing.

Conclusion:

The model can follow citation boundaries when source packets are present. The
next training data gap is refusal behavior for empty-evidence and high-risk
automation prompts.

## Qwen3 14B Refusal Smoke Adapter Evaluation

Artifact:

```text
qwen3_14b_orca_refusal_smoke/
```

Training data:

```text
data/training_sft_plus_grounded_refusals.jsonl
```

Added refusal data:

```text
data/refusal_sft_hard_negatives.jsonl
```

Result:

- Training completed on the RTX 3090 with the same fit-check profile: 512 tokens, LoRA rank 16.
- Adapter size: about 257 MB.
- Final metrics: `eval_loss=1.206`, `train_loss=1.510`, 42 total steps, 41 train / 3 validation examples.
- Unguarded grounded eval remained 11/12; the empty-evidence payment automation prompt still invented a citation.
- After adding a stricter source-packet runtime guard in `scripts/evaluate_adapter_outputs.py`, grounded eval passed 12/12.

Guarded report:

```text
evals/qwen3_14b_orca_refusal_guarded_eval_report.json
```

Guarded result:

- Overall deterministic pass rate: 12/12 = 100%.
- Source-packet citation discipline: 8/8.
- Hard negatives: 4/4.

Conclusion:

The current best local recipe is the refusal-tuned adapter plus a source-packet
runtime guard. Training alone improved refusal behavior but did not fully close
the empty-evidence/high-risk payment failure.

## Gate

Do not publish a trained checkpoint until:

- aggregate eval score is at least 3.2 / 4.0
- no critical hallucination appears in the held-out eval set
- model card documents training data and limitations

## GGUF Smoke Validation

Local artifact:

```text
release/gguf/orca6-qwen3-14b-refusal-q8_0.gguf
```

Validation on 2026-06-24 used the llama.cpp `b9784` Ubuntu x64 CPU binary. The
GGUF loaded successfully and generated a conservative answer for an
empty-evidence, high-risk payment automation prompt. The response recommended
human review/manual approval rather than autonomous payment execution.

Recorded throughput:

```text
Prompt: 36.1 t/s
Generation: 3.6 t/s
```

## Release-Candidate Evaluation Expansion

The v0.1-rc1 release gate now has a reproducible 50-prompt evaluation seed built
from held-out model prompts, DPO seed prompts, and retrieval smoke queries:

```bash
.venv/bin/python scripts/build_release_eval_set.py \
  --output evals/orca_release_eval_prompts.jsonl \
  --limit 50
```

Grounded source-packet prompts can be built from that seed with:

```bash
.venv/bin/python scripts/build_grounded_eval_prompts.py \
  --prompts evals/orca_release_eval_prompts.jsonl \
  --output evals/orca_release_grounded_eval_prompts.jsonl \
  --include-hard-negatives
```

Release-candidate grounded eval artifacts:

```text
evals/orca_release_eval_prompts.jsonl
evals/orca_release_grounded_eval_prompts.jsonl
evals/qwen3_14b_orca_refusal_release_grounded_outputs.jsonl
evals/qwen3_14b_orca_refusal_release_grounded_report.json
```

Result on 2026-06-25:

- Expanded grounded prompt set: 54 prompts total.
- Source-packet prompts: 50/50 passed.
- Hard negatives: 4/4 passed.
- Overall deterministic pass rate: 54/54 = 100%.

Conclusion:

The v0.1-rc1 release candidate now has a broader deterministic groundedness
gate. The clean result depends on the refusal-tuned adapter plus the runtime
source-packet guard; the guard prevents source IDs from being cited when they
are absent from retrieved evidence.