evaluation_report.md

ORCA6 Evaluation Report

Version: v0.1-bootstrap

Current Evaluation Scope

The current verified evaluation covers retrieval, grounded-answer behavior, the first Qwen3 14B LoRA smoke adapter, and a grounded-data fit-check adapter.

The Qwen3 adapter evaluations are intentionally limited: they use tiny training splits and should be treated as pipeline proofs, not publishable quality results.

Retrieval Metrics

Latest smoke-test results:

queries: 20
match_modes: any=14 all=6
calibrated_pass@1: 14/20 = 70.00%
calibrated_pass@3: 20/20 = 100.00%
calibrated_pass@5: 20/20 = 100.00%
hit@1: 20/20 = 100.00%
hit@3: 20/20 = 100.00%
hit@5: 20/20 = 100.00%
all_expected@3: 19/20 = 95.00%
all_expected@5: 20/20 = 100.00%
all_expected@10: 20/20 = 100.00%

Re-check on 2026-06-24 matched these recorded metrics exactly.

Model Evaluation Plan

After fine-tuning:

1. Run the retrieval smoke eval to confirm RAG context quality.
2. Run 30 held-out orchestration prompts.
3. Score each answer on correctness, relevance, actionability, safety, and novelty.
4. Compare base model, grounded RAG answer, and fine-tuned model answer.
5. Record failure modes: hallucinated tool capability, missing tradeoffs, over-complex architecture, weak safety guidance.

Qwen3 14B Smoke Adapter Evaluation

Artifact:

qwen3_14b_orca_smoke/

Generated outputs:

evals/qwen3_14b_orca_smoke_outputs.jsonl

Prompt set:

evals/orca_model_eval_prompts.jsonl

Result:

• 8 held-out ORCA prompts generated successfully.
• Adapter inference works with the local base model at /tmp/orca6-qwen3-14b-download.
• Outputs generally follow the expected shape: summary, architecture, tradeoffs, safety/failure modes, and next steps.
• Quality is not release-ready. The smoke adapter still makes confident unsupported claims and sometimes invents source-style citations or specific integration details.
• This confirms the training/inference path works, but the next improvement needs better grounded SFT data and stricter eval, not more epochs on the same tiny auto-graded set.

Immediate next model-data actions:

1. Replace or augment auto-graded preference data with human-checked examples.
2. Generate SFT examples that explicitly include retrieved evidence and require source-grounded answers.
   • Status: first grounded batch generated with scripts/build_grounded_sft.py.
   • Standalone artifact: data/grounded_sft_eval_prompts.jsonl (8 rows).
   • Merged train artifact: data/training_sft_plus_grounded.jsonl (35 rows).
3. Add hard-negative eval prompts for unsupported tool claims.
4. Re-run scripts/evaluate_adapter_outputs.py after the next adapter.

Qwen3 14B Grounded Smoke Adapter Evaluation

Artifact:

qwen3_14b_orca_grounded_smoke/

Generated outputs:

evals/qwen3_14b_orca_grounded_smoke_outputs.jsonl

Training data:

data/training_sft_plus_grounded.jsonl

Fit-check command used:

python scripts/finetune_qwen_unsloth.py \
  --model_name /tmp/orca6-qwen3-14b-download \
  --data_path data/training_sft_plus_grounded.jsonl \
  --validation_data_path data/validation_sft.jsonl \
  --output_dir qwen3_14b_orca_grounded_smoke \
  --epochs 2 \
  --batch_size 1 \
  --seq_length 512 \
  --eval_steps 5 \
  --save_strategy no \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_dropout 0 \
  --gradient_accumulation_steps 2

Result:

• Training completed on the RTX 3090 after reducing the fit-check profile to 512 tokens and LoRA rank 16.
• Adapter size: about 257 MB.
• Final metrics: eval_loss=1.211, train_loss=1.592, 36 total steps, 35 train / 3 validation examples.
• Earlier attempts at 2048/r64 and 1024/r32 hit CUDA OOM with the longer grounded examples.
• SFTConfig.max_length is now set from --seq_length; without this, TRL tokenized to its default 1024-token length even when the model was loaded at 512.
• Free-form held-out outputs still invent source IDs and links because the eval prompts do not include retrieved evidence packets.

Conclusion:

The grounded-data adapter is a successful hardware/training fit check and has better validation loss than the first Qwen3 smoke, but it does not solve groundedness in free-form evaluation. The next quality step is to evaluate with explicit source-packet prompts and add hard negatives that reward refusing unsupported citations.

Grounded Eval Harness

Prompt set:

evals/orca_grounded_eval_prompts.jsonl

Outputs:

evals/qwen3_14b_orca_grounded_eval_outputs.jsonl

Report:

evals/qwen3_14b_orca_grounded_eval_report.json

Result:

• Overall deterministic pass rate: 11/12 = 91.7%.
• Source-packet citation discipline: 8/8 passed.
• Hard negatives: 3/4 passed.
• Remaining failure: when the evidence packet was empty and the request asked for autonomous payment approval / irreversible bank transfers, the model invented LangGraph/CrewAI citations instead of refusing.

Conclusion:

The model can follow citation boundaries when source packets are present. The next training data gap is refusal behavior for empty-evidence and high-risk automation prompts.

Qwen3 14B Refusal Smoke Adapter Evaluation

Artifact:

qwen3_14b_orca_refusal_smoke/

Training data:

data/training_sft_plus_grounded_refusals.jsonl

Added refusal data:

data/refusal_sft_hard_negatives.jsonl

Result:

• Training completed on the RTX 3090 with the same fit-check profile: 512 tokens, LoRA rank 16.
• Adapter size: about 257 MB.
• Final metrics: eval_loss=1.206, train_loss=1.510, 42 total steps, 41 train / 3 validation examples.
• Unguarded grounded eval remained 11/12; the empty-evidence payment automation prompt still invented a citation.
• After adding a stricter source-packet runtime guard in scripts/evaluate_adapter_outputs.py, grounded eval passed 12/12.

Guarded report:

evals/qwen3_14b_orca_refusal_guarded_eval_report.json

Guarded result:

• Overall deterministic pass rate: 12/12 = 100%.
• Source-packet citation discipline: 8/8.
• Hard negatives: 4/4.

Conclusion:

The current best local recipe is the refusal-tuned adapter plus a source-packet runtime guard. Training alone improved refusal behavior but did not fully close the empty-evidence/high-risk payment failure.

Gate

Do not publish a trained checkpoint until:

• aggregate eval score is at least 3.2 / 4.0
• no critical hallucination appears in the held-out eval set
• model card documents training data and limitations

GGUF Smoke Validation

Local artifact:

release/gguf/orca6-qwen3-14b-refusal-q8_0.gguf

Validation on 2026-06-24 used the llama.cpp b9784 Ubuntu x64 CPU binary. The GGUF loaded successfully and generated a conservative answer for an empty-evidence, high-risk payment automation prompt. The response recommended human review/manual approval rather than autonomous payment execution.

Recorded throughput:

Prompt: 36.1 t/s
Generation: 3.6 t/s

Release-Candidate Evaluation Expansion

The v0.1-rc1 release gate now has a reproducible 50-prompt evaluation seed built from held-out model prompts, DPO seed prompts, and retrieval smoke queries:

.venv/bin/python scripts/build_release_eval_set.py \
  --output evals/orca_release_eval_prompts.jsonl \
  --limit 50

Grounded source-packet prompts can be built from that seed with:

.venv/bin/python scripts/build_grounded_eval_prompts.py \
  --prompts evals/orca_release_eval_prompts.jsonl \
  --output evals/orca_release_grounded_eval_prompts.jsonl \
  --include-hard-negatives

Release-candidate grounded eval artifacts:

evals/orca_release_eval_prompts.jsonl
evals/orca_release_grounded_eval_prompts.jsonl
evals/qwen3_14b_orca_refusal_release_grounded_outputs.jsonl
evals/qwen3_14b_orca_refusal_release_grounded_report.json

Result on 2026-06-25:

• Expanded grounded prompt set: 54 prompts total.
• Source-packet prompts: 50/50 passed.
• Hard negatives: 4/4 passed.
• Overall deterministic pass rate: 54/54 = 100%.

Conclusion:

The v0.1-rc1 release candidate now has a broader deterministic groundedness gate. The clean result depends on the refusal-tuned adapter plus the runtime source-packet guard; the guard prevents source IDs from being cited when they are absent from retrieved evidence.