# ORCA6 Evaluation Report Version: v0.1-bootstrap ## Current Evaluation Scope The current verified evaluation covers retrieval, grounded-answer behavior, the first Qwen3 14B LoRA smoke adapter, and a grounded-data fit-check adapter. The Qwen3 adapter evaluations are intentionally limited: they use tiny training splits and should be treated as pipeline proofs, not publishable quality results. ## Retrieval Metrics Latest smoke-test results: ```text queries: 20 match_modes: any=14 all=6 calibrated_pass@1: 14/20 = 70.00% calibrated_pass@3: 20/20 = 100.00% calibrated_pass@5: 20/20 = 100.00% hit@1: 20/20 = 100.00% hit@3: 20/20 = 100.00% hit@5: 20/20 = 100.00% all_expected@3: 19/20 = 95.00% all_expected@5: 20/20 = 100.00% all_expected@10: 20/20 = 100.00% ``` Re-check on 2026-06-24 matched these recorded metrics exactly. ## Model Evaluation Plan After fine-tuning: 1. Run the retrieval smoke eval to confirm RAG context quality. 2. Run 30 held-out orchestration prompts. 3. Score each answer on correctness, relevance, actionability, safety, and novelty. 4. Compare base model, grounded RAG answer, and fine-tuned model answer. 5. Record failure modes: hallucinated tool capability, missing tradeoffs, over-complex architecture, weak safety guidance. ## Qwen3 14B Smoke Adapter Evaluation Artifact: ```text qwen3_14b_orca_smoke/ ``` Generated outputs: ```text evals/qwen3_14b_orca_smoke_outputs.jsonl ``` Prompt set: ```text evals/orca_model_eval_prompts.jsonl ``` Result: - 8 held-out ORCA prompts generated successfully. - Adapter inference works with the local base model at `/tmp/orca6-qwen3-14b-download`. - Outputs generally follow the expected shape: summary, architecture, tradeoffs, safety/failure modes, and next steps. - Quality is not release-ready. The smoke adapter still makes confident unsupported claims and sometimes invents source-style citations or specific integration details. - This confirms the training/inference path works, but the next improvement needs better grounded SFT data and stricter eval, not more epochs on the same tiny auto-graded set. Immediate next model-data actions: 1. Replace or augment auto-graded preference data with human-checked examples. 2. Generate SFT examples that explicitly include retrieved evidence and require source-grounded answers. - Status: first grounded batch generated with `scripts/build_grounded_sft.py`. - Standalone artifact: `data/grounded_sft_eval_prompts.jsonl` (8 rows). - Merged train artifact: `data/training_sft_plus_grounded.jsonl` (35 rows). 3. Add hard-negative eval prompts for unsupported tool claims. 4. Re-run `scripts/evaluate_adapter_outputs.py` after the next adapter. ## Qwen3 14B Grounded Smoke Adapter Evaluation Artifact: ```text qwen3_14b_orca_grounded_smoke/ ``` Generated outputs: ```text evals/qwen3_14b_orca_grounded_smoke_outputs.jsonl ``` Training data: ```text data/training_sft_plus_grounded.jsonl ``` Fit-check command used: ```bash python scripts/finetune_qwen_unsloth.py \ --model_name /tmp/orca6-qwen3-14b-download \ --data_path data/training_sft_plus_grounded.jsonl \ --validation_data_path data/validation_sft.jsonl \ --output_dir qwen3_14b_orca_grounded_smoke \ --epochs 2 \ --batch_size 1 \ --seq_length 512 \ --eval_steps 5 \ --save_strategy no \ --lora_r 16 \ --lora_alpha 32 \ --lora_dropout 0 \ --gradient_accumulation_steps 2 ``` Result: - Training completed on the RTX 3090 after reducing the fit-check profile to 512 tokens and LoRA rank 16. - Adapter size: about 257 MB. - Final metrics: `eval_loss=1.211`, `train_loss=1.592`, 36 total steps, 35 train / 3 validation examples. - Earlier attempts at 2048/r64 and 1024/r32 hit CUDA OOM with the longer grounded examples. - `SFTConfig.max_length` is now set from `--seq_length`; without this, TRL tokenized to its default 1024-token length even when the model was loaded at 512. - Free-form held-out outputs still invent source IDs and links because the eval prompts do not include retrieved evidence packets. Conclusion: The grounded-data adapter is a successful hardware/training fit check and has better validation loss than the first Qwen3 smoke, but it does not solve groundedness in free-form evaluation. The next quality step is to evaluate with explicit source-packet prompts and add hard negatives that reward refusing unsupported citations. ## Grounded Eval Harness Prompt set: ```text evals/orca_grounded_eval_prompts.jsonl ``` Outputs: ```text evals/qwen3_14b_orca_grounded_eval_outputs.jsonl ``` Report: ```text evals/qwen3_14b_orca_grounded_eval_report.json ``` Result: - Overall deterministic pass rate: 11/12 = 91.7%. - Source-packet citation discipline: 8/8 passed. - Hard negatives: 3/4 passed. - Remaining failure: when the evidence packet was empty and the request asked for autonomous payment approval / irreversible bank transfers, the model invented LangGraph/CrewAI citations instead of refusing. Conclusion: The model can follow citation boundaries when source packets are present. The next training data gap is refusal behavior for empty-evidence and high-risk automation prompts. ## Qwen3 14B Refusal Smoke Adapter Evaluation Artifact: ```text qwen3_14b_orca_refusal_smoke/ ``` Training data: ```text data/training_sft_plus_grounded_refusals.jsonl ``` Added refusal data: ```text data/refusal_sft_hard_negatives.jsonl ``` Result: - Training completed on the RTX 3090 with the same fit-check profile: 512 tokens, LoRA rank 16. - Adapter size: about 257 MB. - Final metrics: `eval_loss=1.206`, `train_loss=1.510`, 42 total steps, 41 train / 3 validation examples. - Unguarded grounded eval remained 11/12; the empty-evidence payment automation prompt still invented a citation. - After adding a stricter source-packet runtime guard in `scripts/evaluate_adapter_outputs.py`, grounded eval passed 12/12. Guarded report: ```text evals/qwen3_14b_orca_refusal_guarded_eval_report.json ``` Guarded result: - Overall deterministic pass rate: 12/12 = 100%. - Source-packet citation discipline: 8/8. - Hard negatives: 4/4. Conclusion: The current best local recipe is the refusal-tuned adapter plus a source-packet runtime guard. Training alone improved refusal behavior but did not fully close the empty-evidence/high-risk payment failure. ## Gate Do not publish a trained checkpoint until: - aggregate eval score is at least 3.2 / 4.0 - no critical hallucination appears in the held-out eval set - model card documents training data and limitations ## GGUF Smoke Validation Local artifact: ```text release/gguf/orca6-qwen3-14b-refusal-q8_0.gguf ``` Validation on 2026-06-24 used the llama.cpp `b9784` Ubuntu x64 CPU binary. The GGUF loaded successfully and generated a conservative answer for an empty-evidence, high-risk payment automation prompt. The response recommended human review/manual approval rather than autonomous payment execution. Recorded throughput: ```text Prompt: 36.1 t/s Generation: 3.6 t/s ``` ## Release-Candidate Evaluation Expansion The v0.1-rc1 release gate now has a reproducible 50-prompt evaluation seed built from held-out model prompts, DPO seed prompts, and retrieval smoke queries: ```bash .venv/bin/python scripts/build_release_eval_set.py \ --output evals/orca_release_eval_prompts.jsonl \ --limit 50 ``` Grounded source-packet prompts can be built from that seed with: ```bash .venv/bin/python scripts/build_grounded_eval_prompts.py \ --prompts evals/orca_release_eval_prompts.jsonl \ --output evals/orca_release_grounded_eval_prompts.jsonl \ --include-hard-negatives ``` Release-candidate grounded eval artifacts: ```text evals/orca_release_eval_prompts.jsonl evals/orca_release_grounded_eval_prompts.jsonl evals/qwen3_14b_orca_refusal_release_grounded_outputs.jsonl evals/qwen3_14b_orca_refusal_release_grounded_report.json ``` Result on 2026-06-25: - Expanded grounded prompt set: 54 prompts total. - Source-packet prompts: 50/50 passed. - Hard negatives: 4/4 passed. - Overall deterministic pass rate: 54/54 = 100%. Conclusion: The v0.1-rc1 release candidate now has a broader deterministic groundedness gate. The clean result depends on the refusal-tuned adapter plus the runtime source-packet guard; the guard prevents source IDs from being cited when they are absent from retrieved evidence.