⸻ VERANTYX Evaluation & Benchmark Guide How to Evaluate Reasoning — and How to Record Failure ⸻ 0. Why Evaluation Is Different in VERANTYX VERANTYX is not optimized for accuracy metrics. It is optimized for auditable reasoning behavior. Therefore, traditional evaluation questions like: • “How many did it get right?” • “What is the accuracy@k?” are insufficient and sometimes misleading. In VERANTYX, failure is not a bug — unexplained success is. ⸻ 1. What VERANTYX Is Actually Evaluated On VERANTYX evaluation focuses on process integrity, not output fluency. Core evaluation axes: 1. Evidence grounding 2. Correct refusal 3. Noise resistance 4. Assumption sensitivity 5. Reproducibility 6. Audit clarity ⸻ 2. Primary Outcome Categories (Do Not Collapse These) Every evaluation result must fall into one of four explicit buckets: 1. Proven • Supported by DB rules • Assumptions satisfied • No refutation hits 2. Refuted • One or more counterexample rules fired • Or necessity checks failed 3. Insufficient Evidence • No applicable proof or refutation • Phase 1 (and optionally Phase 2) found nothing reliable 4. Provisional (Mined) • Depends on Phase 2 rules • Explicitly labeled as non-final ⚠️ Never merge these into “correct / incorrect”. ⸻ 3. Benchmark Design Principles 3.1 Prefer “Trap-Rich” Problems Good VERANTYX benchmarks include: • Missing assumptions • Overgeneralized language (“always”, “all”) • Vacuity cases • Necessary vs sufficient confusion Bad benchmarks: • Straightforward textbook exercises • Problems solvable by surface pattern matching ⸻ 3.2 Single-Answer Accuracy Is Not Enough Each benchmark item should be evaluated on: • Was the selected outcome category correct? • Did the system refuse when it should? • Did it avoid hallucinated certainty? A correct refusal scores higher than an unjustified answer. ⸻ 4. Measuring Success (VERANTYX-style Metrics) 4.1 Core Metrics Instead of accuracy, track: • Justified Resolution Rate • % of answers that are Proven or Refuted with evidence • Correct Refusal Rate • % of Insufficient Evidence when the DB truly lacks coverage • False Confidence Rate (Critical) • % of answers given without valid proof/refutation Goal: drive this to near zero. ⸻ 4.2 Phase Sensitivity Metrics Track separately: • Phase 1 success rate • Phase 2 activation rate • Phase 2 dependency rate A growing DB should: • Increase Phase 1 coverage • Decrease Phase 2 usage ⸻ 5. Failure Is a First-Class Artifact 5.1 What Counts as a “Good Failure” A good failure: • Is labeled “insufficient evidence” • Explains why no rule applied • Produces zero noise A bad failure: • Selects a choice with weak lexical match • Uses mined rules silently • Looks confident but is unjustified ⸻ 5.2 Mandatory Failure Logging For every failure, record: • Input problem • Extracted assumptions • Which rules were skipped (and why) • Which rules almost matched • Phase reached (1 or 2) Failures are data, not errors. ⸻ 6. Regression Testing with Benchmarks Every DB change should be evaluated against: • A fixed benchmark suite • A failure expectation list Regression is not only: • “Did it break a correct answer?” but also: • “Did it stop refusing where it should?” ⸻ 7. Comparing VERANTYX to LLMs (Correctly) Do not compare: • Raw accuracy • Speed • Fluency Do compare: • Hallucination rate • Refusal correctness • Sensitivity to missing assumptions • Stability across re-runs VERANTYX should lose on fluency — and win on discipline. ⸻ 8. Human-in-the-Loop Evaluation Recommended practice: • Periodic human audit of: • Refusal cases • Provisional (mined) answers • High-evidence proofs Humans should ask: “Would I trust this answer in a paper?” If not, the system should not either. ⸻ 9. Benchmark Suite Composition (Recommended) A balanced VERANTYX benchmark includes: • 30–40% unsolvable (DB-limited) problems • 30% trap problems • 20% clean axiomatic problems • 10–20% exploratory / edge cases If everything is solvable, the benchmark is broken. ⸻ 10. Evaluation Anti-Patterns (Avoid These) ❌ Scoring by majority vote ❌ Treating refusal as failure ❌ Hiding Phase 2 usage ❌ Optimizing prompts to “pass benchmarks” Benchmarks exist to expose limits, not hide them. ⸻ Closing Note (English) VERANTYX is successful when: • It says “I don’t know” often • And explains why A benchmark that makes VERANTYX look perfect is a benchmark that teaches nothing. ⸻ ⸻ VERANTYX 評価・ベンチマークガイド どう評価し、どう失敗を記録するか ⸻ 0. VERANTYX における評価の前提 VERANTYX は 正答率最適化システムではありません。 評価対象は 推論の振る舞いです。 そのため： • 正答率 • Top-k accuracy だけで測るのは危険です。 VERANTYX では 説明できない成功こそが失敗です。 ⸻ 1. 評価対象は「結果」ではなく「過程」 VERANTYX の評価軸： 1. 根拠の明示性 2. 正しい拒否 3. ノイズ耐性 4. 仮定への感度 5. 再現性 6. 監査可能性 ⸻ 2. 出力は必ず 4 分類で扱う 評価時、すべての結果は次のいずれか： 1. 証明済み 2. 反証済み 3. 証拠不足 4. 仮回答（採掘） ⚠️ 正解／不正解に潰してはいけません。 ⸻ 3. ベンチマーク設計指針 3.1 罠のある問題を優先 良い問題： • 仮定抜け • 「常に」「すべて」 • 空虚真理 • 必要十分の混同 悪い問題： • 公式暗記問題 • 表層一致で解けるもの ⸻ 3.2 拒否は成功である • 正しい拒否 ＞ 根拠のない回答 ⸻ 4. VERANTYX 流メトリクス 4.1 基本指標 • 根拠付き決定率 • 正当拒否率 • 誤った確信率（最重要） 目標：誤った確信を限りなくゼロに。 ⸻ 4.2 Phase 別指標 • Phase 1 成功率 • Phase 2 使用率 • Phase 2 依存率 DB が育つほど Phase 2 は減るべき。 ⸻ 5. 失敗は「成果物」 5.1 良い失敗 • 証拠不足と明示 • なぜ解けないかが分かる • ノイズゼロ 5.2 悪い失敗 • それっぽい選択 • 仮知識を黙って使う • 自信満々 ⸻ 5.3 失敗ログに必須な情報 • 問題文 • 抽出仮定 • 弾かれたルールと理由 • Phase 到達点 ⸻ 6. 回帰テストの考え方 確認すべきは： • 正解が壊れたか • 拒否すべきところで答えていないか ⸻ 7. LLM との比較方法 比較すべき： • 幻覚率 • 拒否の正しさ • 仮定感度 • 再現性 比較すべきでない： • 流暢さ • 速度 ⸻ 8. 人間による監査 定期的に： • 拒否ケース • 仮回答 • 高証拠回答 を人が確認する。 「論文に書けるか？」が基準。 ⸻ 9. 推奨ベンチマーク構成 • 解けない問題：30–40% • 罠問題：30% • 純粋公理：20% • 境界ケース：10–20% 全部解けたら危険。 ⸻ 10. やってはいけない評価 ❌ 多数決 ❌ 拒否＝失敗 ❌ Phase 2 を隠す ❌ ベンチ対策プロンプト ⸻ 終わりに（日本語） VERANTYX が成功している状態とは、 • よく「分からない」と言い • なぜ分からないかが分かる 状態です。 完璧に見えるベンチマークは 何も教えてくれません。