[NOTICKET] feat(eval): add help-skill eval harness + goal-based language fallback

Live-model eval for the Help skill (eval/help), mirroring eval/intent + eval/readiness:
each case declares state + report_ready + history, streams HelpAgent.astream for real, and
scores RULE compliance (reply language; never/always suggest a report per report_ready) —
not text similarity. Held-out vs carried_over reported separately so prompt overfitting is
visible; orientation cases are manual_review.

help.py: reply-language detection now consults the user-authored goal (objective +
business_questions) when there is no chat turn to read, before the hard Indonesian
fallback. The goal is required at onboarding, so on a fresh analysis it is a reliable
signal — a user whose goal is in English no longer gets an Indonesian reply. Priority:
message > last human turn > goal > default.

help.md v3: language rule promoted to a hard [Reply language] directive; Examples given
stable ids so the eval can mirror them as carried_over regression cases; second example now
uses a real is_report_ready `missing` value instead of the never-emitted
"no comparison over time".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

eval/help/README.md

@@ -0,0 +1,62 @@
+# Help-skill eval
+
+Scores the **live** Help skill (`src/agents/handlers/help.HelpAgent`) — the guide that
+tells a user where they are and what to do next. Each golden case declares an analysis
+state + report-readiness + chat history; the runner streams `HelpAgent.astream` for real
+and asserts the **rules** the reply must obey.
+
+Unlike `eval/readiness` (deterministic, no LLM), this calls the model, so it needs a
+working `.env` (Azure OpenAI) and spends tokens. Run it before a deploy that touches
+`config/prompts/help.md` — not on every commit. The fast, no-LLM guard is
+`tests/unit/agents/handlers/test_help.py` (fake chain); this is the end-to-end
+"does the model actually obey the prompt" layer on top.
+
+## Run
+
+```bash
+uv run python -m eval.help.run_eval
+uv run python -m eval.help.run_eval --limit 4     # smoke test
+uv run python -m eval.help.run_eval --no-table    # summary only
+```
+
+Each run writes a timestamped `results/help_result_<ts>.json` (never overwritten,
+diffable across runs).
+
+## What it measures
+
+Not accuracy — Help replies are free prose with no single correct wording. The metric is
+**compliance**: the % of cases whose reply obeys every rule asserted for it.
+
+- **`language`** — the reply must match the user's language. This is the regression guard
+  for the button-path bug (`/tools/help` passes `message=None`, and the reply used to
+  default to English even for an Indonesian conversation).
+- **`report_guard`** — never suggest generating a report when `report_ready.ready=false`;
+  do suggest it when `true`. Since `generate_report` is the only gated action, this also
+  serves as the "no action leakage" check.
+- **`orientation`** — quality of the suggested starter questions. **Manual review**: these
+  run but are excluded from the auto compliance rate. Read their `output_text` in the JSON.
+
+Assertion types: `language_match {expected}`, `must_not_contain_any {patterns}`,
+`must_contain_any {patterns}`.
+
+## Held-out vs carried-over (why the summary splits them)
+
+`carried_over: true` cases **mirror an example in `help.md`** — the case `id` *is* the
+prompt's `<!-- id: ... -->`. They are a regression guard: if the prompt is refactored, the
+demonstrated rule must still hold. What is mirrored is the **input spec + the assertion**,
+never the example's reply text (temperature > 0 makes exact match invalid).
+
+Held-out cases (`carried_over: false`) are **absent from the prompt**; their compliance is
+the real generalization signal. If held-out compliance drops while carried-over stays at
+100%, the prompt is overfitting to its own examples ("train on test set"). That's why the
+two are reported separately.
+
+**Sync rule (manual, like `intent`):** if `help.md`'s Examples change, keep the mirrored
+`id`s here in sync. Current mirrored ids: `help_ex_orient`, `help_ex_guard_delta`,
+`help_ex_guard_ready`.
+
+## Dataset
+
+`help_dataset.json` — see the `_about` / `_carried_over` doc keys in the file. Language
+detection reuses `help._detect_reply_language`; `report_ready.missing` uses the codes
+`analysis` / `delta` mapped to the real `is_report_ready` strings in the runner.

eval/help/help_dataset.json

@@ -0,0 +1,150 @@
+{
+  "_about": "Golden dataset for the Help skill (`src/agents/handlers/help.HelpAgent`). Unlike intent/readiness this calls the LIVE model: each case declares an analysis state + report-readiness + chat history, the runner streams HelpAgent.astream for real, and asserts RULES the reply must obey (not text similarity — help replies are free prose with no single correct wording). Metric is COMPLIANCE (% of rule assertions that hold), reported separately for held-out vs carried_over cases.",
+  "_groups": "language (reply matches the user's language — the button-path bug), report_guard (never suggest a report when report_ready.ready=false; do suggest it when true — this also IS the 'no action leakage' check, since generate_report is the only gated action), orientation (quality of the suggested starter questions — MANUAL review, not auto-scored).",
+  "_asserts": "language_match {expected} — detect the reply's language (help._detect_reply_language over the OUTPUT) must equal expected. must_not_contain_any {patterns} — none of the (case-insensitive) patterns appear. must_contain_any {patterns} — at least one appears.",
+  "_carried_over": "carried_over:true rows MIRROR an example in config/prompts/help.md (the row `id` IS the help.md `<!-- id: ... -->`). They are the regression guard: if the prompt is refactored, the demonstrated rule must still hold. What is mirrored is the INPUT spec + the assertion — NOT the example's reply text (temperature>0 makes exact match invalid). Held-out rows (carried_over:false) are NOT in the prompt; their compliance is the real generalization signal. If help.md's Examples change, keep these ids in sync (manual, like intent).",
+  "_missing_codes": "report_ready.missing uses codes mapped to the real strings is_report_ready emits (imported in run_eval): analysis -> _MISSING_ANALYSIS, delta -> _MISSING_DELTA. Kept as codes so the dataset survives wording changes.",
+  "schema": {
+    "id": "stable handle; for carried_over rows this equals the help.md example id",
+    "group": "language | report_guard | orientation",
+    "carried_over": "bool — mirrors a help.md example",
+    "manual_review": "bool — run but exclude from the auto compliance rate (read output_text)",
+    "state": "{ analysis_title, objective, business_questions[], report_id }",
+    "report_ready": "{ ready: bool, missing: [analysis|delta] }",
+    "history": "[{ role: human|ai, content }] — drives language on the button path",
+    "message": "the human turn; null = button path (HelpAgent falls back to a per-language trigger)",
+    "asserts": "[{ type, ...spec }] — the rules the reply must obey",
+    "note": "human-readable description"
+  },
+  "cases": [
+    {
+      "id": "lang_01", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis penjualan", "objective": "memahami performa penjualan bulanan", "business_questions": ["produk mana yang paling laku?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [{ "role": "human", "content": "aku baru upload datanya, terus aku harus ngapain?" }],
+      "message": null,
+      "asserts": [{ "type": "language_match", "expected": "Indonesian" }],
+      "note": "REGRESSION of the button-path bug: Indonesian conversation, message=null. Reply must be Indonesian, not English."
+    },
+    {
+      "id": "lang_02", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Sales analysis", "objective": "understand monthly sales performance", "business_questions": ["which products drive revenue?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [{ "role": "human", "content": "okay I uploaded my data, what do I do next?" }],
+      "message": null,
+      "asserts": [{ "type": "language_match", "expected": "English" }],
+      "note": "English conversation, button path — reply must stay English."
+    },
+    {
+      "id": "lang_03", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis churn", "objective": "menurunkan churn pelanggan", "business_questions": ["segmen mana yang paling banyak churn?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [],
+      "message": "gimana caranya mulai analisis ini ya?",
+      "asserts": [{ "type": "language_match", "expected": "Indonesian" }],
+      "note": "Intent path: the real Indonesian user turn drives the language."
+    },
+    {
+      "id": "lang_04", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Retention analysis", "objective": "understand user retention", "business_questions": ["what drives repeat usage?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [],
+      "message": null,
+      "asserts": [{ "type": "language_match", "expected": "English" }],
+      "note": "Fresh analysis, no chat yet, button path — with no turn to read, the user-authored goal (English objective + business_questions, required at onboarding) drives the language."
+    },
+    {
+      "id": "lang_06", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis retensi", "objective": "memahami retensi pengguna", "business_questions": ["apa yang mendorong penggunaan berulang?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [],
+      "message": null,
+      "asserts": [{ "type": "language_match", "expected": "Indonesian" }],
+      "note": "Same fresh-analysis path as lang_04 but the goal is Indonesian — the goal signal must yield Indonesian (not the hard fallback, which only fires when the goal is empty too)."
+    },
+    {
+      "id": "lang_05", "group": "language", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis penjualan", "objective": "memahami tren penjualan", "business_questions": ["bagaimana tren bulanan?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [
+        { "role": "human", "content": "apa saja yang bisa aku tanyakan tentang data ini?" },
+        { "role": "ai", "content": "You can start by asking which products sell the most." }
+      ],
+      "message": null,
+      "asserts": [{ "type": "language_match", "expected": "Indonesian" }],
+      "note": "Last AI turn is English but the human turn is Indonesian — mirror the human, reply Indonesian."
+    },
+    {
+      "id": "help_ex_guard_delta", "group": "report_guard", "carried_over": true, "manual_review": false,
+      "state": { "analysis_title": "Sales analysis", "objective": "understand monthly sales performance", "business_questions": ["which products drive revenue?"], "report_id": "rep-1" },
+      "report_ready": { "ready": false, "missing": ["delta"] },
+      "history": [{ "role": "human", "content": "what should I do next?" }],
+      "message": null,
+      "asserts": [{ "type": "must_not_contain_any", "patterns": ["/report", "generate the report", "generate your report", "create the report"] }],
+      "note": "MIRRORS help.md example help_ex_guard_delta. A report exists and nothing new since — must NOT tell the user to generate a report; steer them to run a fresh analysis first."
+    },
+    {
+      "id": "help_ex_guard_ready", "group": "report_guard", "carried_over": true, "manual_review": false,
+      "state": { "analysis_title": "Sales analysis", "objective": "understand monthly sales performance", "business_questions": ["which products drive revenue?"], "report_id": null },
+      "report_ready": { "ready": true, "missing": [] },
+      "history": [{ "role": "human", "content": "what should I do next?" }],
+      "message": null,
+      "asserts": [{ "type": "must_contain_any", "patterns": ["/report", "report"] }],
+      "note": "MIRRORS help.md example help_ex_guard_ready. Enough analysis done — SHOULD nudge toward the report (mention /report or the report option)."
+    },
+    {
+      "id": "guard_03", "group": "report_guard", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Retention analysis", "objective": "improve 30-day retention", "business_questions": ["which cohort retains best?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [{ "role": "human", "content": "can I get a report now?" }],
+      "message": null,
+      "asserts": [{ "type": "must_not_contain_any", "patterns": ["/report", "generate the report", "generate your report", "you can generate"] }],
+      "note": "No analysis run yet, user asks for a report directly — must NOT offer to generate; redirect to running an analysis first."
+    },
+    {
+      "id": "guard_04", "group": "report_guard", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis penjualan", "objective": "memahami performa penjualan", "business_questions": ["produk mana yang paling laku?"], "report_id": null },
+      "report_ready": { "ready": true, "missing": [] },
+      "history": [{ "role": "human", "content": "selanjutnya aku ngapain?" }],
+      "message": null,
+      "asserts": [
+        { "type": "must_contain_any", "patterns": ["/report", "laporan", "report"] },
+        { "type": "language_match", "expected": "Indonesian" }
+      ],
+      "note": "Ready + Indonesian conversation — should nudge toward the report AND stay in Indonesian (two rules at once)."
+    },
+    {
+      "id": "guard_05", "group": "report_guard", "carried_over": false, "manual_review": false,
+      "state": { "analysis_title": "Analisis churn", "objective": "menurunkan churn", "business_questions": ["segmen mana yang paling churn?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [{ "role": "human", "content": "aku mau bikin laporan dong" }],
+      "message": null,
+      "asserts": [
+        { "type": "must_not_contain_any", "patterns": ["/report", "silakan buat laporan", "kamu bisa membuat laporan", "generate your report"] },
+        { "type": "language_match", "expected": "Indonesian" }
+      ],
+      "note": "Indonesian, not ready, user asks for a report — must NOT offer it and must reply in Indonesian."
+    },
+    {
+      "id": "help_ex_orient", "group": "orientation", "carried_over": true, "manual_review": true,
+      "state": { "analysis_title": "Sales analysis", "objective": "understand monthly sales performance", "business_questions": ["which products drive revenue?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [],
+      "message": null,
+      "asserts": [],
+      "note": "MIRRORS help.md example help_ex_orient. MANUAL: are the 2-3 starter questions concrete, descriptive-first, and tied to the objective? Read output_text."
+    },
+    {
+      "id": "orient_02", "group": "orientation", "carried_over": false, "manual_review": true,
+      "state": { "analysis_title": "Retention analysis", "objective": "improve 30-day retention", "business_questions": ["which acquisition channel retains best?"], "report_id": null },
+      "report_ready": { "ready": false, "missing": ["analysis"] },
+      "history": [
+        { "role": "human", "content": "which channel brings the most signups?" },
+        { "role": "ai", "content": "Organic search brought the most signups last month (1,240)." }
+      ],
+      "message": null,
+      "asserts": [],
+      "note": "MANUAL: one question already answered — does help build on it with a NEW follow-up (retention by channel), not re-suggest the answered question? Read output_text."
+    }
+  ]
+}

eval/help/run_eval.py

@@ -0,0 +1,404 @@
+"""Help-skill eval runner.
+
+Feeds each golden case in `help_dataset.json` to the LIVE Help skill
+(`src/agents/handlers/help.HelpAgent.astream`), then scores whether the streamed
+reply obeys a set of RULE assertions — reply language, never suggesting a report
+when `report_ready.ready=false`, suggesting it when true. Prints a per-case detail
+table + aggregate summary and writes a timestamped JSON report under `results/`
+(never overwritten — one file per run, diffable).
+
+Unlike `eval/readiness` (deterministic, no LLM), this calls the model for real, so
+it needs a working `.env` (Azure OpenAI) and spends tokens — run it before a deploy
+that touches `help.md`, not on every commit. `tests/unit/agents/handlers/test_help.py`
+already covers the deterministic Python guard with a fake chain; this is the
+end-to-end "does the model actually obey the prompt" layer on top.
+
+Two things the metric separates on purpose:
+  - COMPLIANCE = % of rule assertions that hold. NOT accuracy — help replies are free
+    prose with no single correct wording; we score rule-obedience, not similarity.
+  - HELD-OUT vs CARRIED-OVER — carried_over cases mirror a help.md example (regression);
+    held-out cases are absent from the prompt. Held-out compliance is the real
+    generalization signal. If held-out drops while carried_over stays 100%, the prompt
+    is overfitting to its own examples.
+
+`orientation` cases are `manual_review` — run but excluded from the auto compliance
+rate; read their `output_text` in the JSON report to judge suggestion quality.
+
+Invoke as a module so `src` imports resolve:
+
+    uv run python -m eval.help.run_eval
+    uv run python -m eval.help.run_eval --limit 4     # smoke test
+    uv run python -m eval.help.run_eval --no-table    # summary only
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import statistics
+import time
+from dataclasses import asdict, dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+from langchain_core.callbacks import BaseCallbackHandler
+from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
+from langchain_core.outputs import LLMResult
+
+from src.agents.gate import stub_analysis_state
+from src.agents.handlers.help import HelpAgent, ReportReadiness, _detect_reply_language
+from src.agents.report.readiness import _MISSING_ANALYSIS, _MISSING_DELTA
+
+_HERE = Path(__file__).resolve().parent
+DATASET = _HERE / "help_dataset.json"
+RESULTS_DIR = _HERE / "results"
+GROUPS = ["language", "report_guard", "orientation"]
+
+# Dataset short codes -> the exact `missing` strings is_report_ready emits. Imported
+# from the module so the dataset stays readable and survives wording changes.
+_CODE_TO_MISSING = {
+    "analysis": _MISSING_ANALYSIS,
+    "delta": _MISSING_DELTA,
+}
+
+
+class _UsageCollector(BaseCallbackHandler):
+    """Sums token usage across the LLM calls made during one astream()."""
+
+    def __init__(self) -> None:
+        self.input_tokens = 0
+        self.output_tokens = 0
+        self.total_tokens = 0
+
+    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
+        before = self.total_tokens
+        for generation_list in response.generations:
+            for generation in generation_list:
+                message = getattr(generation, "message", None)
+                usage = getattr(message, "usage_metadata", None) if message else None
+                if usage:
+                    self.input_tokens += usage.get("input_tokens", 0)
+                    self.output_tokens += usage.get("output_tokens", 0)
+                    self.total_tokens += usage.get("total_tokens", 0)
+        if self.total_tokens == before and response.llm_output:
+            usage = response.llm_output.get("token_usage") or {}
+            self.input_tokens += usage.get("prompt_tokens", 0)
+            self.output_tokens += usage.get("completion_tokens", 0)
+            self.total_tokens += usage.get("total_tokens", 0)
+
+    @property
+    def tokens(self) -> dict[str, int]:
+        return {
+            "input": self.input_tokens,
+            "output": self.output_tokens,
+            "total": self.total_tokens,
+        }
+
+
+# --- assertion checkers -----------------------------------------------------
+# Each returns (passed, detail). `detail` explains a failure in the table/report.
+
+
+def _check_language_match(output: str, spec: dict[str, Any]) -> tuple[bool, str]:
+    got = _detect_reply_language([], message=output)
+    return got == spec["expected"], f"want {spec['expected']}, got {got}"
+
+
+def _check_must_not_contain_any(output: str, spec: dict[str, Any]) -> tuple[bool, str]:
+    low = output.lower()
+    hits = [p for p in spec["patterns"] if p.lower() in low]
+    return (not hits), (f"found {hits}" if hits else "none present")
+
+
+def _check_must_contain_any(output: str, spec: dict[str, Any]) -> tuple[bool, str]:
+    low = output.lower()
+    hits = [p for p in spec["patterns"] if p.lower() in low]
+    return bool(hits), (f"found {hits}" if hits else f"none of {spec['patterns']}")
+
+
+_ASSERT_CHECKS = {
+    "language_match": _check_language_match,
+    "must_not_contain_any": _check_must_not_contain_any,
+    "must_contain_any": _check_must_contain_any,
+}
+
+
+@dataclass
+class AssertResult:
+    type: str
+    passed: bool
+    detail: str
+
+
+@dataclass
+class CaseResult:
+    id: str
+    group: str
+    carried_over: bool
+    manual_review: bool
+    output_text: str
+    asserts: list[AssertResult]
+    all_passed: bool | None  # None when manual_review (not auto-scored)
+    latency_ms: float
+    tokens: dict[str, int]
+
+
+def load_cases(path: Path) -> list[dict[str, Any]]:
+    """Read the `cases` array, skipping the leading `_*` doc keys and `schema`."""
+    data = json.loads(path.read_text(encoding="utf-8"))
+    return list(data["cases"])
+
+
+def _build_state(spec: dict[str, Any]):
+    """Build an AnalysisState from a case's `state` block (defaults from the stub)."""
+    return stub_analysis_state().model_copy(
+        update={
+            "analysis_title": spec.get("analysis_title", "New analysis"),
+            "objective": spec.get("objective", ""),
+            "business_questions": list(spec.get("business_questions", [])),
+            "report_id": spec.get("report_id"),
+        }
+    )
+
+
+def _build_history(rows: list[dict[str, Any]]) -> list[BaseMessage]:
+    out: list[BaseMessage] = []
+    for row in rows:
+        cls = HumanMessage if row["role"] == "human" else AIMessage
+        out.append(cls(content=row["content"]))
+    return out
+
+
+def _build_readiness(spec: dict[str, Any]) -> ReportReadiness:
+    return ReportReadiness(
+        ready=bool(spec["ready"]),
+        missing=[_CODE_TO_MISSING[c] for c in spec.get("missing", [])],
+    )
+
+
+async def run_case(case: dict[str, Any]) -> CaseResult:
+    """Stream one Help reply and score its assertions; never throws."""
+    state = _build_state(case["state"])
+    history = _build_history(case.get("history", []))
+    readiness = _build_readiness(case["report_ready"])
+    collector = _UsageCollector()
+
+    agent = HelpAgent()  # real Azure chain, constructed lazily on first astream
+    start = time.perf_counter()
+    try:
+        output = "".join(
+            [
+                token
+                async for token in agent.astream(
+                    state,
+                    history=history,
+                    message=case.get("message"),
+                    report_ready=readiness,
+                    callbacks=[collector],
+                )
+            ]
+        )
+    except Exception as exc:  # noqa: BLE001 — one bad case shouldn't kill the run
+        output = f"ERROR:{type(exc).__name__}: {exc}"
+    latency_ms = round((time.perf_counter() - start) * 1000, 1)
+
+    manual = bool(case.get("manual_review"))
+    asserts: list[AssertResult] = []
+    if not manual:
+        for spec in case.get("asserts", []):
+            check = _ASSERT_CHECKS[spec["type"]]
+            passed, detail = check(output, spec)
+            asserts.append(AssertResult(type=spec["type"], passed=passed, detail=detail))
+    all_passed = None if manual else all(a.passed for a in asserts)
+
+    return CaseResult(
+        id=case["id"],
+        group=case["group"],
+        carried_over=bool(case.get("carried_over")),
+        manual_review=manual,
+        output_text=output,
+        asserts=asserts,
+        all_passed=all_passed,
+        latency_ms=latency_ms,
+        tokens=collector.tokens,
+    )
+
+
+def _compliance(results: list[CaseResult]) -> dict[str, Any]:
+    scored = [r for r in results if r.all_passed is not None]
+    passed = sum(1 for r in scored if r.all_passed)
+    return {
+        "n": len(scored),
+        "passed": passed,
+        "compliance": round(passed / len(scored), 3) if scored else 0.0,
+    }
+
+
+def summarize(results: list[CaseResult]) -> dict[str, Any]:
+    scored = [r for r in results if r.all_passed is not None]
+    latencies = [r.latency_ms for r in results]
+    tok_total = sum(r.tokens["total"] for r in results)
+    overall = _compliance(results)
+    by_group = {
+        g: _compliance([r for r in results if r.group == g])
+        for g in GROUPS
+        if any(r.group == g for r in results)
+    }
+    return {
+        "total": len(results),
+        "scored": len(scored),
+        "manual_review": len(results) - len(scored),
+        "passed": overall["passed"],
+        "compliance": overall["compliance"],
+        "runtime_avg_ms": round(statistics.mean(latencies), 1) if latencies else 0,
+        "tokens_total": tok_total,
+        "by_group": by_group,
+        "held_out": _compliance([r for r in scored if not r.carried_over]),
+        "carried_over": _compliance([r for r in scored if r.carried_over]),
+    }
+
+
+def _truncate(text: str, width: int) -> str:
+    text = text.replace("\n", " ")
+    return text if len(text) <= width else text[: width - 3] + "..."
+
+
+def format_table(results: list[CaseResult]) -> str:
+    header = (
+        f"{'ID':<20} {'GROUP':<13} {'C/O':<4} {'ASSERTS':<22} {'OK':<4} {'MS':>7}"
+    )
+    rule = "-" * len(header)
+    lines = [rule, header, rule]
+    for r in results:
+        co = "CO" if r.carried_over else "-"
+        if r.manual_review:
+            atypes, ok = "(manual)", "~"
+        else:
+            atypes = ",".join(a.type.replace("_", "")[:6] for a in r.asserts) or "-"
+            ok = "ok" if r.all_passed else "X"
+        lines.append(
+            f"{r.id:<20} {r.group:<13} {co:<4} {_truncate(atypes, 22):<22} "
+            f"{ok:<4} {r.latency_ms:>7}"
+        )
+    lines.append(rule)
+    return "\n".join(lines)
+
+
+def format_summary(summary: dict[str, Any], results: list[CaseResult]) -> str:
+    lines = ["SUMMARY"]
+    lines.append(
+        f"  Compliance   {summary['passed']}/{summary['scored']} cases obey all rules"
+        f"   ({summary['compliance'] * 100:.1f}%)   avg {summary['runtime_avg_ms']} ms"
+    )
+    lines.append(
+        f"  Manual       {summary['manual_review']} case(s) excluded from the rate"
+        " (read output_text)"
+    )
+    lines.append("")
+    lines.append("  By group")
+    for g, m in summary["by_group"].items():
+        if m["n"]:
+            lines.append(f"    {g:<14} {m['passed']}/{m['n']}  {m['compliance'] * 100:.0f}%")
+        else:
+            lines.append(f"    {g:<14} (manual only)")
+    lines.append("")
+    ho, co = summary["held_out"], summary["carried_over"]
+    lines.append("  Held-out vs carried-over")
+    lines.append(
+        f"    held_out       {ho['passed']}/{ho['n']}  "
+        f"{ho['compliance'] * 100:.0f}%   <- generalization"
+    )
+    lines.append(
+        f"    carried_over   {co['passed']}/{co['n']}  "
+        f"{co['compliance'] * 100:.0f}%   <- regression"
+    )
+    failures = [r for r in results if r.all_passed is False]
+    lines.append("")
+    lines.append(f"  FAILURES ({len(failures)})")
+    for r in failures:
+        bad = [f"{a.type}({a.detail})" for a in r.asserts if not a.passed]
+        lines.append(f"    {r.id:<20} {r.group:<13} {'; '.join(bad)}")
+    return "\n".join(lines)
+
+
+def build_report(
+    results: list[CaseResult], summary: dict[str, Any], meta: dict[str, Any]
+) -> dict[str, Any]:
+    run = {
+        **meta,
+        **{
+            k: summary[k]
+            for k in ("total", "scored", "manual_review", "passed", "compliance",
+                      "runtime_avg_ms", "tokens_total")
+        },
+    }
+    return {
+        "run": run,
+        "by_group": summary["by_group"],
+        "held_out": summary["held_out"],
+        "carried_over": summary["carried_over"],
+        "cases": [asdict(r) for r in results],
+    }
+
+
+def _model_name() -> str:
+    try:
+        from src.config.settings import settings
+
+        return str(settings.azureai_deployment_name_4o)
+    except Exception:  # noqa: BLE001 — meta only; .env may be absent
+        return "gpt-4o"
+
+
+@dataclass
+class _Args:
+    dataset: Path = DATASET
+    limit: int = 0
+    no_table: bool = False
+    extra: dict[str, Any] = field(default_factory=dict)
+
+
+async def main() -> None:
+    parser = argparse.ArgumentParser(description="Help-skill eval")
+    parser.add_argument("--dataset", type=Path, default=DATASET)
+    parser.add_argument("--limit", type=int, default=0, help="run first N cases only")
+    parser.add_argument("--prompt-version", default="help.md")
+    parser.add_argument("--no-table", action="store_true", help="skip the detail table")
+    args = parser.parse_args()
+
+    cases = load_cases(args.dataset)
+    if args.limit:
+        cases = cases[: args.limit]
+
+    started = datetime.now()
+    print(f"Help Skill Eval -- {started:%Y-%m-%d %H:%M:%S}")
+    print(
+        f"dataset: {args.dataset.name} ({len(cases)} cases)  model: {_model_name()}  "
+        f"prompt: {args.prompt_version}  target: HelpAgent.astream (live)"
+    )
+
+    results = [await run_case(case) for case in cases]
+
+    summary = summarize(results)
+    if not args.no_table:
+        print(format_table(results))
+    print(format_summary(summary, results))
+
+    meta = {
+        "timestamp": started.isoformat(timespec="seconds"),
+        "dataset": args.dataset.name,
+        "model": _model_name(),
+        "prompt_version": args.prompt_version,
+        "target": "src/agents/handlers/help.HelpAgent.astream",
+    }
+    report = build_report(results, summary, meta)
+    RESULTS_DIR.mkdir(parents=True, exist_ok=True)
+    out_path = RESULTS_DIR / f"help_result_{started:%Y-%m-%d_%H%M%S}.json"
+    out_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
+    print(f"\n-> saved: {out_path.relative_to(_HERE.parent.parent)}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

src/agents/handlers/help.py

@@ -86,26 +86,43 @@ def _last_human_text(history: list[BaseMessage] | None) -> str:
     return ""
 
 
-def _detect_reply_language(
-    history: list[BaseMessage] | None, message: str | None = None
-) -> str:
-    """Detect the reply language from the last human turn (deterministic, no LLM).
-
-    Prefers an explicit `message` (intent path — the user's real turn) over the last
-    human turn in `history` (button path, where `message` is None). Counts Indonesian vs
-    English marker words; ties or no signal fall back to Indonesian (the team default).
-    Returns "Indonesian" or "English".
-    """
-    text = (message or _last_human_text(history)).lower()
-    if not text.strip():
-        return _FALLBACK_LANGUAGE
-    tokens = re.findall(r"[a-z']+", text)
+def _score_language(text: str) -> str | None:
+    """Return "Indonesian"/"English" from marker-word counts, or None if no signal."""
+    tokens = re.findall(r"[a-z']+", text.lower())
     id_hits = sum(1 for t in tokens if t in _ID_MARKERS)
     en_hits = sum(1 for t in tokens if t in _EN_MARKERS)
     if en_hits > id_hits:
         return "English"
     if id_hits > en_hits:
         return "Indonesian"
+    return None
+
+
+def _detect_reply_language(
+    history: list[BaseMessage] | None,
+    message: str | None = None,
+    goal_texts: list[str] | None = None,
+) -> str:
+    """Detect the reply language deterministically (no LLM), by signal priority.
+
+    1. the user's turn — an explicit `message` (intent path) or the last human turn in
+       `history` (button path, where `message` is None);
+    2. the user-authored goal (`objective` + `business_questions`) — required at
+       onboarding, so it's always present and is a reliable signal on a fresh analysis
+       that has no chat yet;
+    3. the team default (Indonesian) — a safety net only, for a stub/legacy/empty-goal
+       state where nothing above yields a signal.
+
+    Returns "Indonesian" or "English".
+    """
+    primary = (message or _last_human_text(history)).strip()
+    lang = _score_language(primary) if primary else None
+    if lang:
+        return lang
+    goal = " ".join(t for t in (goal_texts or []) if t).strip()
+    lang = _score_language(goal) if goal else None
+    if lang:
+        return lang
     return _FALLBACK_LANGUAGE
 
 
@@ -241,7 +258,11 @@ class HelpAgent:
         """
         readiness = report_ready or ReportReadiness()
         actions = available_actions or _derive_available_actions(state, readiness)
-        reply_language = _detect_reply_language(history, message)
+        goal_texts = [
+            getattr(state, "objective", "") or "",
+            *(getattr(state, "business_questions", None) or []),
+        ]
+        reply_language = _detect_reply_language(history, message, goal_texts=goal_texts)
         logger.info(
             "help guidance",
             report_ready=readiness.ready,
@@ -250,7 +271,9 @@ class HelpAgent:
         )
 
         chain = self._ensure_chain()
-        default_trigger = _DEFAULT_TRIGGERS.get(reply_language, _DEFAULT_TRIGGERS[_FALLBACK_LANGUAGE])
+        default_trigger = _DEFAULT_TRIGGERS.get(
+            reply_language, _DEFAULT_TRIGGERS[_FALLBACK_LANGUAGE]
+        )
         payload: dict[str, Any] = {
             "message": message or default_trigger,
             "history": history or [],

src/config/prompts/help.md

@@ -1,8 +1,14 @@
-<!-- help.md · v2 · Help skill prompt. v2 (2026-06-24, KM-652): removed the problem_statement
-     skill + the problem_validated gate — the goal (objective + business_questions) is now set
-     in the New Analysis form at onboarding, so Help no longer steers users to define/validate a
-     goal in chat. Bump to v3 (don't silently overwrite) on the next major change (e.g. real UI
-     steps from the frontend). -->
+<!-- help.md · v3 · Help skill prompt.
+     v2 (2026-06-24, KM-652): removed the problem_statement skill + the problem_validated gate —
+     the goal (objective + business_questions) is now set in the New Analysis form at onboarding,
+     so Help no longer steers users to define/validate a goal in chat.
+     v3 (2026-07-02): (a) reply language is now a hard rule driven by the [Reply language]
+     directive (the button path was defaulting to English); (b) Examples got stable ids
+     ("id: ..." comment above each) so eval/help can mirror them as carried_over regression
+     cases, and the second example now uses a REAL `missing` value from report/readiness.py —
+     the old "no comparison over time" string is never emitted by is_report_ready.
+     Bump to v4 (don't silently overwrite) on the next major change (e.g. real UI steps from
+     the frontend). -->
 
 You are the **Help guide** for an AI data-analysis assistant. Think of yourself as the
 instruction sheet that comes with a board game: your only job is to tell the user
@@ -92,15 +98,21 @@ spam, no overselling. A few sentences is usually enough.
 ## Examples
 
 ```
-State: chat_history nearly empty
+<!-- id: help_ex_orient -->
+State: objective="understand monthly sales performance",
+       business_questions=["which products drive revenue?"],
+       chat_history empty, report_ready.ready=false, missing=["at least one completed analysis"]
 → "Your goal is set — you can start exploring now. Try a basic question first, like
    'Which products sell the most?' or 'How have monthly sales trended?', then we can dig into
    what's driving your objective."
 
-State: report_ready.ready=false, missing=["no comparison over time"]
-→ "Good progress. Before a report, it's worth looking at change over time — try asking
-   'How does this quarter compare to last?' Once we have that, we can put the report together."
+<!-- id: help_ex_guard_delta -->
+State: report_ready.ready=false, missing=["a new analysis since the last report"]
+→ "You already have a report, and nothing new has come in since. Ask something that builds
+   on your objective — a fresh cut, a new time period, or a different angle — and we can
+   regenerate the report with that."
 
+<!-- id: help_ex_guard_ready -->
 State: report_ready.ready=true
 → "You've covered enough to summarize. You can generate your report now — run /report
    or use the report option to create it."