fix: Stop silent empty-answer fallback; surface tool-error reasons

- _direct_answer's exception handler now logs instead of silently
swallowing -- the last line of defense before the canned placeholder
was failing invisibly with zero trace.
- run() and run_stream() both retry _direct_answer once before ever
showing the placeholder; run_stream() (what the web UI actually uses)
previously didn't attempt this recovery at all. Verified end-to-end on
a real question that previously produced the placeholder: now returns
a real, correctly-formatted answer instead.
- Same _get_grounded_instruction fix: log on failure instead of silently
dropping RAG grounding.
- prompts.py: tool errors often name the exact concept being tested
("uncontrollable", "not positive semi-definite") -- instruct the model
to state that reason as the headline finding instead of routing around
it with a numeric-looking answer that ignores what the error said.
Verified against the 4 known regression cases from today's ControlBench
run: 2 of 4 fixed for real (not just phrasing), 2 remain (one case
where the rule doesn't reliably take, one case that isn't an error at
all -- an ill-conditioned but successful computation -- and needs a
different rule not yet written).
- Add today's ControlBench-v1 run (v3) for the record: 53.3% overall,
down from 77.1% on Aug 17 against the same adapter. Investigated before
concluding anything: most of the code_and_simulation and
underspecified_and_traps drop is a scorer methodology mismatch (fence-
only code detection, 14-keyword trap matching) against today's
correctly-changed behavior (prefer verified tools over pasted code;
terser refusals). Some of it is real: 4 trap items regressed on
ignoring tool-error text, addressed above.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

benchmarks/results/controlai_agent_v3_controlbench_v1.json

@@ -0,0 +1,511 @@
+{
+  "overall_score": 53.3,
+  "pillar_scores": {
+    "theory_and_concepts": 61.7,
+    "numerical_synthesis": 100.0,
+    "code_and_simulation": 6.0,
+    "underspecified_and_traps": 20.0,
+    "real_world_case_studies": 73.8
+  },
+  "total_items": 50,
+  "evaluated_items": 50,
+  "item_results": [
+    {
+      "id": "cb_theory_001",
+      "pillar": "theory_and_concepts",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "concepts_found": 4,
+        "total_concepts": 4,
+        "missing": []
+      }
+    },
+    {
+      "id": "cb_theory_002",
+      "pillar": "theory_and_concepts",
+      "score": 75.0,
+      "details": {
+        "score": 75.0,
+        "concepts_found": 3,
+        "total_concepts": 4,
+        "missing": [
+          "PBH immediately determines stabilizability (if uncontrollable modes have Re(lambda) < 0)"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_003",
+      "pillar": "theory_and_concepts",
+      "score": 25.0,
+      "details": {
+        "score": 25.0,
+        "concepts_found": 1,
+        "total_concepts": 4,
+        "missing": [
+          "RHP zeros cause initial undershoot in step response",
+          "RHP zero at z requires closed-loop bandwidth to be bounded below approximately z/2",
+          "Cannot invert RHP zeros without creating unstable closed-loop pole-zero cancelations"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_004",
+      "pillar": "theory_and_concepts",
+      "score": 50.0,
+      "details": {
+        "score": 50.0,
+        "concepts_found": 2,
+        "total_concepts": 4,
+        "missing": [
+          "Indirect method fails / is inconclusive if any eigenvalue has Re(lambda) = 0 (critical cases)",
+          "Direct method: constructs positive-definite V(x) with negative semi-definite or negative definite V_dot; valid globally or over an estimated invariant set"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_005",
+      "pillar": "theory_and_concepts",
+      "score": 25.0,
+      "details": {
+        "score": 25.0,
+        "concepts_found": 1,
+        "total_concepts": 4,
+        "missing": [
+          "Closed-loop eigenvalues are exactly the union of LQR poles and observer poles",
+          "Doyle (1978) showed that LQG has no guaranteed gain or phase margins (can have arbitrarily small gain margin)",
+          "Observer loop recovery (LTR) is required to restore LQR robustness"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_006",
+      "pillar": "theory_and_concepts",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "concepts_found": 4,
+        "total_concepts": 4,
+        "missing": []
+      }
+    },
+    {
+      "id": "cb_theory_007",
+      "pillar": "theory_and_concepts",
+      "score": 75.0,
+      "details": {
+        "score": 75.0,
+        "concepts_found": 3,
+        "total_concepts": 4,
+        "missing": [
+          "Provides structural robustness against plant parameter variations"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_008",
+      "pillar": "theory_and_concepts",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "concepts_found": 0,
+        "total_concepts": 3,
+        "missing": [
+          "Guaranteed Gain Margin: GM in [1/2, infinity) (or -6 dB to +infinity dB)",
+          "Guaranteed Phase Margin: PM >= 60 degrees (at least +/- 60 deg)",
+          "Result stems from Kalman inequality (return difference inequality): ||I + R^(1/2) K (sI-A)^(-1) B R^(-1/2)|| >= 1 for s=jw"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_009",
+      "pillar": "theory_and_concepts",
+      "score": 66.7,
+      "details": {
+        "score": 66.7,
+        "concepts_found": 2,
+        "total_concepts": 3,
+        "missing": [
+          "Passivity is preferred in physical systems (mechanical, electrical, robotics) with energy storage and collocated actuators/sensors where infinite gain margin is achievable"
+        ]
+      }
+    },
+    {
+      "id": "cb_theory_010",
+      "pillar": "theory_and_concepts",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "concepts_found": 4,
+        "total_concepts": 4,
+        "missing": []
+      }
+    },
+    {
+      "id": "cb_num_001",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_002",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_003",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_004",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_005",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_006",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_007",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_008",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_009",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_010",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_011",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_num_012",
+      "pillar": "numerical_synthesis",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "tool_matched": true,
+        "numeric_verified": true
+      }
+    },
+    {
+      "id": "cb_code_001",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_002",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_003",
+      "pillar": "code_and_simulation",
+      "score": 30.0,
+      "details": {
+        "score": 30.0,
+        "sandbox_executed": false,
+        "message": "AttributeError: Line2D.set() got an unexpected keyword argument 'plotstyle'"
+      }
+    },
+    {
+      "id": "cb_code_004",
+      "pillar": "code_and_simulation",
+      "score": 30.0,
+      "details": {
+        "score": 30.0,
+        "sandbox_executed": false,
+        "message": "ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 1)"
+      }
+    },
+    {
+      "id": "cb_code_005",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_006",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_007",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "matlab_syntax_score": 0.0
+      }
+    },
+    {
+      "id": "cb_code_008",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_009",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_code_010",
+      "pillar": "code_and_simulation",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "status": "No code block found"
+      }
+    },
+    {
+      "id": "cb_trap_001",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_002",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_003",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_004",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_005",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_006",
+      "pillar": "underspecified_and_traps",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "refusal_or_trap_detected": true
+      }
+    },
+    {
+      "id": "cb_trap_007",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_008",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_trap_009",
+      "pillar": "underspecified_and_traps",
+      "score": 100.0,
+      "details": {
+        "score": 100.0,
+        "refusal_or_trap_detected": true
+      }
+    },
+    {
+      "id": "cb_trap_010",
+      "pillar": "underspecified_and_traps",
+      "score": 0.0,
+      "details": {
+        "score": 0.0,
+        "refusal_or_trap_detected": false
+      }
+    },
+    {
+      "id": "cb_case_001",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_002",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_003",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_004",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_005",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_006",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_007",
+      "pillar": "real_world_case_studies",
+      "score": 80.0,
+      "details": {
+        "score": 80.0
+      }
+    },
+    {
+      "id": "cb_case_008",
+      "pillar": "real_world_case_studies",
+      "score": 30.0,
+      "details": {
+        "score": 30.0
+      }
+    }
+  ]
+}

controlai_agent/orchestrator.py

@@ -626,7 +626,8 @@ class ControlAIAgent:
                 "[Nise, Control Systems Engineering, p. 583]); never invent a citation and never print a "
                 "raw filename, file extension, or course code."
             )
-        except Exception:
+        except Exception as exc:
+            print(f"[_get_grounded_instruction] FAILED, continuing ungrounded: {type(exc).__name__}: {exc}")
             return base_instruction
 
     def _direct_answer(
@@ -664,7 +665,11 @@ class ControlAIAgent:
             )
             _, cleaned = _extract_tool_calls(self._generate(rendered, max_tokens=max_tokens))
             return cleaned
-        except Exception:
+        except Exception as exc:
+            # This is the last line of defense before the user sees the canned
+            # placeholder -- if it fails, that failure must leave a trace
+            # instead of vanishing, or a real bug here is undiagnosable.
+            print(f"[_direct_answer] FAILED for prompt {user_prompt[:80]!r}: {type(exc).__name__}: {exc}")
             return ""
 
     def _execute_with_provenance(
@@ -820,6 +825,13 @@ class ControlAIAgent:
         # is stripped down to "" and showing final_output would leak a raw
         # <tool_call>{...}</tool_call> JSON blob straight into the chat.
         final_response = clean_final or self._direct_answer(user_prompt, effective_sys, history)
+        if not final_response:
+            # _direct_answer reliably produces real text when tested in
+            # isolation, so an empty result here is most likely a transient
+            # failure (resource contention, a stray exception) rather than a
+            # repeatable one -- one retry clears most of those before giving
+            # up and showing the placeholder.
+            final_response = self._direct_answer(user_prompt, effective_sys, history)
         if not final_response:
             final_response = "The computational analysis has been completed as detailed above."
         return AgentResult(
@@ -1011,6 +1023,13 @@ class ControlAIAgent:
         final_text = clean_synth
         # Strip any lingering raw json
         final_text = re.sub(r"\{\s*[\"']name[\"']\s*:[\s\S]*?\}\s*\}", "", final_text).strip()
+        if not final_text:
+            # A fresh, tool-free, minimal-context generation reliably produces
+            # real text even when the tool-heavy synthesis turn degenerated --
+            # try it (twice, for a transient failure) before the placeholder.
+            final_text = self._direct_answer(user_prompt, effective_sys, history)
+            if not final_text:
+                final_text = self._direct_answer(user_prompt, effective_sys, history)
         if not final_text:
             final_text = "The computational analysis and simulation have been executed successfully as detailed above."
 

controlai_agent/prompts.py

@@ -37,6 +37,6 @@ When answering theoretical principles, derivations, proofs, comparisons, or limi
 3. **NO RAW LATEX DOCUMENT TAGS**: NEVER output `\begin{figure}`, `\includegraphics`, `\caption`, `\centering`, `\section*`, or `\end{figure}`. Use standard markdown headers (e.g. `### Section Title`).
 4. **MANDATORY TOOL CALL FOR PLOTS**: When asked to plot, visualize, or simulate (Nyquist plot, Bode diagram, Root Locus, Step Response, Phase Portrait), NEVER hallucinate a fake image filename. You MUST explicitly call `execute_python_code` (using `control as ct` or `matplotlib.pyplot`) or `simulate_step_response` to generate and display the real plot!
 5. **Deterministic Grounding**: When a tool executes successfully, present the exact numerical results and generated plot.
-6. **No Infinite Retries**: If a tool returns an error, correct the parameters or synthesize the analytical answer directly.
+6. **No Infinite Retries, but READ the error first**: If a tool returns an error, the error text itself is often the entire answer to what's actually being asked -- "uncontrollable", "singular", "poles can't be placed", "not positive semi-definite" name the exact concept the question is testing. Before doing anything else: quote or restate that reason in your final answer as the headline finding, in plain engineering language (e.g. "this system is uncontrollable because..."). Only after stating it should you correct parameters and retry, or move on. NEVER silently produce a numeric-looking final answer (a bare $$K = ...$$, a computed matrix) that routes around an error without ever mentioning what it said -- an ignored error is a worse answer than no answer.
 7. **No Forced Citations**: Do NOT cite arbitrary random file paths unless the user explicitly requests literature references.
 """