docs/ARCHITECTURE.md

# Architecture

How the pieces fit together, and why each design decision was made.

## Request flow

```
   browser
      │
      ▼
┌─────────────────────────────────────────────────────────────────────┐
│  app.py — Gradio ChatInterface                                      │
│                                                                     │
│  1. input guardrail (regex blocklist)                               │
│     └─ blocked  → canned refusal + footer  (model never called)     │
│                                                                     │
│  2. memory layer (per-turn)                                         │
│     RunnableWithMessageHistory.invoke({"input": msg}, session_id)   │
│       loads last 6 turns from SQLChatMessageHistory (SQLite)        │
│                                                                     │
│  3. assistant._respond                                              │
│     ├─ SystemMessage + trimmed history + HumanMessage               │
│     └─ tool-calling loop (≤ 4 rounds):                              │
│           model.invoke → if tool_calls → run_tool_call → repeat     │
│                                                                     │
│  4. output guardrail (Claude Haiku 4.5 moderation)                  │
│     └─ blocked  → refusal text, AND rewrite stored history          │
│                                                                     │
│  5. status footer (assistant | tools_used | guardrail states)       │
│                                                                     │
│  Everything above is wrapped in a Langfuse @observe trace tagged    │
│  with session_id and assistant_type; tool/model spans nest under.   │
└─────────────────────────────────────────────────────────────────────┘
```

## Module map

| File                              | Responsibility                                                           |
|-----------------------------------|--------------------------------------------------------------------------|
| `app.py`                          | Gradio entry; orchestrates guardrails → memory → assistant per turn.     |
| `src/config.py`                   | pydantic-settings loader; drops empty env vars that would shadow `.env`. |
| `src/assistants/base.py`          | `BaseAssistant` ABC; shared tool-calling loop + history trimming.        |
| `src/assistants/frontier.py`      | `ChatAnthropic` wrapper (Claude Sonnet 4.5).                             |
| `src/assistants/oss.py`           | `QwenChatModel` — custom LangChain chat model with native tool template. |
| `src/memory.py`                   | SQLChatMessageHistory + `build_conversational()` factory.                |
| `src/tools.py`                    | `calculator` (sandboxed AST eval) + `web_search` (Tavily).               |
| `src/guardrails.py`               | input regex blocklist + Haiku-4.5 output moderation.                     |
| `src/observability.py`            | Langfuse client init + `@observe` decorator (no-op fallback).            |
| `eval/datasets.py`                | TruthfulQA / BBQ / AdvBench loaders, seed=42.                            |
| `eval/run_eval.py`                | Resumable JSONL runner.                                                  |
| `eval/judge.py`                   | Claude Sonnet 4.5 LLM-as-judge with dataset-aware rubric.                |
| `eval/report.py`                  | Bootstrap CIs, matplotlib charts, EVALUATION_REPORT.md.                  |

## Key design decisions and why

### 1. Why a single `BaseAssistant` with the tool-loop in the base class

For the comparison to be fair, both assistants must have *identical capabilities*. Putting the tool-calling loop, system prompt, history trimming, and memory plumbing in `BaseAssistant` means the only differences between Claude and Qwen are (a) the underlying LangChain chat model and (b) inference latency. Subclasses implement only `_build_model()`.

### 2. Why we built a custom `QwenChatModel` instead of using `ChatHuggingFace`

`langchain-huggingface.ChatHuggingFace.bind_tools()` does not render tool schemas into Qwen's chat template — so `bind_tools` is silently a no-op and Qwen never emits tool calls through it. Tested with a deliberate calculator question: Qwen wrote prose, never invoked the tool.

Qwen2.5-Instruct's *native* chat template fully supports tools and emits well-formed `<tool_call>{...}</tool_call>` blocks. `QwenChatModel`:

1. Overrides `bind_tools()` to attach OpenAI-style tool schemas to the runnable.
2. Calls `tokenizer.apply_chat_template(messages, tools=schemas, ...)` so Qwen sees the tools.
3. Parses `<tool_call>` blocks out of the output back into LangChain's `AIMessage.tool_calls` format.

Result: Qwen genuinely uses the calculator/search, matching the Claude interface.

### 3. Why guardrails live in the UI layer, not in the assistants

The evaluation must measure *raw* model behavior — that's the only way to honestly compare hallucination, bias, and safety between OSS and frontier. If guardrails ran inside `assistant.chat()`, the eval would measure the *protected* system, not the model itself. So:

- `BaseAssistant.chat()` is stateless and unmoderated → used by the eval.
- `app.respond()` wraps that with input guardrail → memory invocation → output moderation → footer → used by the UI.

This keeps the eval honest while still demonstrating real guardrail behavior in the deployed app.

### 4. Why output moderation also rewrites stored history

A blocked unsafe reply, if persisted, would leak into the next turn's context and could prime the model. So on a block we both replace what the user sees and overwrite the last assistant message in `SQLChatMessageHistory` with the refusal. Rare path, so the `clear() + add_messages()` cycle is cheap.

### 5. Why `RunnableWithMessageHistory` + manual tool loop (rather than LangGraph)

`RunnableWithMessageHistory` is deprecated in LangChain 1.x in favor of LangGraph persistence — but it's still the canonical LangChain memory primitive that most tutorials and docs use, and adding `langgraph` would have meant a much larger dependency surface. The manual tool loop (capped at 4 rounds for safety) is ~15 lines, fully traceable, and easy to reason about.

### 6. Why a 6-turn memory window

Qwen-1.5B has a small effective context; we trim to the last 12 messages (6 user/assistant pairs) before each call so long sessions don't degrade Qwen's quality. SQLite still stores the full session (useful for audit/debugging); the trim is only at *read* time.

### 7. Why pydantic-settings drops empty env vars at config load

pydantic-settings ranks real environment variables above `.env`. The dev environment exported `ANTHROPIC_API_KEY=` (empty string) which silently shadowed the real key in `.env`. The config module deletes any empty-string entries for our known secrets so `.env` can fill them in. This is safe on Hugging Face Spaces — secrets there arrive as non-empty env vars and still win.

### 8. Why the judge uses a shared schema with per-dataset guidance

A single `{hallucinated, biased, refused, harmful, reasoning}` schema means all rows can be aggregated cleanly. But a one-size rubric would mis-grade: e.g., the "correct" BBQ ambig answer is *"Can't be determined"*, which a generic rubric might flag as a refusal. Per-dataset guidance in the prompt tells the judge which dimensions matter and how to interpret edge cases.

## Trade-offs accepted

- **`RunnableWithMessageHistory` deprecation**: future-LangChain incompatibility risk, but it remains the canonical memory primitive in LangChain tutorials and avoids pulling in `langgraph`.
- **Judge self-bias**: the judge is the same model family as one assistant under test. Disclosed in the report; mitigation would be a second judge or human spot-check on a subset.
- **No per-browser session id on Spaces**: a single process-global session id is used; fine for single-user demo, would need `gr.State` + cookie-derived id for genuine multi-user. Noted in README.
- **CPU-only deployment**: Qwen on shared CPU is slow. The `@spaces.GPU` decorator is in place so switching to ZeroGPU is a one-line YAML change once a PRO subscription is active.