# Architecture Design decisions and rationale for prisma-chatbot. This document captures *why* the system is built the way it is, not just *what* it does. See [`README.md`](README.md) for a user-facing overview and [`ROADMAP.md`](ROADMAP.md) for the deployment plan and milestones. ## System overview Each user turn flows through a single, linear path. The Gradio app (`app.py`) appends the user message to the running history held in `gr.State`, prepended with the dual-role system prompt built once at startup by `src.prompt.build_system_prompt`. The full history is handed to `PrismaInferenceClient.generate`, which calls Llama 3.3 70B Instruct via the Hugging Face Inference API with `response_format={"type": "json_object"}` and returns the raw JSON content. `parse_model_output` strips any stray markdown fences, validates the schema (string `response`, integer scores per attribute in `[1, 7]`), and returns a `ParsedTurn`. The app then appends the assistant response to the chat display, records the evaluation in state, and re-renders the impressions panel (colored bar cells) and the trajectory plot. There is exactly one model call per user turn. ## Key design decisions ### Dual-role prompt, single LLM call per turn The model is asked to produce both the conversational reply and the evaluation in a single structured response. Two reasons drive this. The first is operational: one network call keeps per-turn latency and cost predictable, and avoids the engineering of stitching two calls together under partial failure. The second is semantic: the response and the evaluation are grounded in exactly the same context window, so the evaluation reflects the model's actual current impression rather than a separately-prompted second-pass judgment. The trade-off is that the prompt is more elaborate than two narrowly-scoped calls would be, and the JSON contract has to be enforced both at the API boundary (`response_format`) and in the parser. This is accepted because the research thesis is precisely that the model's response and its perception of the user are two facets of one act of interpretation, and collapsing them into a single call mirrors that framing. ### Structured JSON output The model returns a JSON object with a string `response` field and an `evaluation` object mapping each attribute name to an integer score in `[1, 7]`. This makes downstream rendering deterministic: the chat display reads `response` directly, and the impressions panel and trajectory plot iterate over `evaluation` without any natural-language parsing. JSON output is enforced both by the prompt and by passing `response_format={"type": "json_object"}` to the Inference API — belt-and-suspenders, because Llama 3.3 70B occasionally drifts toward conversational preamble before the JSON when relying on prompt instructions alone. Two defensive choices in `parse_model_output` deserve a note. Markdown code fences are stripped if present, because some model snapshots wrap structured output in ``` blocks despite instructions otherwise. And because `bool` is a subclass of `int` in Python, the validator rejects `True`/`False` explicitly — without that check, a model returning `true` for a score would silently pass type validation and be coerced to `1`. ### Six evaluation attributes The v1 attribute set is *competent, likeable, considerate, polite, formal, demanding* — defined once in `src/config.DEFAULT_ATTRIBUTES` and consumed by the prompt builder, the parser, and the UI renderer. The set is intentionally a **general-purpose** one, chosen to produce meaningful variance across the diverse, unconstrained conversational inputs a public demo will receive. It is **not** identical to the attribute set used in the CMCL 2026 / EMNLP 2026 studies: the research-specific dimensions there (notably *pedantic* and *well-prepared*) are highly informative around precision-related stimuli but produce flat scores in casual chat, which would make the live demo look broken. Those research attributes, alongside other candidates (*pushy, knowledgeable, helpful, arrogant, warm, evasive*), are slated for the v2 user-selectable extended list. The demo's research framing is therefore *methodological* — surfacing the model's ongoing social perception of the user — rather than a literal replication of the paper's stimuli. ### Llama 3.3 70B Instruct via HF Inference API Hosted inference on Hugging Face was chosen over self-hosting and over proprietary APIs (OpenAI, Anthropic) for three reasons. First, the deployment surface is minimal: no GPU provisioning, no model serving, no separate auth domain — the Space and the model live on the same platform and a single `HF_TOKEN` covers both. Second, the audience that arrives via the research papers is already familiar with the Hugging Face platform and trusts it, which removes a friction point that a custom-hosted endpoint or a third-party key requirement would introduce. Third, a 70B-class instruct model is empirically the threshold at which the structured JSON contract holds reliably across varied conversational inputs and the dual-role persona is maintained without prompt drift; smaller open-weight models tend to break the schema or leak the evaluation rationale into the reply. The trade-off is dependency on HF endpoint availability and the (low) per-call rate limits applied to public Spaces, which the per-session turn cap (`SESSION_TURN_CAP = 12`) helps absorb. ### Gradio on Hugging Face Spaces Gradio is the lowest-friction path to a public, shareable artifact: a single `app.py` declares the UI, the deployment is `git push`, and the HF Space provides the public URL and HTTPS termination. It also integrates natively with the HF Inference API used for the model call. The cost is limited UI flexibility compared to a custom React frontend, which is accepted because the demo's value is in the interaction itself, not in bespoke visual design. ## Module responsibilities - `src/config.py` — single source of truth for the v1 attribute set, the model ID, decoding parameters, the session turn cap, and the attribute-to-color mapping. Centralizing these means the prompt, the parser, and the UI cannot drift out of sync with each other. - `src/prompt.py` — builds the dual-role system prompt from the attribute list at module import time. Templated rather than hardcoded so that the v2 selectable attributes (see above) plug in without touching the inference layer. - `src/inference.py` — thin wrapper around `huggingface_hub. InferenceClient`. Forces `response_format={"type": "json_object"}` uniformly, distinguishes API errors (`InferenceError`) from parse errors (`EvaluationParseError`) so the app layer can react differently, and validates the response envelope before handing the content to the parser. - `src/evaluation.py` — parses the JSON, validates the schema against the expected attribute list, and formats scores for display. Owns the intensifier scale (`1 → "not at all"`, ..., `7 → "extremely"`) that pairs verbal labels with numeric scores in the UI. - `app.py` — Gradio Blocks assembly, theme/CSS, event wiring, and the rendering of the impressions panel (HTML bar cells) and trajectory plot (matplotlib). On parse or inference failure the user's message is rolled back from state and surfaced as a `gr.Warning` toast rather than as a fake assistant turn, so retries send clean history. ## Open design questions - **Single-call vs. two-call architecture if the attribute set grows.** At six attributes the dual-role prompt is comfortable; if v2 lets users pick from a longer extended list, JSON-output reliability and attention to the response itself may degrade enough to justify splitting into two calls. - **Whether to expose decoding controls (temperature) as a user setting.** Currently fixed at `0.7` in config. Exposing it would let visitors probe how stochasticity affects the evaluation, which aligns with the research framing — but adds a knob that most visitors won't understand. - **Behavior on non-English input.** The persona prompt is English and the attribute labels are English; the model handles other languages reasonably but the social perception it produces has not been validated outside English. The honest disposition for v1 is to accept it silently; a more careful v2 might detect and either surface a disclaimer or refuse.