Improved rules and added claude.md

.agents/rules/agent-workflow.md

@@ -31,9 +31,9 @@ Interactive Dash app for exploring LLM internals through visualization and exper
 ## Data Files
 | File | Path | Purpose |
 |------|------|---------|
-| Sessions | `.context/data/sessions.md` | Running work log (append-only) |
-| Decisions | `.context/data/decisions.md` | Decision records with reasoning (append-only) |
-| Lessons | `.context/data/lessons.md` | Hard-won knowledge and past mistakes (append-only) |
+| Sessions | `.context/data/sessions.md` | Running work log |
+| Decisions | `.context/data/decisions.md` | Decision records with reasoning |
+| Lessons | `.context/data/lessons.md` | Hard-won knowledge and past mistakes |
 
 ## Memory Maintenance
 
@@ -49,13 +49,16 @@ Always look for opportunities to update the memory system:
 3. Never update memory mid-task without mentioning it
 
 **Rules**:
-- Data files are append-only — add entries, never remove or overwrite past entries
-- Modules can be edited but changes should be targeted, not full rewrites
-- After substantive work sessions, append a summary to `.context/data/sessions.md`
+- Data files are append-first with compression — new entries are appended, but older entries may be compressed when:
+  - **Conflicting information**: A newer entry supersedes an older one (e.g., a feature is replaced or deprecated). Compress the old entry into a brief note that preserves the history without full detail.
+  - **Excessive size**: When a file exceeds ~30 entries, summarize the oldest ~20 into a dense "Archive" block at the top (a few lines), retaining the most recent ~10 as full entries. This keeps long-term memory without bloating context.
+- When compressing, never delete history entirely — always leave a condensed trace so the evolution of the project is visible.
+- Modules can be edited but changes should be targeted, not full rewrites.
+- After substantive work sessions, append a summary to `.context/data/sessions.md`.
 
 ## Preferences
 - Don't ask permission for changes that fall within an approved plan — just execute
 - Commit normal changes to main; feature branches for major components/refactors. Never merge branches.
 - Keep `todo.md` and `plans.md` current before/after changes. Tasks should be atomic.
 - When in doubt, research options and make a minimal reasonable choice, noting it in `todo.md`
-- Explain manual tests clearly — what to look for, expected behavior, where to check
+- Explain manual tests clearly — what to look for, expected behavior, where to check

.claude/commands/update-memory.md

@@ -0,0 +1,13 @@
+This session is wrapping up. Review what was accomplished and update the memory system per the CLAUDE.md instructions.
+
+Steps:
+1. Run `git diff HEAD~3` and `git log --oneline -10` to see recent changes and commits from this session.
+2. Read the current state of the memory/data files:
+   - `.context/data/sessions.md` — append a session summary
+   - `.context/data/decisions.md` — append any decisions made with reasoning
+   - `.context/data/lessons.md` — append any hard-won knowledge or mistakes caught
+3. Check if any `.context/modules/` files need targeted updates based on what changed (new patterns, conventions, architecture shifts).
+4. For each proposed update: state which file changes and what the change is, then wait for approval before writing.
+5. Update `todo.md` and `plans.md` if tasks were completed or new ones emerged.
+
+Additional context from the user (if any): $ARGUMENTS

.cursor/rules/AGENTS.md

@@ -38,9 +38,9 @@ Interactive Dash app for exploring LLM internals through visualization and exper
 ## Data Files
 | File | Path | Purpose |
 |------|------|---------|
-| Sessions | `.context/data/sessions.md` | Running work log (append-only) |
-| Decisions | `.context/data/decisions.md` | Decision records with reasoning (append-only) |
-| Lessons | `.context/data/lessons.md` | Hard-won knowledge and past mistakes (append-only) |
+| Sessions | `.context/data/sessions.md` | Running work log |
+| Decisions | `.context/data/decisions.md` | Decision records with reasoning |
+| Lessons | `.context/data/lessons.md` | Hard-won knowledge and past mistakes |
 
 ## Memory Maintenance
 
@@ -56,9 +56,12 @@ Always look for opportunities to update the memory system:
 3. Never update memory mid-task without mentioning it
 
 **Rules**:
-- Data files are append-only — add entries, never remove or overwrite past entries
-- Modules can be edited but changes should be targeted, not full rewrites
-- After substantive work sessions, append a summary to `.context/data/sessions.md`
+- Data files are append-first with compression — new entries are appended, but older entries may be compressed when:
+  - **Conflicting information**: A newer entry supersedes an older one (e.g., a feature is replaced or deprecated). Compress the old entry into a brief note that preserves the history without full detail.
+  - **Excessive size**: When a file exceeds ~30 entries, summarize the oldest ~20 into a dense "Archive" block at the top (a few lines), retaining the most recent ~10 as full entries. This keeps long-term memory without bloating context.
+- When compressing, never delete history entirely — always leave a condensed trace so the evolution of the project is visible.
+- Modules can be edited but changes should be targeted, not full rewrites.
+- After substantive work sessions, append a summary to `.context/data/sessions.md`.
 
 ## Preferences
 - Don't ask permission for changes that fall within an approved plan — just execute

CLAUDE.md

@@ -0,0 +1,60 @@
+# Transformer Explanation Dashboard
+Interactive Dash app for exploring LLM internals through visualization and experimentation. Built with Python, Dash, PyTorch, HF Transformers, Bertviz, pyvene.
+
+## Critical Rules
+- **Dead code cleanup is mandatory during refactors.** Sweep for orphaned code from deprecated/deleted components every time.
+- **Conceptual understanding over math rigor.** Skip complex derivations; focus on motivation and intuition that builds correct mental models.
+- **Sanity check every change:** (1) Does this help someone learn about transformers? (2) Is the simplification accurate enough? (3) Am I in a rabbit hole?
+- **TDD for backend logic.** Write failing tests first for anything in `utils/`. Skip tests for UI-only changes. Run `pytest` after every change; iterate until green.
+- **Don't run the app.** Describe manual verification steps; the user will test themselves.
+- **Surgical edits over rewrites.** Reuse existing files. Only create new modules when existing ones can't be extended.
+- **No new dependencies** unless strictly necessary.
+- **Push back on bad ideas.** Think through problems fully. Don't be a yes-man — challenge flawed approaches.
+- **Components = layout only** (no ML logic). **Utils = computation only** (no Dash imports). **app.py = glue.**
+- **Dash callbacks must stay responsive.** No heavy sync work in callbacks without feedback indicators.
+- Don't reformat unrelated code or alter indentation styles.
+- Check for zombie processes before debugging server errors.
+
+## Module Map
+| Module | Path | Load when |
+|--------|------|-----------|
+| Product | `.context/modules/product.md` | Understanding vision, features, brand voice, visual identity, UX patterns |
+| Architecture | `.context/modules/architecture.md` | Understanding system structure, data flow, or component boundaries |
+| Conventions | `.context/modules/conventions.md` | Writing or reviewing code style, naming, commits, dead code cleanup |
+| Education | `.context/modules/education.md` | Creating or editing educational content, visualizations, explanations |
+| Testing | `.context/modules/testing.md` | Writing tests, running pytest, TDD workflow |
+
+## Data Files
+| File | Path | Purpose |
+|------|------|---------|
+| Sessions | `.context/data/sessions.md` | Running work log |
+| Decisions | `.context/data/decisions.md` | Decision records with reasoning |
+| Lessons | `.context/data/lessons.md` | Hard-won knowledge and past mistakes |
+
+## Memory Maintenance
+
+Always look for opportunities to update the memory system:
+- **New patterns**: "We've been doing X consistently — should I add it to conventions?"
+- **Decisions made**: "We decided Y — should I record this in decisions.md?"
+- **Mistakes caught**: "This went wrong because Z — should I add it to lessons.md?"
+- **Scope changes**: "The project now includes W — should I create a new module?"
+
+**Before any memory update**:
+1. State which file(s) would change and what the change would be
+2. Wait for approval
+3. Never update memory mid-task without mentioning it
+
+**Rules**:
+- Data files are append-first with compression — new entries are appended, but older entries may be compressed when:
+  - **Conflicting information**: A newer entry supersedes an older one (e.g., a feature is replaced or deprecated). Compress the old entry into a brief note that preserves the history without full detail.
+  - **Excessive size**: When a file exceeds ~30 entries, summarize the oldest ~20 into a dense "Archive" block at the top (a few lines), retaining the most recent ~10 as full entries. This keeps long-term memory without bloating context.
+- When compressing, never delete history entirely — always leave a condensed trace so the evolution of the project is visible.
+- Modules can be edited but changes should be targeted, not full rewrites.
+- After substantive work sessions, append a summary to `.context/data/sessions.md`.
+
+## Preferences
+- Don't ask permission for changes that fall within an approved plan — just execute
+- Commit normal changes to main; feature branches for major components/refactors. Never merge branches.
+- Keep `todo.md` and `plans.md` current before/after changes. Tasks should be atomic.
+- When in doubt, research options and make a minimal reasonable choice, noting it in `todo.md`
+- Explain manual tests clearly — what to look for, expected behavior, where to check

RESEARCH-jargon-reduction.md

@@ -0,0 +1,78 @@
+# RESEARCH-jargon-reduction
+
+## Objective
+Analyze the LLMVis project for technical jargon (e.g., "head", "ablation", "tokenizing") that may confuse younger or less technical audiences, and outline plain-English alternatives and explanations. The goal is to make the educational dashboard more accessible without completely losing technical accuracy.
+
+## Findings: Jargon in User-Facing Components
+
+### 1. "Ablation" / "Ablate"
+**Locations found:**
+- `app.py`: "Ablation" tab in the Investigation Panel
+- `components/ablation_panel.py`: "What is Ablation?", "Add Head to Ablation List", "Run Ablation Experiment", "Ablated Output"
+
+**Issue:** "Ablation" is a highly specialized term originally from neuroscience/medicine, adapted for machine learning to mean removing a component to observe its effect. It is completely opaque to a layperson.
+
+**Proposed Replacements:**
+- Replace primary "Ablation" headers with "Component Removal" or "Test by Removing".
+- Replace "Run Ablation Experiment" with "Test What Happens When Removed".
+- Keep "Ablation" only in an advanced glossary, a tooltip or as a secondary subtitle: "Test by Removing (Ablation)".
+
+### 2. "Token" / "Tokenization" / "Tokenizing"
+**Locations found:**
+- `app.py`: "Number of New Tokens" 
+- `components/pipeline.py`: "Tokenization" pipeline stage, "Your text is split into tokens"
+- `components/investigation_panel.py`: "Token Attribution", "Target Token"
+
+**Issue:** While "token" is explained somewhat in the app ("pieces that the model can understand"), phrases like "Token Attribution" or "Tokenization" sound overly academic as top-level headings.
+
+**Proposed Replacements:**
+- Use "Text Chunking" or "Word Pieces" as the primary heading, introducing "Tokens" as the technical equivalent inside the explanation.
+- Change "Token Attribution" to "Which words influenced the answer?" or "Word Influence Score".
+
+### 3. "Attention Head" / "Head"
+**Locations found:**
+- `components/pipeline.py`: "Attention has multiple heads"
+- `components/ablation_panel.py`: "Select heads to ablate", "Attention Head Roles"
+
+**Issue:** "Head" in "Attention Head" is an architectural metaphor that doesn't organically mean anything to someone unfamiliar with the transformer architecture (multi-head attention).
+
+**Proposed Replacements:**
+- Reframe as "Attention Pattern", "Context Detector" or "Feature Spotter".
+- Explain "Heads" as "Specialized Detectors" or "Information Filters" (e.g., "The model uses multiple 'Detectors' (called heads) to find relationships...").
+
+### 4. "Embedding" / "Vector"
+**Locations found:**
+- `components/pipeline.py`: "Embedding" stage, "high-dimensional vector", "lookup table"
+
+**Issue:** "Vector" and "high-dimensional" sound like advanced mathematical phrases which may intimidate a younger user.
+
+**Proposed Replacements:**
+- Instead of "Embedding", use "Meaning Translation" or "Number Dictionary".
+- Instead of "high-dimensional vector", use "list of numbers representing meaning".
+
+### 5. "MLP" / "Feed-Forward"
+**Locations found:**
+- `components/pipeline.py`: "MLP (Feed-Forward)", "MLP layers"
+
+**Issue:** MLP (Multi-Layer Perceptron) is legacy neural network jargon that is not intuitive in the context of what it actually does (storing factual knowledge).
+
+**Proposed Replacements:**
+- Rename the pipeline stage to "Factual Knowledge" or "Memory Layers".
+- Include "(MLP/Feed-Forward)" only as a small subtitle for those who want the exact technical term.
+
+### 6. "Logit" / "Probability" / "Activation Score"
+**Locations found:**
+- `components/ablation_panel.py`: "Average prob shift", "activation: 85%"
+
+**Issue:** Jargon related to raw model outputs. "Activation score" sounds like a futuristic/sci-fi term.
+
+**Proposed Replacements:**
+- "Activation score" -> "Confidence level" or "Strength of focus".
+- "Average prob shift" -> "Average change in confidence".
+
+## What Was Ruled Out
+- **Completely eliminating all technical terms:** We ruled out removing words like "Token" or "Model" entirely because learning some valid terminology is part of the educational value of the tool. Instead, we will prioritize plain English for headings and actions, and introduce the technical terms in parentheses or explanatory sentences.
+- **Rewriting the RAG Docs right now:** The `rag_docs/` folder contains extensive markdown documentation. For this primary pass, we ruled out modifying those long-form files, focusing entirely on the primary UI headers, buttons, and short stage descriptions in `app.py` and the `components/` directory to maximize immediate impact.
+
+## Next Steps
+Review this document. If these jargon replacements aligned with the intended goal for younger audiences, we can draft `SPEC-jargon-reduction.md` with explicit strings to be updated across the application.

spec_attention_color_matching.md

@@ -1,199 +0,0 @@
-# SPEC: Head Color Swatches in LLMVis
-**Status:** Ready for implementation  
-**Date:** 2026-03-05  
-
----
-
-## Goal
-
-Add colored square swatches next to attention head labels in:
-1. The attention head category panels (pipeline.py)
-2. The ablation panel's selected head chips (ablation_panel.py)
-
-The colors must match what BertViz's head_view renders for each head index, so students can visually correlate "L4-H11 [magenta square]" in the panel with the magenta checkbox in the BertViz visualization.
-
----
-
-## Core Insight
-
-BertViz's `head_view(..., html_action='return')` returns the full JS inlined as a string. We can patch `d3.schemeCategory10` → a custom 24-color palette at generation time. The swatches in our UI then reference the same palette — perfect correlation, zero color collisions.
-
-**Why not schemeCategory10?** GPT-2 (default model) has 12 heads/layer. schemeCategory10 cycles every 10, so H10 = H0 (both blue) and H11 = H1 (both orange). That defeats the purpose.
-
----
-
-## The Palette
-
-Dark24 (Plotly) with two near-invisible entries replaced:
-
-```python
-BERTVIZ_HEAD_COLORS = [
-    '#2E91E5',  # H0  - blue
-    '#E15F99',  # H1  - pink-red
-    '#1CA71C',  # H2  - green
-    '#FB0D0D',  # H3  - red
-    '#DA16FF',  # H4  - purple
-    '#00B4D8',  # H5  - cyan       (was #222A2A near-black → swapped)
-    '#B68100',  # H6  - gold
-    '#06D6A0',  # H7  - teal       (was #750D86 very dark purple → swapped)
-    '#EB663B',  # H8  - orange
-    '#511CFB',  # H9  - indigo
-    '#00A08B',  # H10 - dark teal
-    '#FB00D1',  # H11 - magenta
-    '#FC0080',  # H12 - hot pink
-    '#B2828D',  # H13 - mauve
-    '#6C7C32',  # H14 - olive
-    '#778AAE',  # H15 - slate blue
-]
-```
-
-16 colors covers all supported models (max 16 heads/layer: gpt2-medium, pythia-410m, Qwen2.5-0.5B).
-
----
-
-## Files to Change
-
-### 1. New file: `utils/colors.py`
-Define `BERTVIZ_HEAD_COLORS` here as the single source of truth. Import from here everywhere else.
-
-```python
-"""Shared color constants for LLMVis."""
-
-# 16-color palette for BertViz head_view.
-# Patched into BertViz at generation time (model_patterns.py) and mirrored
-# in UI swatches (pipeline.py, ablation_panel.py) for visual correlation.
-# Based on Plotly Dark24, with H5 and H7 replaced for legibility.
-BERTVIZ_HEAD_COLORS = [
-    '#2E91E5',  # H0  - blue
-    '#E15F99',  # H1  - pink-red
-    '#1CA71C',  # H2  - green
-    '#FB0D0D',  # H3  - red
-    '#DA16FF',  # H4  - purple
-    '#00B4D8',  # H5  - cyan
-    '#B68100',  # H6  - gold
-    '#06D6A0',  # H7  - teal
-    '#EB663B',  # H8  - orange
-    '#511CFB',  # H9  - indigo
-    '#00A08B',  # H10 - dark teal
-    '#FB00D1',  # H11 - magenta
-    '#FC0080',  # H12 - hot pink
-    '#B2828D',  # H13 - mauve
-    '#6C7C32',  # H14 - olive
-    '#778AAE',  # H15 - slate blue
-]
-
-def head_color(head_index: int) -> str:
-    """Return the swatch color for a given head index."""
-    return BERTVIZ_HEAD_COLORS[head_index % len(BERTVIZ_HEAD_COLORS)]
-```
-
----
-
-### 2. `utils/model_patterns.py` — `generate_bertviz_html()`
-
-After the `head_view(...)` call, patch the returned HTML string to replace BertViz's default color scheme with ours. Both the primary and fallback d3 lines must be replaced.
-
-**Locate:** the `else:` block in `generate_bertviz_html()` (around line 1414):
-```python
-html_result = head_view(attentions, tokens, html_action='return')
-return html_result.data if hasattr(html_result, 'data') else str(html_result)
-```
-
-**Replace with:**
-```python
-from utils.colors import BERTVIZ_HEAD_COLORS
-html_result = head_view(attentions, tokens, html_action='return')
-html_str = html_result.data if hasattr(html_result, 'data') else str(html_result)
-
-# Patch BertViz color scheme to match our swatch palette (no collisions for ≤16 heads)
-_colors_js = repr(BERTVIZ_HEAD_COLORS).replace("'", '"')  # JSON-safe array literal
-_patch = f"headColors = d3.scaleOrdinal({_colors_js});"
-html_str = html_str.replace(
-    'headColors = d3.scaleOrdinal(d3.schemeCategory10);',
-    _patch
-)
-html_str = html_str.replace(
-    'headColors = d3.scale.category10();',
-    _patch
-)
-return html_str
-```
-
----
-
-### 3. `components/pipeline.py` — `create_attention_content()`
-
-Import `head_color` from `utils.colors` at the top of the file.
-
-In the `head_items` loop (around line 536), the head label is currently:
-```python
-html.Span(label, style={
-    'fontFamily': 'monospace', 'fontSize': '12px', 'fontWeight': '500',
-    'minWidth': '60px', 'color': '#495057' if is_active else '#aaa',
-}, title=f"See Layer {head_info['layer']}, Head {head_info['head']} in the visualization below"),
-```
-
-**Replace with:**
-```python
-html.Span([
-    html.Span("■ ", style={
-        'color': head_color(head_info['head']),
-        'fontSize': '14px',
-    }),
-    label
-], style={
-    'fontFamily': 'monospace', 'fontSize': '12px', 'fontWeight': '500',
-    'minWidth': '60px', 'color': '#495057' if is_active else '#aaa',
-    'display': 'inline-flex', 'alignItems': 'center',
-}, title=f"See Layer {head_info['layer']}, Head {head_info['head']} in the visualization below"),
-```
-
-Note: The swatch always renders at full color even for inactive heads — the parent `html.Div` already applies `opacity: 0.5` for inactive heads, which will naturally dim the swatch too. No special handling needed.
-
----
-
-### 4. `components/ablation_panel.py` — `create_selected_heads_display()`
-
-Import `head_color` from `utils.colors` at the top of the file.
-
-In the chips loop, the label is currently:
-```python
-label = f"L{layer}-H{head}"
-
-chips.append(
-    html.Span([
-        html.Span(label, style={'marginRight': '6px'}),
-        ...
-    ], ...)
-)
-```
-
-**Replace** the `html.Span(label, ...)` with:
-```python
-html.Span([
-    html.Span("■ ", style={
-        'color': head_color(head),
-        'fontSize': '14px',
-        'marginRight': '2px',
-    }),
-    html.Span(label, style={'marginRight': '6px'}),
-], style={'display': 'inline-flex', 'alignItems': 'center'}),
-```
-
----
-
-## Exit Criteria
-
-- [ ] BertViz head_view renders with the new palette — H0 is blue, H10 is dark teal, H11 is magenta (visually distinct from H1)
-- [ ] Every head label in the attention category panels shows a colored square matching BertViz
-- [ ] Every ablation chip shows a matching colored square
-- [ ] Spot checks: `L0-H0` = `#2E91E5`, `L4-H10` = `#00A08B`, `L4-H11` = `#FB00D1`
-- [ ] `BERTVIZ_HEAD_COLORS` / `head_color()` defined in exactly one place (`utils/colors.py`), imported everywhere else
-- [ ] No functional changes to callbacks, stores, or any other logic
-- [ ] Existing tests still pass
-
-## Non-Goals
-- No changes to model_view (not used in this app)
-- No hover/interactive effects on swatches
-- No changes to per-category color scheme (the purple/blue/green category accent colors are separate)
-- No changes beyond the 4 files listed above