Upload CONTEXT.md

docs/CONTEXT.md

@@ -2,6 +2,19 @@
 
 > **Upload this file to refresh Claude's context when continuing work on this project.**
 
+**Last Updated:** January 2026
+
+---
+
+## 📁 Files Changed in Latest Session
+
+| File | What Changed |
+|------|--------------|
+| `agents/extractor.py` | Enhanced 5-source extraction (DOM, CSS vars, SVG, inline, stylesheets) |
+| `core/preview_generator.py` | Added AS-IS previews for Stage 1 (colors, spacing, radius, shadows) |
+| `app.py` | Stage 1 UI now has 5 preview tabs, enhanced logging shows extraction sources |
+| `docs/CONTEXT.md` | Updated with all changes, enhanced architecture diagrams |
+
 ---
 
 ## 🎯 Project Goal
@@ -9,12 +22,95 @@
 Build a **semi-automated, human-in-the-loop agentic system** that:
 1. Reverse-engineers a design system from a live website
 2. Reconstructs and upgrades it into a modern, scalable design system
-3. Outputs production-ready JSON tokens
+3. Outputs production-ready JSON tokens (Figma Tokens Studio compatible)
 
 **Philosophy:** This is a design-aware co-pilot, NOT a magic button. Humans decide, agents propose.
 
 ---
 
+## 🤔 Why This Project? (Market Differentiation)
+
+### The Problem We Solve
+
+| Pain Point | Who Has It | Current Solutions | Why They Fail |
+|------------|------------|-------------------|---------------|
+| Legacy websites with no design system | Enterprise teams | Manual audit (weeks) | Time-consuming, error-prone |
+| Inconsistent design tokens scattered in CSS | Agencies inheriting projects | Figma plugins (style extractors) | Only extract from Figma, not live sites |
+| Need to modernize without breaking existing | Product teams | Design system generators | Generate new, don't reverse-engineer existing |
+| AA compliance gaps unknown | Accessibility teams | Contrast checkers | Check one color at a time, no system view |
+
+### Existing Tools & Their Gaps
+
+| Tool | What It Does | Gap We Fill |
+|------|--------------|-------------|
+| **Figma Tokens Studio** | Manages tokens in Figma | Doesn't extract from websites |
+| **Style Dictionary** | Transforms tokens to code | Needs tokens first (we create them) |
+| **Polypane/VisBug** | Inspect live sites | No systematic extraction or upgrade |
+| **AI Design Tools** (Galileo, Uizard) | Generate new designs | Don't reverse-engineer existing |
+| **CSS Stats** | Analyze CSS files | Statistics only, no actionable tokens |
+| **Chromatic/Percy** | Visual regression | Compare, don't extract or upgrade |
+
+### Our Unique Value Proposition
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                     WHAT MAKES US DIFFERENT                                 │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  1. REVERSE-ENGINEERING (not generation)                                    │
+│     • Extracts from LIVE websites, not design files                         │
+│     • Preserves what's working, upgrades what's broken                      │
+│     • Respects existing brand decisions                                     │
+│                                                                             │
+│  2. MULTI-AGENT REASONING (not single LLM)                                  │
+│     • Two analysts with different perspectives                              │
+│     • HEAD compiler resolves conflicts                                      │
+│     • Shows reasoning, not just results                                     │
+│                                                                             │
+│  3. HUMAN-IN-THE-LOOP (not magic button)                                    │
+│     • Designer reviews every stage                                          │
+│     • Accept/reject individual tokens                                       │
+│     • Choose from upgrade OPTIONS, not forced decisions                     │
+│                                                                             │
+│  4. VISUAL PREVIEWS (not just data tables)                                  │
+│     • Typography rendered in actual detected font                           │
+│     • Color ramps with AA compliance per shade                              │
+│     • See before you export                                                 │
+│                                                                             │
+│  5. COST-TRANSPARENT (not black box)                                        │
+│     • Shows token usage and cost per analysis                               │
+│     • Uses HF free tier ($0.10/mo) or Pro ($2/mo)                          │
+│     • ~$0.05 per full analysis                                              │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Target Users
+
+| User | Use Case | Value |
+|------|----------|-------|
+| **UX Managers** (like you!) | Modernize legacy booking platforms | Weeks → Hours |
+| **Design System Teams** | Audit and standardize existing properties | Systematic, not ad-hoc |
+| **Agencies** | Onboard client projects with no documentation | Instant design inventory |
+| **Accessibility Consultants** | AA compliance audit with fixes | Full palette view |
+| **Developers** | Get production-ready tokens from designer's website | No manual translation |
+
+### Why Not Just Use [X]?
+
+**"Why not just inspect the CSS manually?"**
+→ You could, but it takes weeks for a complex site. We do it in minutes with systematic coverage.
+
+**"Why not use Figma's native styles?"**
+→ Many legacy sites were never in Figma. We extract from the source of truth: the live website.
+
+**"Why do you need AI? Can't rules handle this?"**
+→ Rules extract tokens. AI understands *design intent* — why is this color used here? What scale was intended? Where does it deviate from best practices?
+
+**"Isn't this just CSS Stats with AI?"**
+→ CSS Stats tells you what exists. We tell you what it *should* be and give you actionable upgrade paths.
+
+---
+
 ## 🏗️ Architecture Overview
 
 ```
@@ -23,10 +119,10 @@ Build a **semi-automated, human-in-the-loop agentic system** that:
 ├─────────────────────────────────────────────────────────────────────────────┤
 │  Frontend:        Gradio (long-scroll, sectioned UI with live preview)      │
 │  Orchestration:   LangGraph (agent state management & workflow)             │
-│  Models:          HuggingFace Inference API (see model assignments below)   │
+│  Models:          HuggingFace Inference Providers (Novita, Groq, etc.)     │
 │  Hosting:         Hugging Face Spaces                                       │
 │  Storage:         HF Spaces persistent storage                              │
-│  Output:          Platform-agnostic JSON tokens                             │
+│  Output:          Platform-agnostic JSON tokens (Figma Tokens Studio)       │
 └─────────────────────────────────────────────────────────────────────────────┘
 ```
 
@@ -34,7 +130,7 @@ Build a **semi-automated, human-in-the-loop agentic system** that:
 
 ## 🧠 Model Assignments
 
-### Stage 2: Multi-Agent Analysis (NEW!)
+### Stage 2: Multi-Agent Analysis
 
 | Agent | Role | Model | Provider | Cost |
 |-------|------|-------|----------|------|
@@ -45,14 +141,29 @@ Build a **semi-automated, human-in-the-loop agentic system** that:
 
 **Architecture:**
 ```
-LLM 1 (Qwen) ─┐
-              ├──> HEAD (Llama) ──> Final Recommendations
-LLM 2 (Llama) ─┘
-Rules ─────────┘
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                         PARALLEL ANALYSIS                                    │
+│                                                                             │
+│   LLM 1 (Qwen)          LLM 2 (Llama)         Rule Engine                  │
+│   ┌──────────┐          ┌──────────┐          ┌──────────┐                 │
+│   │ Global   │          │ Western  │          │ Math     │                 │
+│   │ Design   │          │ Design   │          │ Only     │                 │
+│   │ Patterns │          │ Patterns │          │ (FREE)   │                 │
+│   └────┬─────┘          └────┬─────┘          └────┬─────┘                 │
+│        │                     │                     │                        │
+│        └─────────────────────┼─────────────────────┘                        │
+│                              │                                              │
+│                              ▼                                              │
+│                    ┌──────────���──────┐                                      │
+│                    │   HEAD COMPILER │                                      │
+│                    │                 │                                      │
+│                    │ • Compare views │                                      │
+│                    │ • Resolve diff  │                                      │
+│                    │ • Final recs    │                                      │
+│                    └─────────────────┘                                      │
+└─────────────────────────────────────────────────────────────────────────────┘
 ```
 
-LLM 1 and LLM 2 run in **PARALLEL** via LangGraph, then HEAD compiles results.
-
 ### Other Agents
 
 | Agent | Role | Model | Provider | Why |
@@ -80,17 +191,127 @@ Estimated cost per Stage 2 analysis: **~$0.05**
 
 ---
 
+## 👁️ Visual Previews
+
+### Stage 1: AS-IS Previews (No Enhancements)
+
+Shows raw extracted values exactly as found on the website:
+
+| Preview | What It Shows |
+|---------|---------------|
+| **Typography** | Actual font rendered with detected styles |
+| **Colors** | Simple swatches with hex, frequency, context, AA status |
+| **Spacing** | Visual bars representing each spacing value |
+| **Radius** | Boxes with each border-radius applied |
+| **Shadows** | Cards with each box-shadow applied |
+
+### Stage 2: Enhanced Previews (Upgraded)
+
+Shows proposed upgrades and improvements:
+
+| Preview | What It Shows |
+|---------|---------------|
+| **Typography** | Type scale comparison (1.2, 1.25, 1.333 ratios) |
+| **Color Ramps** | 11 shades (50-950) with AA compliance per shade |
+
+---
+
+## 🔍 Enhanced Extraction (Agent 1)
+
+Agent 1 now extracts from **5 sources** to capture ALL colors:
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                    ENHANCED EXTRACTION SOURCES                               │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  1. DOM Computed Styles                                                     │
+│     • window.getComputedStyle(element)                                      │
+│     • Captures: color, background-color, border-color, etc.                 │
+│                                                                             │
+│  2. CSS Variables                                                           │
+│     • :root { --primary-color: #3860be; }                                  │
+│     • Parses all stylesheets for CSS custom properties                     │
+│                                                                             │
+│  3. SVG Colors                                                              │
+│     • <svg fill="#00c4cc">                                                 │
+│     • <path stroke="#3860be">                                              │
+│                                                                             │
+│  4. Inline Styles                                                           │
+│     • <div style="background-color: #bcd432;">                             │
+│     • Parses style attributes for color values                             │
+│                                                                             │
+│  5. Stylesheet Rules                                                        │
+│     • Parses CSS rules that may not be applied to visible elements         │
+│     • Catches hover states, pseudo-elements, etc.                          │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 📋 Enhanced Logging
+
+### Stage 1 Extraction Logs
+
+Shows detailed extraction progress:
+```
+============================================================
+🖥️ DESKTOP EXTRACTION (1440px)
+============================================================
+
+📡 Enhanced extraction from 5 sources:
+   1. DOM computed styles (getComputedStyle)
+   2. CSS variables (:root { --color: })
+   3. SVG colors (fill, stroke)
+   4. Inline styles (style='color:')
+   5. Stylesheet rules (CSS files)
+
+📊 EXTRACTION RESULTS:
+   Colors:     45 unique
+   Typography: 12 styles
+   Spacing:    28 values
+   Radius:     8 values
+   Shadows:    4 values
+
+🎨 CSS Variables found: 15
+   --primary-color: #3860be
+   --accent-color: #00c4cc
+   --brand-lime: #bcd432
+   ... and 12 more
+
+🔄 Normalizing (deduping, naming)...
+   ✅ Normalized: 32 colors, 10 typography, 18 spacing
+```
+
+### Stage 2 LLM Analysis Logs
+
+Shows detailed reasoning from each agent:
+- What they analyzed
+- Scores per category (Typography, Colors, AA, Spacing)
+- Specific findings and recommendations
+- Confidence levels
+- How HEAD resolved disagreements
+- Cost per call and total
+
+---
+
 ## 🤖 Agent Personas
 
-### Agent 1: Website Crawler & Extractor
+### Agent 1: Website Crawler & Enhanced Extractor
 - **Persona:** Meticulous Design Archaeologist
 - **Tool:** Playwright
 - **Job:** 
   - Auto-discover 10+ pages from base URL
   - Crawl Desktop (1440px) + Mobile (375px) separately
   - Scroll to bottom + wait for network idle
-  - Extract: colors, typography, spacing, radius, shadows
-- **Output:** Raw tokens with frequency, context, confidence
+  - **ENHANCED: Extract from 5 sources:**
+    1. DOM computed styles (`getComputedStyle`)
+    2. CSS variables (`:root { --primary: #xxx }`)
+    3. SVG colors (`fill`, `stroke` attributes)
+    4. Inline styles (`style="background-color: #xxx"`)
+    5. Stylesheet rules (CSS files, hover states, pseudo-elements)
+- **Output:** Raw tokens with frequency, context, confidence, source type
 
 ### Agent 2: Token Normalizer & Structurer
 - **Persona:** Design System Librarian
@@ -114,20 +335,30 @@ Estimated cost per Stage 2 analysis: **~$0.05**
   - Convert finalized tokens to Figma-compatible JSON
   - Generate: typography, color (with tints/shades), spacing variables
   - Maintain Desktop + Mobile + version metadata
-- **Output:** Production-ready JSON
+- **Output:** Production-ready JSON (flat structure for Figma Tokens Studio)
 
 ---
 
 ## 🖥️ UI Stages (3 Stages)
 
-### Stage 1: Extraction Review
-- **Purpose:** Trust building
-- **Shows:** Token tables, color swatches, font previews, confidence indicators
+### Stage 1: Extraction Review (AS-IS)
+- **Purpose:** Trust building — show exactly what was extracted
+- **Shows:** 
+  - Token tables (colors, typography, spacing)
+  - **5 Visual Preview Tabs (AS-IS, no enhancements):**
+    1. 🔤 Typography — actual font rendered
+    2. 🎨 Colors — simple swatches (no ramps)
+    3. 📏 Spacing — visual bars
+    4. 🔘 Radius — rounded boxes
+    5. 🌑 Shadows — shadow cards
 - **Human Actions:** Accept/reject tokens, flag anomalies, toggle Desktop↔Mobile
 
 ### Stage 2: Upgrade Playground (MOST IMPORTANT)
 - **Purpose:** Decision-making through live visuals
-- **Shows:** Side-by-side option selector + live preview
+- **Shows:** 
+  - Side-by-side option selector + live preview
+  - **Color Ramps (50-950 shades with AA compliance)**
+  - Type scale options (1.2, 1.25, 1.333)
 - **Human Actions:** Select type scale A/B/C, spacing system, color ramps — preview updates instantly
 
 ### Stage 3: Final Review & Export
@@ -358,32 +589,48 @@ class Viewport(Enum):
 
 ---
 
-## 🛠️ Implementation Phases
+## 🛠️ Implementation Phases & Current Status
 
-### Phase 1 (Current)
+### Phase 1 ✅ COMPLETE
 - [x] Project structure
 - [x] Configuration files
-- [ ] Token schema (Pydantic models)
-- [ ] Agent 1: Crawler
-- [ ] Agent 1: Extractor
-- [ ] Agent 2: Normalizer
-- [ ] Stage 1 UI
-- [ ] LangGraph basic workflow
-
-### Phase 2
-- [ ] Agent 3: Advisor
-- [ ] Stage 2 UI (Upgrade Playground)
-- [ ] Live preview system
-
-### Phase 3
-- [ ] Agent 4: Generator
-- [ ] Stage 3 UI
-- [ ] Export functionality
-
-### Phase 4
+- [x] Token schema (Pydantic models)
+- [x] Agent 1: Crawler (page discovery)
+- [x] Agent 1: Enhanced Extractor (5-source extraction)
+- [x] Agent 2: Normalizer
+- [x] Stage 1 UI with 5 AS-IS preview tabs
+- [x] LangGraph basic workflow
+- [x] JSON export (flat structure for Figma)
+
+### Phase 2 ✅ MOSTLY COMPLETE
+- [x] Agent 3: Multi-LLM Advisor (Qwen + Llama + HEAD)
+- [x] Stage 2 UI (Upgrade Playground)
+- [x] Live preview system (typography, color ramps)
+- [x] Enhanced LLM logging with reasoning
+- [ ] Accept/Reject checkbox wiring to export
+
+### Phase 3 🔄 IN PROGRESS
+- [ ] Agent 4: Generator (component patterns)
+- [ ] Stage 3 UI (diff view)
+- [ ] Arabic page filtering
+
+### Phase 4 ⏳ PENDING
 - [ ] Full LangGraph orchestration
 - [ ] HF Spaces deployment
 - [ ] Persistent storage
+- [ ] MCP Claude / Figma plugin integration (Part 2 of article)
+
+---
+
+## 🐛 Known Issues & Pending Fixes
+
+| Issue | Status | Fix |
+|-------|--------|-----|
+| Arabic pages included | Pending | Filter `/ar/` URLs in crawler |
+| Accept/Reject not wired | Pending | Export should respect checkbox state |
+| Stage 1 vs Stage 2 preview confusion | ✅ Fixed | Stage 1 now shows AS-IS (no ramps) |
+| Colors missed from CSS variables | ✅ Fixed | Enhanced 5-source extraction |
+| JSON nested structure | ✅ Fixed | Flat structure for Figma compatibility |
 
 ---
 
@@ -414,6 +661,8 @@ When continuing this project:
 4. **Maintain LangGraph state** consistency across agents
 5. **Use Gradio components** from ui/components.py for consistency
 6. **Test with** real websites before deployment
+7. **Enhanced extraction** captures from 5 sources — check logs to verify
+8. **Stage 1 = AS-IS** (no ramps), **Stage 2 = Enhanced** (with ramps)
 
 ---