Upload 20 files

Dockerfile

@@ -0,0 +1,60 @@
+FROM python:3.11-slim
+
+# Set working directory
+WORKDIR /app
+
+# Install system dependencies for Playwright
+RUN apt-get update && apt-get install -y \
+    wget \
+    gnupg \
+    ca-certificates \
+    fonts-liberation \
+    libasound2 \
+    libatk-bridge2.0-0 \
+    libatk1.0-0 \
+    libatspi2.0-0 \
+    libcups2 \
+    libdbus-1-3 \
+    libdrm2 \
+    libgbm1 \
+    libgtk-3-0 \
+    libnspr4 \
+    libnss3 \
+    libxcomposite1 \
+    libxdamage1 \
+    libxfixes3 \
+    libxkbcommon0 \
+    libxrandr2 \
+    xdg-utils \
+    libu2f-udev \
+    libvulkan1 \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for caching
+COPY requirements.txt .
+
+# Install Python dependencies
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Install Playwright browsers
+RUN playwright install chromium
+RUN playwright install-deps chromium
+
+# Copy application code
+COPY . .
+
+# Create non-root user for security
+RUN useradd -m -u 1000 user
+USER user
+
+# Set environment variables
+ENV HOME=/home/user \
+    PATH=/home/user/.local/bin:$PATH \
+    GRADIO_SERVER_NAME=0.0.0.0 \
+    GRADIO_SERVER_PORT=7860
+
+# Expose port
+EXPOSE 7860
+
+# Run the application
+CMD ["python", "app.py"]

README.md

@@ -1,10 +1,233 @@
 ---
-title: Design System Extractor 2
-emoji: 🐠
-colorFrom: blue
-colorTo: purple
+title: Design System Extractor v2
+emoji: 🎨
+colorFrom: purple
+colorTo: blue
 sdk: docker
 pinned: false
+license: mit
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Design System Extractor v2
+
+> 🎨 A semi-automated, human-in-the-loop agentic system that reverse-engineers design systems from live websites.
+
+## 🎯 What It Does
+
+When you have a website but no design system documentation (common when the original Sketch/Figma files are lost), this tool helps you:
+
+1. **Crawl** your website to discover pages
+2. **Extract** design tokens (colors, typography, spacing, shadows)
+3. **Review** and validate extracted tokens with visual previews
+4. **Upgrade** your system with modern best practices (optional)
+5. **Export** production-ready JSON tokens for Figma/code
+
+## 🧠 Philosophy
+
+This is **not a magic button** — it's a design-aware co-pilot.
+
+- **Agents propose → Humans decide**
+- **Every action is visible, reversible, and previewed**
+- **No irreversible automation**
+
+## 🏗️ Architecture
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│                        TECH STACK                            │
+├──────────────────────────────────────────────────────────────┤
+│  Frontend:       Gradio (interactive UI with live preview)   │
+│  Orchestration:  LangGraph (agent workflow management)       │
+│  Models:         Claude API (reasoning) + Rule-based         │
+│  Browser:        Playwright (crawling & extraction)          │
+│  Hosting:        Hugging Face Spaces                         │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Agent Personas
+
+| Agent | Persona | Job |
+|-------|---------|-----|
+| **Agent 1** | Design Archaeologist | Discover pages, extract raw tokens |
+| **Agent 2** | Design System Librarian | Normalize, dedupe, structure tokens |
+| **Agent 3** | Senior DS Architect | Recommend upgrades (type scales, spacing, a11y) |
+| **Agent 4** | Automation Engineer | Generate final JSON for Figma/code |
+
+## 🚀 Quick Start
+
+### Prerequisites
+
+- Python 3.11+
+- Node.js (for some dependencies)
+
+### Installation
+
+```bash
+# Clone the repository
+git clone <repo-url>
+cd design-system-extractor
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # or `venv\Scripts\activate` on Windows
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Install Playwright browsers
+playwright install chromium
+
+# Copy environment file
+cp config/.env.example config/.env
+# Edit .env and add your ANTHROPIC_API_KEY
+```
+
+### Running
+
+```bash
+python app.py
+```
+
+Open `http://localhost:7860` in your browser.
+
+## 📖 Usage Guide
+
+### Stage 1: Discovery
+
+1. Enter your website URL (e.g., `https://example.com`)
+2. Click "Discover Pages"
+3. Review discovered pages and select which to extract from
+4. Ensure you have a mix of page types (homepage, listing, detail, etc.)
+
+### Stage 2: Extraction
+
+1. Choose viewport (Desktop 1440px or Mobile 375px)
+2. Click "Extract Tokens"
+3. Review extracted:
+   - **Colors**: With frequency, context, and AA compliance
+   - **Typography**: Font families, sizes, weights
+   - **Spacing**: Values with 8px grid fit indicators
+4. Accept or reject individual tokens
+
+### Stage 3: Export
+
+1. Review final token set
+2. Export as JSON
+3. Import into Figma via Tokens Studio or your plugin
+
+## 📁 Project Structure
+
+```
+design-system-extractor/
+├── app.py                          # Main Gradio application
+├── requirements.txt
+├── README.md
+│
+├── config/
+│   ├── .env.example                # Environment template
+│   ├── agents.yaml                 # Agent personas & settings
+│   └── settings.py                 # Configuration loader
+│
+├── agents/
+│   ├── state.py                    # LangGraph state definitions
+│   ├── graph.py                    # Workflow orchestration
+│   ├── crawler.py                  # Agent 1: Page discovery
+│   ├── extractor.py                # Agent 1: Token extraction
+│   ├── normalizer.py               # Agent 2: Normalization
+│   ├── advisor.py                  # Agent 3: Best practices
+│   └── generator.py                # Agent 4: JSON generation
+│
+├── core/
+│   ├── token_schema.py             # Pydantic data models
+│   └── color_utils.py              # Color analysis utilities
+│
+├── ui/
+│   └── (Gradio components)
+│
+└── docs/
+    └── CONTEXT.md                  # Context file for AI assistance
+```
+
+## 🔧 Configuration
+
+### Environment Variables
+
+```env
+# Required
+ANTHROPIC_API_KEY=your_key_here
+
+# Optional
+DEBUG=false
+LOG_LEVEL=INFO
+BROWSER_HEADLESS=true
+```
+
+### Agent Configuration
+
+Agent personas and behavior are defined in `config/agents.yaml`. This includes:
+
+- Extraction targets (colors, typography, spacing)
+- Naming conventions
+- Confidence thresholds
+- Upgrade options
+
+## 🛠️ Development
+
+### Running Tests
+
+```bash
+pytest tests/
+```
+
+### Adding New Features
+
+1. Update token schema in `core/token_schema.py`
+2. Add agent logic in `agents/`
+3. Update UI in `app.py`
+4. Update `docs/CONTEXT.md` for AI assistance
+
+## 📦 Output Format
+
+Tokens are exported in a platform-agnostic JSON format:
+
+```json
+{
+  "metadata": {
+    "source_url": "https://example.com",
+    "version": "v1-recovered",
+    "viewport": "desktop"
+  },
+  "colors": {
+    "primary-500": {
+      "value": "#007bff",
+      "source": "detected",
+      "contrast_white": 4.5
+    }
+  },
+  "typography": {
+    "heading-lg": {
+      "fontFamily": "Inter",
+      "fontSize": "24px",
+      "fontWeight": 700
+    }
+  },
+  "spacing": {
+    "md": {
+      "value": "16px",
+      "source": "detected"
+    }
+  }
+}
+```
+
+## 🤝 Contributing
+
+Contributions are welcome! Please read the contribution guidelines first.
+
+## 📄 License
+
+MIT
+
+---
+
+Built with ❤️ for designers who've lost their source files.

agents/__init__.py

@@ -0,0 +1,32 @@
+"""
+Agents for Design System Extractor v2.
+
+This package contains the LangGraph agents:
+- Agent 1: Crawler & Extractor (Design Archaeologist)
+- Agent 2: Normalizer (Design System Librarian)
+- Agent 3: Advisor (Senior Staff DS Architect)
+- Agent 4: Generator (Automation Engineer)
+"""
+
+from agents.state import (
+    AgentState,
+    create_initial_state,
+    get_stage_progress,
+)
+
+from agents.graph import (
+    build_workflow_graph,
+    WorkflowRunner,
+    create_workflow,
+)
+
+__all__ = [
+    # State
+    "AgentState",
+    "create_initial_state",
+    "get_stage_progress",
+    # Graph
+    "build_workflow_graph",
+    "WorkflowRunner",
+    "create_workflow",
+]

agents/crawler.py

@@ -0,0 +1,350 @@
+"""
+Agent 1: Website Crawler
+Design System Extractor v2
+
+Persona: Meticulous Design Archaeologist
+
+Responsibilities:
+- Auto-discover pages from base URL
+- Classify page types (homepage, listing, detail, etc.)
+- Prepare page list for user confirmation
+"""
+
+import asyncio
+import re
+from urllib.parse import urljoin, urlparse
+from typing import Optional, Callable
+from datetime import datetime
+
+from playwright.async_api import async_playwright, Browser, Page, BrowserContext
+
+from core.token_schema import DiscoveredPage, PageType, Viewport
+from config.settings import get_settings
+
+
+class PageDiscoverer:
+    """
+    Discovers pages from a website for design system extraction.
+    
+    This is the first part of Agent 1's job — finding pages before
+    the human confirms which ones to crawl.
+    """
+    
+    def __init__(self):
+        self.settings = get_settings()
+        self.browser: Optional[Browser] = None
+        self.context: Optional[BrowserContext] = None
+        self.visited_urls: set[str] = set()
+        self.discovered_pages: list[DiscoveredPage] = []
+        
+    async def __aenter__(self):
+        """Async context manager entry."""
+        await self._init_browser()
+        return self
+    
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        """Async context manager exit."""
+        await self._close_browser()
+    
+    async def _init_browser(self):
+        """Initialize Playwright browser."""
+        playwright = await async_playwright().start()
+        self.browser = await playwright.chromium.launch(
+            headless=self.settings.browser.headless
+        )
+        self.context = await self.browser.new_context(
+            viewport={
+                "width": self.settings.viewport.desktop_width,
+                "height": self.settings.viewport.desktop_height,
+            },
+            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
+        )
+    
+    async def _close_browser(self):
+        """Close browser and cleanup."""
+        if self.context:
+            await self.context.close()
+        if self.browser:
+            await self.browser.close()
+    
+    def _normalize_url(self, url: str, base_url: str) -> Optional[str]:
+        """Normalize and validate URL."""
+        # Handle relative URLs
+        if not url.startswith(('http://', 'https://')):
+            url = urljoin(base_url, url)
+        
+        parsed = urlparse(url)
+        base_parsed = urlparse(base_url)
+        
+        # Only allow same domain
+        if parsed.netloc != base_parsed.netloc:
+            return None
+        
+        # Remove fragments and normalize
+        normalized = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"
+        
+        # Remove trailing slash for consistency
+        if normalized.endswith('/') and len(normalized) > len(f"{parsed.scheme}://{parsed.netloc}/"):
+            normalized = normalized.rstrip('/')
+        
+        return normalized
+    
+    def _classify_page_type(self, url: str, title: str = "") -> PageType:
+        """
+        Classify page type based on URL patterns and title.
+        
+        This is a heuristic — not perfect, but good enough for discovery.
+        """
+        url_lower = url.lower()
+        title_lower = title.lower() if title else ""
+        
+        # Check URL patterns
+        patterns = {
+            PageType.HOMEPAGE: [r'/$', r'/home$', r'/index'],
+            PageType.LISTING: [r'/products', r'/catalog', r'/list', r'/category', r'/collection', r'/search'],
+            PageType.DETAIL: [r'/product/', r'/item/', r'/detail/', r'/p/', r'/[a-z-]+/\d+'],
+            PageType.FORM: [r'/contact', r'/form', r'/apply', r'/submit', r'/register'],
+            PageType.AUTH: [r'/login', r'/signin', r'/signup', r'/auth', r'/account'],
+            PageType.CHECKOUT: [r'/cart', r'/checkout', r'/basket', r'/payment'],
+            PageType.MARKETING: [r'/landing', r'/promo', r'/campaign', r'/offer'],
+            PageType.ABOUT: [r'/about', r'/team', r'/company', r'/story'],
+            PageType.CONTACT: [r'/contact', r'/support', r'/help'],
+        }
+        
+        for page_type, url_patterns in patterns.items():
+            for pattern in url_patterns:
+                if re.search(pattern, url_lower):
+                    return page_type
+        
+        # Check title patterns
+        title_patterns = {
+            PageType.HOMEPAGE: ['home', 'welcome'],
+            PageType.LISTING: ['products', 'catalog', 'collection', 'browse'],
+            PageType.DETAIL: ['product', 'item'],
+            PageType.AUTH: ['login', 'sign in', 'sign up', 'register'],
+            PageType.ABOUT: ['about', 'our story', 'team'],
+            PageType.CONTACT: ['contact', 'get in touch', 'support'],
+        }
+        
+        for page_type, keywords in title_patterns.items():
+            for keyword in keywords:
+                if keyword in title_lower:
+                    return page_type
+        
+        return PageType.OTHER
+    
+    async def _extract_links(self, page: Page, base_url: str) -> list[str]:
+        """Extract all internal links from a page."""
+        links = await page.evaluate("""
+            () => {
+                const links = Array.from(document.querySelectorAll('a[href]'));
+                return links.map(a => a.href).filter(href => 
+                    href && 
+                    !href.startsWith('javascript:') && 
+                    !href.startsWith('mailto:') &&
+                    !href.startsWith('tel:') &&
+                    !href.includes('#')
+                );
+            }
+        """)
+        
+        # Normalize and filter
+        valid_links = []
+        for link in links:
+            normalized = self._normalize_url(link, base_url)
+            if normalized and normalized not in self.visited_urls:
+                valid_links.append(normalized)
+        
+        return list(set(valid_links))
+    
+    async def _get_page_title(self, page: Page) -> str:
+        """Get page title."""
+        try:
+            return await page.title()
+        except Exception:
+            return ""
+    
+    async def discover(
+        self,
+        base_url: str,
+        max_pages: int = None,
+        progress_callback: Optional[Callable[[float], None]] = None
+    ) -> list[DiscoveredPage]:
+        """
+        Discover pages from a website.
+        
+        Args:
+            base_url: The starting URL
+            max_pages: Maximum pages to discover (default from settings)
+            progress_callback: Optional callback for progress updates
+        
+        Returns:
+            List of discovered pages
+        """
+        max_pages = max_pages or self.settings.crawl.max_pages
+        
+        async with self:
+            # Start with homepage
+            normalized_base = self._normalize_url(base_url, base_url)
+            if not normalized_base:
+                raise ValueError(f"Invalid base URL: {base_url}")
+            
+            queue = [normalized_base]
+            self.visited_urls = set()
+            self.discovered_pages = []
+            
+            while queue and len(self.discovered_pages) < max_pages:
+                current_url = queue.pop(0)
+                
+                if current_url in self.visited_urls:
+                    continue
+                
+                self.visited_urls.add(current_url)
+                
+                try:
+                    page = await self.context.new_page()
+                    
+                    # Navigate to page
+                    await page.goto(
+                        current_url,
+                        wait_until="networkidle",
+                        timeout=self.settings.browser.timeout
+                    )
+                    
+                    # Get page info
+                    title = await self._get_page_title(page)
+                    page_type = self._classify_page_type(current_url, title)
+                    depth = len(urlparse(current_url).path.split('/')) - 1
+                    
+                    # Create discovered page
+                    discovered = DiscoveredPage(
+                        url=current_url,
+                        title=title,
+                        page_type=page_type,
+                        depth=depth,
+                        selected=True,
+                    )
+                    self.discovered_pages.append(discovered)
+                    
+                    # Extract links for further crawling
+                    new_links = await self._extract_links(page, base_url)
+                    
+                    # Prioritize certain page types
+                    priority_patterns = ['/product', '/listing', '/category', '/about', '/contact']
+                    priority_links = [l for l in new_links if any(p in l.lower() for p in priority_patterns)]
+                    other_links = [l for l in new_links if l not in priority_links]
+                    
+                    # Add to queue (priority first)
+                    for link in priority_links + other_links:
+                        if link not in self.visited_urls and link not in queue:
+                            queue.append(link)
+                    
+                    await page.close()
+                    
+                    # Progress callback
+                    if progress_callback:
+                        progress = len(self.discovered_pages) / max_pages
+                        progress_callback(min(progress, 1.0))
+                    
+                    # Rate limiting
+                    await asyncio.sleep(self.settings.crawl.crawl_delay_ms / 1000)
+                    
+                except Exception as e:
+                    # Log error but continue
+                    discovered = DiscoveredPage(
+                        url=current_url,
+                        title="",
+                        page_type=PageType.OTHER,
+                        depth=0,
+                        selected=False,
+                        error=str(e),
+                    )
+                    self.discovered_pages.append(discovered)
+            
+            return self.discovered_pages
+    
+    def get_pages_by_type(self) -> dict[PageType, list[DiscoveredPage]]:
+        """Group discovered pages by type."""
+        grouped: dict[PageType, list[DiscoveredPage]] = {}
+        for page in self.discovered_pages:
+            if page.page_type not in grouped:
+                grouped[page.page_type] = []
+            grouped[page.page_type].append(page)
+        return grouped
+    
+    def get_suggested_pages(self, min_pages: int = None) -> list[DiscoveredPage]:
+        """
+        Get suggested pages for extraction.
+        
+        Ensures diversity of page types and prioritizes key templates.
+        """
+        min_pages = min_pages or self.settings.crawl.min_pages
+        
+        # Priority order for page types
+        priority_types = [
+            PageType.HOMEPAGE,
+            PageType.LISTING,
+            PageType.DETAIL,
+            PageType.FORM,
+            PageType.MARKETING,
+            PageType.AUTH,
+            PageType.ABOUT,
+            PageType.CONTACT,
+            PageType.OTHER,
+        ]
+        
+        selected = []
+        grouped = self.get_pages_by_type()
+        
+        # First pass: get at least one of each priority type
+        for page_type in priority_types:
+            if page_type in grouped and grouped[page_type]:
+                # Take the first (usually shallowest) page of this type
+                page = sorted(grouped[page_type], key=lambda p: p.depth)[0]
+                if page not in selected:
+                    selected.append(page)
+        
+        # Second pass: fill up to min_pages with remaining pages
+        remaining = [p for p in self.discovered_pages if p not in selected and not p.error]
+        remaining.sort(key=lambda p: p.depth)
+        
+        while len(selected) < min_pages and remaining:
+            selected.append(remaining.pop(0))
+        
+        # Mark as selected
+        for page in selected:
+            page.selected = True
+        
+        return selected
+
+
+# =============================================================================
+# CONVENIENCE FUNCTIONS
+# =============================================================================
+
+async def discover_pages(base_url: str, max_pages: int = 20) -> list[DiscoveredPage]:
+    """Convenience function to discover pages."""
+    discoverer = PageDiscoverer()
+    return await discoverer.discover(base_url, max_pages)
+
+
+async def quick_discover(base_url: str) -> dict:
+    """Quick discovery returning summary dict."""
+    pages = await discover_pages(base_url)
+    
+    return {
+        "total_found": len(pages),
+        "by_type": {
+            pt.value: len([p for p in pages if p.page_type == pt])
+            for pt in PageType
+        },
+        "pages": [
+            {
+                "url": p.url,
+                "title": p.title,
+                "type": p.page_type.value,
+                "selected": p.selected,
+            }
+            for p in pages
+        ],
+    }

agents/extractor.py

@@ -0,0 +1,608 @@
+"""
+Agent 1: Token Extractor
+Design System Extractor v2
+
+Persona: Meticulous Design Archaeologist
+
+Responsibilities:
+- Crawl pages at specified viewport
+- Extract computed styles from all elements
+- Collect colors, typography, spacing, radius, shadows
+- Track frequency and context for each token
+"""
+
+import asyncio
+import re
+from typing import Optional, Callable
+from datetime import datetime
+from collections import defaultdict
+
+from playwright.async_api import async_playwright, Browser, Page, BrowserContext
+
+from core.token_schema import (
+    Viewport,
+    ExtractedTokens,
+    ColorToken,
+    TypographyToken,
+    SpacingToken,
+    RadiusToken,
+    ShadowToken,
+    FontFamily,
+    TokenSource,
+    Confidence,
+)
+from core.color_utils import (
+    normalize_hex,
+    parse_color,
+    get_contrast_with_white,
+    get_contrast_with_black,
+    check_wcag_compliance,
+)
+from config.settings import get_settings
+
+
+class TokenExtractor:
+    """
+    Extracts design tokens from web pages.
+    
+    This is the second part of Agent 1's job — after pages are confirmed,
+    we crawl and extract all CSS values.
+    """
+    
+    def __init__(self, viewport: Viewport = Viewport.DESKTOP):
+        self.settings = get_settings()
+        self.viewport = viewport
+        self.browser: Optional[Browser] = None
+        self.context: Optional[BrowserContext] = None
+        
+        # Token collection
+        self.colors: dict[str, ColorToken] = {}
+        self.typography: dict[str, TypographyToken] = {}
+        self.spacing: dict[str, SpacingToken] = {}
+        self.radius: dict[str, RadiusToken] = {}
+        self.shadows: dict[str, ShadowToken] = {}
+        
+        # Font tracking
+        self.font_families: dict[str, FontFamily] = {}
+        
+        # Statistics
+        self.total_elements = 0
+        self.errors: list[str] = []
+        self.warnings: list[str] = []
+    
+    async def __aenter__(self):
+        """Async context manager entry."""
+        await self._init_browser()
+        return self
+    
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        """Async context manager exit."""
+        await self._close_browser()
+    
+    async def _init_browser(self):
+        """Initialize Playwright browser."""
+        playwright = await async_playwright().start()
+        self.browser = await playwright.chromium.launch(
+            headless=self.settings.browser.headless
+        )
+        
+        # Set viewport based on extraction mode
+        if self.viewport == Viewport.DESKTOP:
+            width = self.settings.viewport.desktop_width
+            height = self.settings.viewport.desktop_height
+        else:
+            width = self.settings.viewport.mobile_width
+            height = self.settings.viewport.mobile_height
+        
+        self.context = await self.browser.new_context(
+            viewport={"width": width, "height": height},
+            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
+        )
+    
+    async def _close_browser(self):
+        """Close browser and cleanup."""
+        if self.context:
+            await self.context.close()
+        if self.browser:
+            await self.browser.close()
+    
+    async def _scroll_page(self, page: Page):
+        """Scroll page to load lazy content."""
+        await page.evaluate("""
+            async () => {
+                const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
+                const height = document.body.scrollHeight;
+                const step = window.innerHeight;
+                
+                for (let y = 0; y < height; y += step) {
+                    window.scrollTo(0, y);
+                    await delay(100);
+                }
+                
+                // Scroll back to top
+                window.scrollTo(0, 0);
+            }
+        """)
+        
+        # Wait for network idle after scrolling
+        await page.wait_for_load_state("networkidle", timeout=self.settings.browser.network_idle_timeout)
+    
+    async def _extract_styles_from_page(self, page: Page) -> dict:
+        """
+        Extract computed styles from all elements on the page.
+        
+        This is the core extraction logic — we get getComputedStyle for every element.
+        """
+        styles_data = await page.evaluate("""
+            () => {
+                const elements = document.querySelectorAll('*');
+                const results = {
+                    colors: [],
+                    typography: [],
+                    spacing: [],
+                    radius: [],
+                    shadows: [],
+                    elements_count: elements.length,
+                };
+                
+                const colorProperties = [
+                    'color', 'background-color', 'border-color',
+                    'border-top-color', 'border-right-color', 
+                    'border-bottom-color', 'border-left-color',
+                    'outline-color', 'text-decoration-color',
+                ];
+                
+                const spacingProperties = [
+                    'margin-top', 'margin-right', 'margin-bottom', 'margin-left',
+                    'padding-top', 'padding-right', 'padding-bottom', 'padding-left',
+                    'gap', 'row-gap', 'column-gap',
+                ];
+                
+                elements.forEach(el => {
+                    const tag = el.tagName.toLowerCase();
+                    const styles = window.getComputedStyle(el);
+                    
+                    // Skip invisible elements
+                    if (styles.display === 'none' || styles.visibility === 'hidden') {
+                        return;
+                    }
+                    
+                    // --- COLORS ---
+                    colorProperties.forEach(prop => {
+                        const value = styles.getPropertyValue(prop);
+                        if (value && value !== 'rgba(0, 0, 0, 0)' && value !== 'transparent') {
+                            results.colors.push({
+                                value: value,
+                                property: prop,
+                                element: tag,
+                                context: prop.includes('background') ? 'background' : 
+                                        prop.includes('border') ? 'border' : 'text',
+                            });
+                        }
+                    });
+                    
+                    // --- TYPOGRAPHY ---
+                    const fontFamily = styles.getPropertyValue('font-family');
+                    const fontSize = styles.getPropertyValue('font-size');
+                    const fontWeight = styles.getPropertyValue('font-weight');
+                    const lineHeight = styles.getPropertyValue('line-height');
+                    const letterSpacing = styles.getPropertyValue('letter-spacing');
+                    
+                    if (fontSize && fontFamily) {
+                        results.typography.push({
+                            fontFamily: fontFamily,
+                            fontSize: fontSize,
+                            fontWeight: fontWeight,
+                            lineHeight: lineHeight,
+                            letterSpacing: letterSpacing,
+                            element: tag,
+                        });
+                    }
+                    
+                    // --- SPACING ---
+                    spacingProperties.forEach(prop => {
+                        const value = styles.getPropertyValue(prop);
+                        if (value && value !== '0px' && value !== 'auto' && value !== 'normal') {
+                            const px = parseFloat(value);
+                            if (!isNaN(px) && px > 0 && px < 500) {
+                                results.spacing.push({
+                                    value: value,
+                                    valuePx: Math.round(px),
+                                    property: prop,
+                                    context: prop.includes('margin') ? 'margin' : 
+                                            prop.includes('padding') ? 'padding' : 'gap',
+                                });
+                            }
+                        }
+                    });
+                    
+                    // --- BORDER RADIUS ---
+                    const radiusProps = [
+                        'border-radius', 'border-top-left-radius', 
+                        'border-top-right-radius', 'border-bottom-left-radius',
+                        'border-bottom-right-radius',
+                    ];
+                    
+                    radiusProps.forEach(prop => {
+                        const value = styles.getPropertyValue(prop);
+                        if (value && value !== '0px') {
+                            results.radius.push({
+                                value: value,
+                                element: tag,
+                            });
+                        }
+                    });
+                    
+                    // --- BOX SHADOW ---
+                    const shadow = styles.getPropertyValue('box-shadow');
+                    if (shadow && shadow !== 'none') {
+                        results.shadows.push({
+                            value: shadow,
+                            element: tag,
+                        });
+                    }
+                });
+                
+                return results;
+            }
+        """)
+        
+        return styles_data
+    
+    def _process_color(self, color_data: dict) -> Optional[str]:
+        """Process and normalize a color value."""
+        value = color_data.get("value", "")
+        
+        # Parse and normalize
+        parsed = parse_color(value)
+        if not parsed:
+            return None
+        
+        return parsed.hex
+    
+    def _aggregate_colors(self, raw_colors: list[dict]):
+        """Aggregate color data from extraction."""
+        for color_data in raw_colors:
+            hex_value = self._process_color(color_data)
+            if not hex_value:
+                continue
+            
+            if hex_value not in self.colors:
+                # Calculate contrast ratios
+                contrast_white = get_contrast_with_white(hex_value)
+                contrast_black = get_contrast_with_black(hex_value)
+                compliance = check_wcag_compliance(hex_value, "#ffffff")
+                
+                self.colors[hex_value] = ColorToken(
+                    value=hex_value,
+                    frequency=0,
+                    contexts=[],
+                    elements=[],
+                    css_properties=[],
+                    contrast_white=round(contrast_white, 2),
+                    contrast_black=round(contrast_black, 2),
+                    wcag_aa_large_text=compliance["aa_large_text"],
+                    wcag_aa_small_text=compliance["aa_normal_text"],
+                )
+            
+            # Update frequency and context
+            token = self.colors[hex_value]
+            token.frequency += 1
+            
+            context = color_data.get("context", "")
+            if context and context not in token.contexts:
+                token.contexts.append(context)
+            
+            element = color_data.get("element", "")
+            if element and element not in token.elements:
+                token.elements.append(element)
+            
+            prop = color_data.get("property", "")
+            if prop and prop not in token.css_properties:
+                token.css_properties.append(prop)
+    
+    def _aggregate_typography(self, raw_typography: list[dict]):
+        """Aggregate typography data from extraction."""
+        for typo_data in raw_typography:
+            # Create unique key
+            font_family = typo_data.get("fontFamily", "")
+            font_size = typo_data.get("fontSize", "")
+            font_weight = typo_data.get("fontWeight", "400")
+            line_height = typo_data.get("lineHeight", "normal")
+            
+            key = f"{font_size}|{font_weight}|{font_family[:50]}"
+            
+            if key not in self.typography:
+                # Parse font size to px
+                font_size_px = None
+                if font_size.endswith("px"):
+                    try:
+                        font_size_px = float(font_size.replace("px", ""))
+                    except ValueError:
+                        pass
+                
+                # Parse line height
+                line_height_computed = None
+                if line_height and line_height != "normal":
+                    if line_height.endswith("px") and font_size_px:
+                        try:
+                            lh_px = float(line_height.replace("px", ""))
+                            line_height_computed = round(lh_px / font_size_px, 2)
+                        except ValueError:
+                            pass
+                    else:
+                        try:
+                            line_height_computed = float(line_height)
+                        except ValueError:
+                            pass
+                
+                self.typography[key] = TypographyToken(
+                    font_family=font_family.split(",")[0].strip().strip('"\''),
+                    font_size=font_size,
+                    font_size_px=font_size_px,
+                    font_weight=int(font_weight) if font_weight.isdigit() else 400,
+                    line_height=line_height,
+                    line_height_computed=line_height_computed,
+                    letter_spacing=typo_data.get("letterSpacing"),
+                    frequency=0,
+                    elements=[],
+                )
+            
+            # Update
+            token = self.typography[key]
+            token.frequency += 1
+            
+            element = typo_data.get("element", "")
+            if element and element not in token.elements:
+                token.elements.append(element)
+            
+            # Track font families
+            primary_font = token.font_family
+            if primary_font not in self.font_families:
+                self.font_families[primary_font] = FontFamily(
+                    name=primary_font,
+                    fallbacks=[f.strip().strip('"\'') for f in font_family.split(",")[1:]],
+                    frequency=0,
+                )
+            self.font_families[primary_font].frequency += 1
+    
+    def _aggregate_spacing(self, raw_spacing: list[dict]):
+        """Aggregate spacing data from extraction."""
+        for space_data in raw_spacing:
+            value = space_data.get("value", "")
+            value_px = space_data.get("valuePx", 0)
+            
+            key = str(value_px)
+            
+            if key not in self.spacing:
+                self.spacing[key] = SpacingToken(
+                    value=f"{value_px}px",
+                    value_px=value_px,
+                    frequency=0,
+                    contexts=[],
+                    properties=[],
+                    fits_base_4=value_px % 4 == 0,
+                    fits_base_8=value_px % 8 == 0,
+                )
+            
+            token = self.spacing[key]
+            token.frequency += 1
+            
+            context = space_data.get("context", "")
+            if context and context not in token.contexts:
+                token.contexts.append(context)
+            
+            prop = space_data.get("property", "")
+            if prop and prop not in token.properties:
+                token.properties.append(prop)
+    
+    def _aggregate_radius(self, raw_radius: list[dict]):
+        """Aggregate border radius data."""
+        for radius_data in raw_radius:
+            value = radius_data.get("value", "")
+            
+            # Normalize to simple format
+            # "8px 8px 8px 8px" -> "8px"
+            parts = value.split()
+            if len(set(parts)) == 1:
+                value = parts[0]
+            
+            if value not in self.radius:
+                value_px = None
+                if value.endswith("px"):
+                    try:
+                        value_px = int(float(value.replace("px", "")))
+                    except ValueError:
+                        pass
+                
+                self.radius[value] = RadiusToken(
+                    value=value,
+                    value_px=value_px,
+                    frequency=0,
+                    elements=[],
+                    fits_base_4=value_px % 4 == 0 if value_px else False,
+                    fits_base_8=value_px % 8 == 0 if value_px else False,
+                )
+            
+            token = self.radius[value]
+            token.frequency += 1
+            
+            element = radius_data.get("element", "")
+            if element and element not in token.elements:
+                token.elements.append(element)
+    
+    def _aggregate_shadows(self, raw_shadows: list[dict]):
+        """Aggregate box shadow data."""
+        for shadow_data in raw_shadows:
+            value = shadow_data.get("value", "")
+            
+            if value not in self.shadows:
+                self.shadows[value] = ShadowToken(
+                    value=value,
+                    frequency=0,
+                    elements=[],
+                )
+            
+            token = self.shadows[value]
+            token.frequency += 1
+            
+            element = shadow_data.get("element", "")
+            if element and element not in token.elements:
+                token.elements.append(element)
+    
+    def _calculate_confidence(self, frequency: int) -> Confidence:
+        """Calculate confidence level based on frequency."""
+        if frequency >= 10:
+            return Confidence.HIGH
+        elif frequency >= 3:
+            return Confidence.MEDIUM
+        return Confidence.LOW
+    
+    def _detect_spacing_base(self) -> Optional[int]:
+        """Detect the base spacing unit (4 or 8)."""
+        fits_4 = sum(1 for s in self.spacing.values() if s.fits_base_4)
+        fits_8 = sum(1 for s in self.spacing.values() if s.fits_base_8)
+        
+        total = len(self.spacing)
+        if total == 0:
+            return None
+        
+        # If 80%+ values fit base 8, use 8
+        if fits_8 / total >= 0.8:
+            return 8
+        # If 80%+ values fit base 4, use 4
+        elif fits_4 / total >= 0.8:
+            return 4
+        
+        return None
+    
+    async def extract(
+        self,
+        pages: list[str],
+        progress_callback: Optional[Callable[[float], None]] = None
+    ) -> ExtractedTokens:
+        """
+        Extract tokens from a list of pages.
+        
+        Args:
+            pages: List of URLs to crawl
+            progress_callback: Optional callback for progress updates
+        
+        Returns:
+            ExtractedTokens with all discovered tokens
+        """
+        start_time = datetime.now()
+        pages_crawled = []
+        
+        async with self:
+            for i, url in enumerate(pages):
+                try:
+                    page = await self.context.new_page()
+                    
+                    # Navigate
+                    await page.goto(
+                        url,
+                        wait_until="domcontentloaded",
+                        timeout=self.settings.browser.timeout
+                    )
+                    
+                    # Scroll to load lazy content
+                    await self._scroll_page(page)
+                    
+                    # Extract styles
+                    styles = await self._extract_styles_from_page(page)
+                    
+                    # Aggregate
+                    self._aggregate_colors(styles.get("colors", []))
+                    self._aggregate_typography(styles.get("typography", []))
+                    self._aggregate_spacing(styles.get("spacing", []))
+                    self._aggregate_radius(styles.get("radius", []))
+                    self._aggregate_shadows(styles.get("shadows", []))
+                    
+                    self.total_elements += styles.get("elements_count", 0)
+                    pages_crawled.append(url)
+                    
+                    await page.close()
+                    
+                    # Progress callback
+                    if progress_callback:
+                        progress_callback((i + 1) / len(pages))
+                    
+                    # Rate limiting
+                    await asyncio.sleep(self.settings.crawl.crawl_delay_ms / 1000)
+                    
+                except Exception as e:
+                    self.errors.append(f"Error extracting {url}: {str(e)}")
+        
+        # Calculate confidence for all tokens
+        for token in self.colors.values():
+            token.confidence = self._calculate_confidence(token.frequency)
+        for token in self.typography.values():
+            token.confidence = self._calculate_confidence(token.frequency)
+        for token in self.spacing.values():
+            token.confidence = self._calculate_confidence(token.frequency)
+        
+        # Detect spacing base
+        spacing_base = self._detect_spacing_base()
+        
+        # Mark outliers in spacing
+        if spacing_base:
+            for token in self.spacing.values():
+                if spacing_base == 8 and not token.fits_base_8:
+                    token.is_outlier = True
+                elif spacing_base == 4 and not token.fits_base_4:
+                    token.is_outlier = True
+        
+        # Determine primary font
+        if self.font_families:
+            primary_font = max(self.font_families.values(), key=lambda f: f.frequency)
+            primary_font.usage = "primary"
+        
+        # Build result
+        end_time = datetime.now()
+        duration_ms = int((end_time - start_time).total_seconds() * 1000)
+        
+        return ExtractedTokens(
+            viewport=self.viewport,
+            source_url=pages[0] if pages else "",
+            pages_crawled=pages_crawled,
+            colors=list(self.colors.values()),
+            typography=list(self.typography.values()),
+            spacing=list(self.spacing.values()),
+            radius=list(self.radius.values()),
+            shadows=list(self.shadows.values()),
+            font_families=list(self.font_families.values()),
+            spacing_base=spacing_base,
+            extraction_timestamp=start_time,
+            extraction_duration_ms=duration_ms,
+            total_elements_analyzed=self.total_elements,
+            unique_colors=len(self.colors),
+            unique_font_sizes=len(set(t.font_size for t in self.typography.values())),
+            unique_spacing_values=len(self.spacing),
+            errors=self.errors,
+            warnings=self.warnings,
+        )
+
+
+# =============================================================================
+# CONVENIENCE FUNCTIONS
+# =============================================================================
+
+async def extract_from_pages(
+    pages: list[str],
+    viewport: Viewport = Viewport.DESKTOP
+) -> ExtractedTokens:
+    """Convenience function to extract tokens from pages."""
+    extractor = TokenExtractor(viewport=viewport)
+    return await extractor.extract(pages)
+
+
+async def extract_both_viewports(pages: list[str]) -> tuple[ExtractedTokens, ExtractedTokens]:
+    """Extract tokens from both desktop and mobile viewports."""
+    desktop_extractor = TokenExtractor(viewport=Viewport.DESKTOP)
+    mobile_extractor = TokenExtractor(viewport=Viewport.MOBILE)
+    
+    desktop_result = await desktop_extractor.extract(pages)
+    mobile_result = await mobile_extractor.extract(pages)
+    
+    return desktop_result, mobile_result

agents/graph.py

@@ -0,0 +1,540 @@
+"""
+LangGraph Workflow Orchestration
+Design System Extractor v2
+
+Defines the main workflow graph with agents, checkpoints, and transitions.
+"""
+
+from typing import Literal
+from datetime import datetime
+from langgraph.graph import StateGraph, END
+from langgraph.checkpoint.memory import MemorySaver
+
+from agents.state import AgentState, create_initial_state, get_stage_progress
+from core.token_schema import Viewport
+
+
+# =============================================================================
+# NODE FUNCTIONS (Agent Entry Points)
+# =============================================================================
+
+async def discover_pages(state: AgentState) -> AgentState:
+    """
+    Agent 1 - Part 1: Discover pages from base URL.
+    
+    This node:
+    1. Takes the base URL
+    2. Crawls to find linked pages
+    3. Classifies page types (homepage, listing, detail, etc.)
+    4. Returns discovered pages for user confirmation
+    """
+    from agents.crawler import PageDiscoverer
+    
+    state["current_stage"] = "discover"
+    state["stage_started_at"] = datetime.now()
+    
+    try:
+        discoverer = PageDiscoverer()
+        pages = await discoverer.discover(state["base_url"])
+        
+        state["discovered_pages"] = pages
+        state["awaiting_human_input"] = True
+        state["checkpoint_name"] = "confirm_pages"
+        
+    except Exception as e:
+        state["errors"].append(f"Discovery failed: {str(e)}")
+    
+    return state
+
+
+async def extract_tokens_desktop(state: AgentState) -> AgentState:
+    """
+    Agent 1 - Part 2a: Extract tokens from desktop viewport.
+    """
+    from agents.extractor import TokenExtractor
+    
+    state["current_stage"] = "extract"
+    
+    try:
+        extractor = TokenExtractor(viewport=Viewport.DESKTOP)
+        result = await extractor.extract(
+            pages=state["pages_to_crawl"],
+            progress_callback=lambda p: state.update({"desktop_crawl_progress": p})
+        )
+        
+        state["desktop_extraction"] = result
+        
+    except Exception as e:
+        state["errors"].append(f"Desktop extraction failed: {str(e)}")
+    
+    return state
+
+
+async def extract_tokens_mobile(state: AgentState) -> AgentState:
+    """
+    Agent 1 - Part 2b: Extract tokens from mobile viewport.
+    """
+    from agents.extractor import TokenExtractor
+    
+    try:
+        extractor = TokenExtractor(viewport=Viewport.MOBILE)
+        result = await extractor.extract(
+            pages=state["pages_to_crawl"],
+            progress_callback=lambda p: state.update({"mobile_crawl_progress": p})
+        )
+        
+        state["mobile_extraction"] = result
+        
+    except Exception as e:
+        state["errors"].append(f"Mobile extraction failed: {str(e)}")
+    
+    return state
+
+
+async def normalize_tokens(state: AgentState) -> AgentState:
+    """
+    Agent 2: Normalize and structure extracted tokens.
+    """
+    from agents.normalizer import TokenNormalizer
+    
+    state["current_stage"] = "normalize"
+    state["stage_started_at"] = datetime.now()
+    
+    try:
+        normalizer = TokenNormalizer()
+        
+        if state["desktop_extraction"]:
+            state["desktop_normalized"] = normalizer.normalize(state["desktop_extraction"])
+        
+        if state["mobile_extraction"]:
+            state["mobile_normalized"] = normalizer.normalize(state["mobile_extraction"])
+        
+        # After normalization, wait for human review
+        state["awaiting_human_input"] = True
+        state["checkpoint_name"] = "review_tokens"
+        
+    except Exception as e:
+        state["errors"].append(f"Normalization failed: {str(e)}")
+    
+    return state
+
+
+async def generate_recommendations(state: AgentState) -> AgentState:
+    """
+    Agent 3: Generate upgrade recommendations.
+    """
+    from agents.advisor import DesignSystemAdvisor
+    
+    state["current_stage"] = "advise"
+    state["stage_started_at"] = datetime.now()
+    
+    try:
+        advisor = DesignSystemAdvisor()
+        recommendations = await advisor.analyze_and_recommend(
+            desktop=state["desktop_normalized"],
+            mobile=state["mobile_normalized"],
+        )
+        
+        state["upgrade_recommendations"] = recommendations
+        
+        # Wait for human to select upgrades
+        state["awaiting_human_input"] = True
+        state["checkpoint_name"] = "select_upgrades"
+        
+    except Exception as e:
+        state["errors"].append(f"Recommendation generation failed: {str(e)}")
+    
+    return state
+
+
+async def generate_final_tokens(state: AgentState) -> AgentState:
+    """
+    Agent 4: Generate final token JSON.
+    """
+    from agents.generator import TokenGenerator
+    
+    state["current_stage"] = "generate"
+    state["stage_started_at"] = datetime.now()
+    
+    try:
+        generator = TokenGenerator()
+        
+        # Build selection config from user choices
+        selections = {
+            "type_scale": state["selected_type_scale"],
+            "spacing_system": state["selected_spacing_system"],
+            "naming_convention": state["selected_naming_convention"],
+            "color_ramps": state["selected_color_ramps"],
+            "a11y_fixes": state["selected_a11y_fixes"],
+        }
+        
+        if state["desktop_normalized"]:
+            state["desktop_final"] = generator.generate(
+                normalized=state["desktop_normalized"],
+                selections=selections,
+                version=state["version_label"],
+            )
+        
+        if state["mobile_normalized"]:
+            state["mobile_final"] = generator.generate(
+                normalized=state["mobile_normalized"],
+                selections=selections,
+                version=state["version_label"],
+            )
+        
+        # Wait for human to approve export
+        state["awaiting_human_input"] = True
+        state["checkpoint_name"] = "approve_export"
+        
+    except Exception as e:
+        state["errors"].append(f"Token generation failed: {str(e)}")
+    
+    return state
+
+
+async def complete_workflow(state: AgentState) -> AgentState:
+    """
+    Final node: Mark workflow as complete.
+    """
+    state["current_stage"] = "export"
+    state["awaiting_human_input"] = False
+    state["checkpoint_name"] = None
+    
+    return state
+
+
+# =============================================================================
+# HUMAN CHECKPOINT HANDLERS
+# =============================================================================
+
+def handle_page_confirmation(state: AgentState, confirmed_pages: list[str]) -> AgentState:
+    """Handle human confirmation of pages to crawl."""
+    state["pages_to_crawl"] = confirmed_pages
+    state["awaiting_human_input"] = False
+    state["checkpoint_name"] = None
+    return state
+
+
+def handle_token_review(
+    state: AgentState,
+    color_decisions: dict[str, bool],
+    typography_decisions: dict[str, bool],
+    spacing_decisions: dict[str, bool],
+) -> AgentState:
+    """Handle human review of extracted tokens."""
+    state["accepted_colors"] = [k for k, v in color_decisions.items() if v]
+    state["rejected_colors"] = [k for k, v in color_decisions.items() if not v]
+    state["accepted_typography"] = [k for k, v in typography_decisions.items() if v]
+    state["rejected_typography"] = [k for k, v in typography_decisions.items() if not v]
+    state["accepted_spacing"] = [k for k, v in spacing_decisions.items() if v]
+    state["rejected_spacing"] = [k for k, v in spacing_decisions.items() if not v]
+    
+    state["awaiting_human_input"] = False
+    state["checkpoint_name"] = None
+    return state
+
+
+def handle_upgrade_selection(
+    state: AgentState,
+    type_scale: str | None,
+    spacing_system: str | None,
+    naming_convention: str | None,
+    color_ramps: dict[str, bool],
+    a11y_fixes: list[str],
+) -> AgentState:
+    """Handle human selection of upgrade options."""
+    state["selected_type_scale"] = type_scale
+    state["selected_spacing_system"] = spacing_system
+    state["selected_naming_convention"] = naming_convention
+    state["selected_color_ramps"] = color_ramps
+    state["selected_a11y_fixes"] = a11y_fixes
+    
+    state["awaiting_human_input"] = False
+    state["checkpoint_name"] = None
+    return state
+
+
+def handle_export_approval(state: AgentState, version_label: str) -> AgentState:
+    """Handle human approval of final export."""
+    state["version_label"] = version_label
+    state["awaiting_human_input"] = False
+    state["checkpoint_name"] = None
+    return state
+
+
+# =============================================================================
+# ROUTING FUNCTIONS
+# =============================================================================
+
+def route_after_discovery(state: AgentState) -> Literal["wait_for_pages", "extract"]:
+    """Route after discovery: wait for human or continue."""
+    if state["awaiting_human_input"]:
+        return "wait_for_pages"
+    return "extract"
+
+
+def route_after_extraction(state: AgentState) -> Literal["normalize", "error"]:
+    """Route after extraction: normalize or handle error."""
+    if state["desktop_extraction"] is None and state["mobile_extraction"] is None:
+        return "error"
+    return "normalize"
+
+
+def route_after_normalization(state: AgentState) -> Literal["wait_for_review", "advise"]:
+    """Route after normalization: wait for review or continue."""
+    if state["awaiting_human_input"]:
+        return "wait_for_review"
+    return "advise"
+
+
+def route_after_recommendations(state: AgentState) -> Literal["wait_for_selection", "generate"]:
+    """Route after recommendations: wait for selection or continue."""
+    if state["awaiting_human_input"]:
+        return "wait_for_selection"
+    return "generate"
+
+
+def route_after_generation(state: AgentState) -> Literal["wait_for_approval", "complete"]:
+    """Route after generation: wait for approval or complete."""
+    if state["awaiting_human_input"]:
+        return "wait_for_approval"
+    return "complete"
+
+
+# =============================================================================
+# GRAPH BUILDER
+# =============================================================================
+
+def build_workflow_graph() -> StateGraph:
+    """
+    Build the main LangGraph workflow.
+    
+    Flow:
+    1. discover_pages -> [human confirms pages]
+    2. extract_desktop + extract_mobile (parallel)
+    3. normalize_tokens -> [human reviews tokens]
+    4. generate_recommendations -> [human selects upgrades]
+    5. generate_final_tokens -> [human approves export]
+    6. complete
+    """
+    
+    # Create the graph
+    workflow = StateGraph(AgentState)
+    
+    # -------------------------------------------------------------------------
+    # ADD NODES
+    # -------------------------------------------------------------------------
+    
+    # Discovery
+    workflow.add_node("discover", discover_pages)
+    
+    # Extraction (will be parallel in subgraph)
+    workflow.add_node("extract_desktop", extract_tokens_desktop)
+    workflow.add_node("extract_mobile", extract_tokens_mobile)
+    
+    # Normalization
+    workflow.add_node("normalize", normalize_tokens)
+    
+    # Advisor
+    workflow.add_node("advise", generate_recommendations)
+    
+    # Generator
+    workflow.add_node("generate", generate_final_tokens)
+    
+    # Completion
+    workflow.add_node("complete", complete_workflow)
+    
+    # Human checkpoint placeholder nodes (these just pass through)
+    workflow.add_node("wait_for_pages", lambda s: s)
+    workflow.add_node("wait_for_review", lambda s: s)
+    workflow.add_node("wait_for_selection", lambda s: s)
+    workflow.add_node("wait_for_approval", lambda s: s)
+    
+    # -------------------------------------------------------------------------
+    # ADD EDGES
+    # -------------------------------------------------------------------------
+    
+    # Entry point
+    workflow.set_entry_point("discover")
+    
+    # Discovery -> (wait or extract)
+    workflow.add_conditional_edges(
+        "discover",
+        route_after_discovery,
+        {
+            "wait_for_pages": "wait_for_pages",
+            "extract": "extract_desktop",
+        }
+    )
+    
+    # After human confirms pages -> extract
+    workflow.add_edge("wait_for_pages", "extract_desktop")
+    
+    # Parallel extraction
+    workflow.add_edge("extract_desktop", "extract_mobile")
+    
+    # After extraction -> normalize
+    workflow.add_conditional_edges(
+        "extract_mobile",
+        route_after_extraction,
+        {
+            "normalize": "normalize",
+            "error": END,
+        }
+    )
+    
+    # Normalization -> (wait or advise)
+    workflow.add_conditional_edges(
+        "normalize",
+        route_after_normalization,
+        {
+            "wait_for_review": "wait_for_review",
+            "advise": "advise",
+        }
+    )
+    
+    # After human reviews -> advise
+    workflow.add_edge("wait_for_review", "advise")
+    
+    # Advisor -> (wait or generate)
+    workflow.add_conditional_edges(
+        "advise",
+        route_after_recommendations,
+        {
+            "wait_for_selection": "wait_for_selection",
+            "generate": "generate",
+        }
+    )
+    
+    # After human selects upgrades -> generate
+    workflow.add_edge("wait_for_selection", "generate")
+    
+    # Generation -> (wait or complete)
+    workflow.add_conditional_edges(
+        "generate",
+        route_after_generation,
+        {
+            "wait_for_approval": "wait_for_approval",
+            "complete": "complete",
+        }
+    )
+    
+    # After human approves -> complete
+    workflow.add_edge("wait_for_approval", "complete")
+    
+    # Complete -> END
+    workflow.add_edge("complete", END)
+    
+    return workflow
+
+
+# =============================================================================
+# WORKFLOW RUNNER
+# =============================================================================
+
+class WorkflowRunner:
+    """
+    Manages workflow execution with human-in-the-loop support.
+    """
+    
+    def __init__(self):
+        self.graph = build_workflow_graph()
+        self.checkpointer = MemorySaver()
+        self.app = self.graph.compile(checkpointer=self.checkpointer)
+        self.current_state: AgentState | None = None
+        self.thread_id: str | None = None
+    
+    async def start(self, base_url: str, thread_id: str | None = None) -> AgentState:
+        """Start a new workflow."""
+        self.thread_id = thread_id or f"workflow_{datetime.now().timestamp()}"
+        self.current_state = create_initial_state(base_url)
+        
+        config = {"configurable": {"thread_id": self.thread_id}}
+        
+        # Run until first human checkpoint
+        async for event in self.app.astream(self.current_state, config):
+            self.current_state = event
+            if self.current_state.get("awaiting_human_input"):
+                break
+        
+        return self.current_state
+    
+    async def resume(self, human_input: dict) -> AgentState:
+        """Resume workflow after human input."""
+        if not self.current_state or not self.thread_id:
+            raise ValueError("No active workflow to resume")
+        
+        checkpoint = self.current_state.get("checkpoint_name")
+        
+        # Apply human input based on checkpoint
+        if checkpoint == "confirm_pages":
+            self.current_state = handle_page_confirmation(
+                self.current_state,
+                human_input.get("confirmed_pages", [])
+            )
+        elif checkpoint == "review_tokens":
+            self.current_state = handle_token_review(
+                self.current_state,
+                human_input.get("color_decisions", {}),
+                human_input.get("typography_decisions", {}),
+                human_input.get("spacing_decisions", {}),
+            )
+        elif checkpoint == "select_upgrades":
+            self.current_state = handle_upgrade_selection(
+                self.current_state,
+                human_input.get("type_scale"),
+                human_input.get("spacing_system"),
+                human_input.get("naming_convention"),
+                human_input.get("color_ramps", {}),
+                human_input.get("a11y_fixes", []),
+            )
+        elif checkpoint == "approve_export":
+            self.current_state = handle_export_approval(
+                self.current_state,
+                human_input.get("version_label", "v1")
+            )
+        
+        config = {"configurable": {"thread_id": self.thread_id}}
+        
+        # Continue until next checkpoint or completion
+        async for event in self.app.astream(self.current_state, config):
+            self.current_state = event
+            if self.current_state.get("awaiting_human_input"):
+                break
+        
+        return self.current_state
+    
+    def get_progress(self) -> dict:
+        """Get current workflow progress."""
+        if not self.current_state:
+            return {"status": "not_started"}
+        return get_stage_progress(self.current_state)
+    
+    def get_state(self) -> AgentState | None:
+        """Get current state."""
+        return self.current_state
+
+
+# =============================================================================
+# CONVENIENCE FUNCTIONS
+# =============================================================================
+
+def create_workflow() -> WorkflowRunner:
+    """Create a new workflow runner instance."""
+    return WorkflowRunner()
+
+
+async def run_discovery_only(base_url: str) -> list:
+    """Run only the discovery phase (for testing)."""
+    from agents.crawler import PageDiscoverer
+    
+    discoverer = PageDiscoverer()
+    return await discoverer.discover(base_url)
+
+
+async def run_extraction_only(pages: list[str], viewport: Viewport) -> dict:
+    """Run only the extraction phase (for testing)."""
+    from agents.extractor import TokenExtractor
+    
+    extractor = TokenExtractor(viewport=viewport)
+    return await extractor.extract(pages)

agents/state.py

@@ -0,0 +1,292 @@
+"""
+LangGraph State Definitions
+Design System Extractor v2
+
+Defines the state schema and type hints for LangGraph workflow.
+"""
+
+from typing import TypedDict, Annotated, Sequence, Optional
+from datetime import datetime
+from langgraph.graph.message import add_messages
+
+from core.token_schema import (
+    DiscoveredPage,
+    ExtractedTokens,
+    NormalizedTokens,
+    UpgradeRecommendations,
+    FinalTokens,
+    Viewport,
+)
+
+
+# =============================================================================
+# STATE ANNOTATIONS
+# =============================================================================
+
+def merge_lists(left: list, right: list) -> list:
+    """Merge two lists, avoiding duplicates."""
+    seen = set()
+    result = []
+    for item in left + right:
+        key = str(item) if not hasattr(item, 'url') else item.url
+        if key not in seen:
+            seen.add(key)
+            result.append(item)
+    return result
+
+
+def replace_value(left, right):
+    """Replace left with right (simple override)."""
+    return right if right is not None else left
+
+
+# =============================================================================
+# MAIN WORKFLOW STATE
+# =============================================================================
+
+class AgentState(TypedDict):
+    """
+    Main state for the LangGraph workflow.
+    
+    This state is passed between all agents and accumulates data
+    as the workflow progresses through stages.
+    """
+    
+    # -------------------------------------------------------------------------
+    # INPUT
+    # -------------------------------------------------------------------------
+    base_url: str  # The website URL to extract from
+    
+    # -------------------------------------------------------------------------
+    # DISCOVERY STAGE (Agent 1 - Part 1)
+    # -------------------------------------------------------------------------
+    discovered_pages: Annotated[list[DiscoveredPage], merge_lists]
+    pages_to_crawl: list[str]  # User-confirmed pages
+    
+    # -------------------------------------------------------------------------
+    # EXTRACTION STAGE (Agent 1 - Part 2)
+    # -------------------------------------------------------------------------
+    # Desktop extraction
+    desktop_extraction: Optional[ExtractedTokens]
+    desktop_crawl_progress: float  # 0.0 to 1.0
+    
+    # Mobile extraction
+    mobile_extraction: Optional[ExtractedTokens]
+    mobile_crawl_progress: float  # 0.0 to 1.0
+    
+    # -------------------------------------------------------------------------
+    # NORMALIZATION STAGE (Agent 2)
+    # -------------------------------------------------------------------------
+    desktop_normalized: Optional[NormalizedTokens]
+    mobile_normalized: Optional[NormalizedTokens]
+    
+    # User decisions from Stage 1 review
+    accepted_colors: list[str]  # List of accepted color values
+    rejected_colors: list[str]  # List of rejected color values
+    accepted_typography: list[str]
+    rejected_typography: list[str]
+    accepted_spacing: list[str]
+    rejected_spacing: list[str]
+    
+    # -------------------------------------------------------------------------
+    # ADVISOR STAGE (Agent 3)
+    # -------------------------------------------------------------------------
+    upgrade_recommendations: Optional[UpgradeRecommendations]
+    
+    # User selections from Stage 2 playground
+    selected_type_scale: Optional[str]  # ID of selected scale
+    selected_spacing_system: Optional[str]
+    selected_naming_convention: Optional[str]
+    selected_color_ramps: dict[str, bool]  # {"primary": True, "secondary": False}
+    selected_a11y_fixes: list[str]  # IDs of accepted fixes
+    
+    # -------------------------------------------------------------------------
+    # GENERATION STAGE (Agent 4)
+    # -------------------------------------------------------------------------
+    desktop_final: Optional[FinalTokens]
+    mobile_final: Optional[FinalTokens]
+    
+    # Version info
+    version_label: str  # e.g., "v1-recovered", "v2-upgraded"
+    
+    # -------------------------------------------------------------------------
+    # WORKFLOW METADATA
+    # -------------------------------------------------------------------------
+    current_stage: str  # "discover", "extract", "normalize", "advise", "generate", "export"
+    
+    # Human checkpoints
+    awaiting_human_input: bool
+    checkpoint_name: Optional[str]  # "confirm_pages", "review_tokens", "select_upgrades", "approve_export"
+    
+    # Errors and warnings (accumulated)
+    errors: Annotated[list[str], merge_lists]
+    warnings: Annotated[list[str], merge_lists]
+    
+    # Messages for LLM agents (if using chat-based agents)
+    messages: Annotated[Sequence[dict], add_messages]
+    
+    # Timing
+    started_at: Optional[datetime]
+    stage_started_at: Optional[datetime]
+
+
+# =============================================================================
+# STAGE-SPECIFIC STATES (for parallel execution)
+# =============================================================================
+
+class DiscoveryState(TypedDict):
+    """State for page discovery sub-graph."""
+    base_url: str
+    discovered_pages: list[DiscoveredPage]
+    discovery_complete: bool
+    error: Optional[str]
+
+
+class ExtractionState(TypedDict):
+    """State for extraction sub-graph (per viewport)."""
+    viewport: Viewport
+    pages_to_crawl: list[str]
+    extraction_result: Optional[ExtractedTokens]
+    progress: float
+    current_page: Optional[str]
+    error: Optional[str]
+
+
+class NormalizationState(TypedDict):
+    """State for normalization sub-graph."""
+    raw_tokens: ExtractedTokens
+    normalized_tokens: Optional[NormalizedTokens]
+    duplicates_found: list[tuple[str, str]]
+    error: Optional[str]
+
+
+class AdvisorState(TypedDict):
+    """State for advisor sub-graph."""
+    normalized_desktop: NormalizedTokens
+    normalized_mobile: Optional[NormalizedTokens]
+    recommendations: Optional[UpgradeRecommendations]
+    error: Optional[str]
+
+
+class GenerationState(TypedDict):
+    """State for generation sub-graph."""
+    normalized_tokens: NormalizedTokens
+    selected_upgrades: dict[str, str]
+    final_tokens: Optional[FinalTokens]
+    error: Optional[str]
+
+
+# =============================================================================
+# CHECKPOINT STATES (Human-in-the-loop)
+# =============================================================================
+
+class PageConfirmationState(TypedDict):
+    """State for page confirmation checkpoint."""
+    discovered_pages: list[DiscoveredPage]
+    confirmed_pages: list[str]
+    user_confirmed: bool
+
+
+class TokenReviewState(TypedDict):
+    """State for token review checkpoint (Stage 1 UI)."""
+    desktop_tokens: NormalizedTokens
+    mobile_tokens: Optional[NormalizedTokens]
+    
+    # User decisions
+    color_decisions: dict[str, bool]  # {value: accepted}
+    typography_decisions: dict[str, bool]
+    spacing_decisions: dict[str, bool]
+    
+    user_confirmed: bool
+
+
+class UpgradeSelectionState(TypedDict):
+    """State for upgrade selection checkpoint (Stage 2 UI)."""
+    recommendations: UpgradeRecommendations
+    current_tokens: NormalizedTokens
+    
+    # User selections
+    selected_options: dict[str, str]  # {category: option_id}
+    
+    user_confirmed: bool
+
+
+class ExportApprovalState(TypedDict):
+    """State for export approval checkpoint (Stage 3 UI)."""
+    desktop_final: FinalTokens
+    mobile_final: Optional[FinalTokens]
+    
+    version_label: str
+    user_confirmed: bool
+
+
+# =============================================================================
+# STATE FACTORY FUNCTIONS
+# =============================================================================
+
+def create_initial_state(base_url: str) -> AgentState:
+    """Create initial state for a new workflow."""
+    return {
+        # Input
+        "base_url": base_url,
+        
+        # Discovery
+        "discovered_pages": [],
+        "pages_to_crawl": [],
+        
+        # Extraction
+        "desktop_extraction": None,
+        "desktop_crawl_progress": 0.0,
+        "mobile_extraction": None,
+        "mobile_crawl_progress": 0.0,
+        
+        # Normalization
+        "desktop_normalized": None,
+        "mobile_normalized": None,
+        "accepted_colors": [],
+        "rejected_colors": [],
+        "accepted_typography": [],
+        "rejected_typography": [],
+        "accepted_spacing": [],
+        "rejected_spacing": [],
+        
+        # Advisor
+        "upgrade_recommendations": None,
+        "selected_type_scale": None,
+        "selected_spacing_system": None,
+        "selected_naming_convention": None,
+        "selected_color_ramps": {},
+        "selected_a11y_fixes": [],
+        
+        # Generation
+        "desktop_final": None,
+        "mobile_final": None,
+        "version_label": "v1-recovered",
+        
+        # Workflow
+        "current_stage": "discover",
+        "awaiting_human_input": False,
+        "checkpoint_name": None,
+        "errors": [],
+        "warnings": [],
+        "messages": [],
+        
+        # Timing
+        "started_at": datetime.now(),
+        "stage_started_at": datetime.now(),
+    }
+
+
+def get_stage_progress(state: AgentState) -> dict:
+    """Get progress information for the current workflow."""
+    stages = ["discover", "extract", "normalize", "advise", "generate", "export"]
+    current_idx = stages.index(state["current_stage"]) if state["current_stage"] in stages else 0
+    
+    return {
+        "current_stage": state["current_stage"],
+        "stage_index": current_idx,
+        "total_stages": len(stages),
+        "progress_percent": (current_idx / len(stages)) * 100,
+        "awaiting_human": state["awaiting_human_input"],
+        "checkpoint": state["checkpoint_name"],
+    }

app.py

@@ -0,0 +1,453 @@
+"""
+Design System Extractor v2 — Main Application
+==============================================
+
+A semi-automated, human-in-the-loop agentic system that reverse-engineers
+design systems from live websites.
+
+Usage:
+    python app.py
+"""
+
+import os
+import asyncio
+import gradio as gr
+from datetime import datetime
+
+# Get HF token from environment if available
+HF_TOKEN_FROM_ENV = os.getenv("HF_TOKEN", "")
+
+# =============================================================================
+# GLOBAL STATE
+# =============================================================================
+
+current_extraction: dict = {}
+user_hf_token: str = ""
+
+
+# =============================================================================
+# HF TOKEN MANAGEMENT
+# =============================================================================
+
+def set_hf_token(token: str) -> str:
+    """Set the HF token globally."""
+    global user_hf_token
+    
+    if not token or len(token) < 10:
+        return "❌ Please enter a valid HuggingFace token"
+    
+    user_hf_token = token.strip()
+    os.environ["HF_TOKEN"] = user_hf_token
+    
+    return "✅ Token saved! You can now use the extractor."
+
+
+# =============================================================================
+# LAZY IMPORTS (avoid circular imports at startup)
+# =============================================================================
+
+_crawler_module = None
+_extractor_module = None
+_schema_module = None
+
+def get_crawler():
+    global _crawler_module
+    if _crawler_module is None:
+        from agents import crawler as _crawler_module
+    return _crawler_module
+
+def get_extractor():
+    global _extractor_module
+    if _extractor_module is None:
+        from agents import extractor as _extractor_module
+    return _extractor_module
+
+def get_schema():
+    global _schema_module
+    if _schema_module is None:
+        from core import token_schema as _schema_module
+    return _schema_module
+
+
+# =============================================================================
+# STAGE 1: URL INPUT & PAGE DISCOVERY
+# =============================================================================
+
+async def discover_site_pages(url: str, progress=gr.Progress()) -> tuple:
+    """
+    Discover pages from a website URL.
+    
+    Returns tuple of (status_message, pages_dataframe, pages_json)
+    """
+    if not url or not url.startswith(("http://", "https://")):
+        return "❌ Please enter a valid URL starting with http:// or https://", None, None
+    
+    progress(0, desc="Initializing browser...")
+    
+    try:
+        crawler = get_crawler()
+        discoverer = crawler.PageDiscoverer()
+        
+        def update_progress(p):
+            progress(p, desc=f"Discovering pages... ({int(p*100)}%)")
+        
+        pages = await discoverer.discover(url, progress_callback=update_progress)
+        
+        # Format for display
+        pages_data = []
+        for page in pages:
+            pages_data.append({
+                "Select": page.selected,
+                "URL": page.url,
+                "Title": page.title or "(No title)",
+                "Type": page.page_type.value,
+                "Status": "✓" if not page.error else f"⚠ {page.error}",
+            })
+        
+        # Store for later use
+        current_extraction["discovered_pages"] = pages
+        current_extraction["base_url"] = url
+        
+        status = f"✅ Found {len(pages)} pages. Select the pages you want to extract tokens from."
+        
+        return status, pages_data, [p.model_dump() for p in pages]
+        
+    except Exception as e:
+        import traceback
+        return f"❌ Error: {str(e)}\n\n{traceback.format_exc()}", None, None
+
+
+async def start_extraction(pages_selection: list, viewport_choice: str, progress=gr.Progress()) -> tuple:
+    """
+    Start token extraction from selected pages.
+    
+    Returns tuple of (status, colors_data, typography_data, spacing_data)
+    """
+    if not pages_selection:
+        return "❌ Please select at least one page", None, None, None
+    
+    # Get selected URLs
+    selected_urls = []
+    for row in pages_selection:
+        if row.get("Select", False):
+            selected_urls.append(row["URL"])
+    
+    if not selected_urls:
+        return "❌ Please select at least one page using the checkboxes", None, None, None
+    
+    # Determine viewport
+    schema = get_schema()
+    viewport = schema.Viewport.DESKTOP if viewport_choice == "Desktop (1440px)" else schema.Viewport.MOBILE
+    
+    progress(0, desc=f"Starting {viewport.value} extraction...")
+    
+    try:
+        extractor_mod = get_extractor()
+        extractor = extractor_mod.TokenExtractor(viewport=viewport)
+        
+        def update_progress(p):
+            progress(p, desc=f"Extracting tokens... ({int(p*100)}%)")
+        
+        result = await extractor.extract(selected_urls, progress_callback=update_progress)
+        
+        # Store result
+        current_extraction[f"{viewport.value}_tokens"] = result
+        
+        # Format colors for display
+        colors_data = []
+        for color in sorted(result.colors, key=lambda c: -c.frequency)[:50]:
+            colors_data.append({
+                "Accept": True,
+                "Color": color.value,
+                "Frequency": color.frequency,
+                "Context": ", ".join(color.contexts[:3]),
+                "Contrast (White)": f"{color.contrast_white}:1",
+                "AA Text": "✓" if color.wcag_aa_small_text else "✗",
+                "Confidence": color.confidence.value,
+            })
+        
+        # Format typography for display
+        typography_data = []
+        for typo in sorted(result.typography, key=lambda t: -t.frequency)[:30]:
+            typography_data.append({
+                "Accept": True,
+                "Font": typo.font_family,
+                "Size": typo.font_size,
+                "Weight": typo.font_weight,
+                "Line Height": typo.line_height,
+                "Elements": ", ".join(typo.elements[:3]),
+                "Frequency": typo.frequency,
+            })
+        
+        # Format spacing for display
+        spacing_data = []
+        for space in sorted(result.spacing, key=lambda s: s.value_px)[:20]:
+            spacing_data.append({
+                "Accept": True,
+                "Value": space.value,
+                "Frequency": space.frequency,
+                "Context": ", ".join(space.contexts[:2]),
+                "Fits 8px": "✓" if space.fits_base_8 else "",
+                "Outlier": "⚠" if space.is_outlier else "",
+            })
+        
+        # Summary
+        status = f"""✅ Extraction Complete ({viewport.value})
+        
+**Summary:**
+- Pages crawled: {len(result.pages_crawled)}
+- Colors found: {len(result.colors)}
+- Typography styles: {len(result.typography)}
+- Spacing values: {len(result.spacing)}
+- Font families: {len(result.font_families)}
+- Detected spacing base: {result.spacing_base or 'Unknown'}px
+- Duration: {result.extraction_duration_ms}ms
+"""
+        
+        if result.warnings:
+            status += f"\n⚠️ Warnings: {len(result.warnings)}"
+        if result.errors:
+            status += f"\n❌ Errors: {len(result.errors)}"
+        
+        return status, colors_data, typography_data, spacing_data
+        
+    except Exception as e:
+        import traceback
+        return f"❌ Extraction failed: {str(e)}\n\n{traceback.format_exc()}", None, None, None
+
+
+def export_tokens_json():
+    """Export current tokens to JSON."""
+    import json
+    
+    result = {}
+    
+    if "desktop_tokens" in current_extraction:
+        desktop = current_extraction["desktop_tokens"]
+        result["desktop"] = {
+            "colors": [c.model_dump() for c in desktop.colors],
+            "typography": [t.model_dump() for t in desktop.typography],
+            "spacing": [s.model_dump() for s in desktop.spacing],
+            "metadata": desktop.summary(),
+        }
+    
+    if "mobile_tokens" in current_extraction:
+        mobile = current_extraction["mobile_tokens"]
+        result["mobile"] = {
+            "colors": [c.model_dump() for c in mobile.colors],
+            "typography": [t.model_dump() for t in mobile.typography],
+            "spacing": [s.model_dump() for s in mobile.spacing],
+            "metadata": mobile.summary(),
+        }
+    
+    if not result:
+        return '{"error": "No tokens extracted yet. Please run extraction first."}'
+    
+    return json.dumps(result, indent=2, default=str)
+
+
+# =============================================================================
+# UI BUILDING
+# =============================================================================
+
+def create_ui():
+    """Create the Gradio interface."""
+    
+    with gr.Blocks(
+        title="Design System Extractor v2",
+        theme=gr.themes.Soft(),
+    ) as app:
+        
+        # Header
+        gr.Markdown("""
+        # 🎨 Design System Extractor v2
+        
+        **Reverse-engineer design systems from live websites.**
+        
+        Extract colors, typography, and spacing tokens from any website and export to Figma-compatible JSON.
+        
+        ---
+        """)
+        
+        # =================================================================
+        # CONFIGURATION SECTION
+        # =================================================================
+        
+        with gr.Accordion("⚙️ Configuration", open=not bool(HF_TOKEN_FROM_ENV)):
+            
+            gr.Markdown("""
+            **HuggingFace Token** is required for AI-powered features (Agent 2-4).
+            Get your token at: [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
+            
+            *Note: Basic extraction (Agent 1) works without a token.*
+            """)
+            
+            with gr.Row():
+                hf_token_input = gr.Textbox(
+                    label="HuggingFace Token",
+                    placeholder="hf_xxxxxxxxxxxxxxxxxxxx",
+                    type="password",
+                    scale=4,
+                    value=HF_TOKEN_FROM_ENV if HF_TOKEN_FROM_ENV else "",
+                )
+                save_token_btn = gr.Button("💾 Save Token", scale=1)
+            
+            token_status = gr.Markdown(
+                "✅ Token loaded from environment" if HF_TOKEN_FROM_ENV else "⏳ Enter your HF token to enable all features"
+            )
+            
+            save_token_btn.click(
+                fn=set_hf_token,
+                inputs=[hf_token_input],
+                outputs=[token_status],
+            )
+        
+        # =================================================================
+        # STAGE 1: URL Input & Discovery
+        # =================================================================
+        
+        with gr.Accordion("📍 Stage 1: Website Discovery", open=True):
+            
+            gr.Markdown("""
+            **Step 1:** Enter your website URL and discover pages.
+            The system will automatically find and classify pages for extraction.
+            """)
+            
+            with gr.Row():
+                url_input = gr.Textbox(
+                    label="Website URL",
+                    placeholder="https://example.com",
+                    scale=4,
+                )
+                discover_btn = gr.Button("🔍 Discover Pages", variant="primary", scale=1)
+            
+            discovery_status = gr.Markdown("")
+            
+            pages_table = gr.Dataframe(
+                headers=["Select", "URL", "Title", "Type", "Status"],
+                datatype=["bool", "str", "str", "str", "str"],
+                interactive=True,
+                label="Discovered Pages",
+                visible=False,
+            )
+            
+            pages_json = gr.JSON(visible=False)
+        
+        # =================================================================
+        # STAGE 2: Extraction
+        # =================================================================
+        
+        with gr.Accordion("🔬 Stage 2: Token Extraction", open=False):
+            
+            gr.Markdown("""
+            **Step 2:** Select pages and viewport, then extract design tokens.
+            """)
+            
+            with gr.Row():
+                viewport_radio = gr.Radio(
+                    choices=["Desktop (1440px)", "Mobile (375px)"],
+                    value="Desktop (1440px)",
+                    label="Viewport",
+                )
+                extract_btn = gr.Button("🚀 Extract Tokens", variant="primary")
+            
+            extraction_status = gr.Markdown("")
+            
+            with gr.Tabs():
+                with gr.Tab("🎨 Colors"):
+                    colors_table = gr.Dataframe(
+                        headers=["Accept", "Color", "Frequency", "Context", "Contrast (White)", "AA Text", "Confidence"],
+                        datatype=["bool", "str", "number", "str", "str", "str", "str"],
+                        interactive=True,
+                        label="Extracted Colors",
+                    )
+                
+                with gr.Tab("📝 Typography"):
+                    typography_table = gr.Dataframe(
+                        headers=["Accept", "Font", "Size", "Weight", "Line Height", "Elements", "Frequency"],
+                        datatype=["bool", "str", "str", "number", "str", "str", "number"],
+                        interactive=True,
+                        label="Extracted Typography",
+                    )
+                
+                with gr.Tab("📏 Spacing"):
+                    spacing_table = gr.Dataframe(
+                        headers=["Accept", "Value", "Frequency", "Context", "Fits 8px", "Outlier"],
+                        datatype=["bool", "str", "number", "str", "str", "str"],
+                        interactive=True,
+                        label="Extracted Spacing",
+                    )
+        
+        # =================================================================
+        # STAGE 3: Export
+        # =================================================================
+        
+        with gr.Accordion("📦 Stage 3: Export", open=False):
+            
+            gr.Markdown("""
+            **Step 3:** Review and export your design tokens.
+            """)
+            
+            with gr.Row():
+                export_btn = gr.Button("📥 Export JSON", variant="secondary")
+                
+            export_output = gr.Code(
+                label="Exported Tokens (JSON)",
+                language="json",
+                lines=20,
+            )
+        
+        # =================================================================
+        # EVENT HANDLERS
+        # =================================================================
+        
+        # Discovery
+        discover_btn.click(
+            fn=discover_site_pages,
+            inputs=[url_input],
+            outputs=[discovery_status, pages_table, pages_json],
+        ).then(
+            fn=lambda: gr.update(visible=True),
+            outputs=[pages_table],
+        )
+        
+        # Extraction
+        extract_btn.click(
+            fn=start_extraction,
+            inputs=[pages_table, viewport_radio],
+            outputs=[extraction_status, colors_table, typography_table, spacing_table],
+        )
+        
+        # Export
+        export_btn.click(
+            fn=export_tokens_json,
+            outputs=[export_output],
+        )
+        
+        # =================================================================
+        # FOOTER
+        # =================================================================
+        
+        gr.Markdown("""
+        ---
+        
+        **Design System Extractor v2** | Built with LangGraph + Gradio + HuggingFace
+        
+        *A semi-automated co-pilot for design system recovery and modernization.*
+        
+        **Models:** Microsoft Phi (Normalizer) • Meta Llama (Advisor) • Mistral Codestral (Generator)
+        """)
+    
+    return app
+
+
+# =============================================================================
+# MAIN
+# =============================================================================
+
+if __name__ == "__main__":
+    app = create_ui()
+    app.launch(
+        server_name="0.0.0.0",
+        server_port=7860,
+    )

config/.env.example

@@ -0,0 +1,143 @@
+# =============================================================================
+# Design System Extractor v2 — Environment Variables
+# =============================================================================
+# Copy this file to .env and fill in your values
+# NEVER commit .env to version control
+# =============================================================================
+
+# -----------------------------------------------------------------------------
+# REQUIRED: Hugging Face Token (Pro recommended for best models)
+# -----------------------------------------------------------------------------
+
+# HuggingFace Token (for Spaces deployment and model access)
+# Get yours at: https://huggingface.co/settings/tokens
+# Pro subscription unlocks: Llama 3.1 405B, Qwen 72B, Command R+, etc.
+HF_TOKEN=your_huggingface_token_here
+
+# HuggingFace Space name (for deployment)
+HF_SPACE_NAME=your-username/design-system-extractor
+
+# -----------------------------------------------------------------------------
+# MODEL CONFIGURATION — Diverse Models for Different Tasks
+# -----------------------------------------------------------------------------
+
+# === Agent 1 (Crawler/Extractor): NO LLM NEEDED ===
+# Pure rule-based extraction using Playwright + CSS parsing
+
+# === Agent 2 (Normalizer): FAST STRUCTURED OUTPUT ===
+# Task: Token naming, duplicate detection, pattern inference
+# Needs: Good instruction following, JSON output, SPEED
+# 
+# Options (pick one):
+#   - microsoft/Phi-3.5-mini-instruct     (Fast, great for structured tasks)
+#   - mistralai/Mistral-7B-Instruct-v0.3  (Fast, good JSON)
+#   - google/gemma-2-9b-it                (Balanced speed/quality)
+#   - Qwen/Qwen2.5-7B-Instruct            (Good all-rounder)
+AGENT2_MODEL=microsoft/Phi-3.5-mini-instruct
+
+# === Agent 3 (Advisor): STRONG REASONING — Most Important! ===
+# Task: Design system analysis, best practice recommendations, trade-off analysis
+# Needs: Deep reasoning, design knowledge, creative suggestions
+#
+# Options (pick one - Pro tier recommended):
+#   - meta-llama/Llama-3.1-70B-Instruct   (Excellent reasoning, long context)
+#   - CohereForAI/c4ai-command-r-plus     (Great for analysis & recommendations)
+#   - Qwen/Qwen2.5-72B-Instruct           (Strong reasoning, good design knowledge)
+#   - mistralai/Mixtral-8x22B-Instruct-v0.1 (Large MoE, good balance)
+#   - meta-llama/Llama-3.1-405B-Instruct  (BEST - if you have Pro++)
+AGENT3_MODEL=meta-llama/Llama-3.1-70B-Instruct
+
+# === Agent 4 (Generator): CODE/JSON SPECIALIST ===
+# Task: Generate Tokens Studio JSON, CSS variables, structured output
+# Needs: Precise formatting, code generation, schema adherence
+#
+# Options (pick one):
+#   - codellama/CodeLlama-34b-Instruct-hf (Code specialist)
+#   - bigcode/starcoder2-15b-instruct-v0.1 (Code generation)
+#   - mistralai/Codestral-22B-v0.1        (Mistral's code model)
+#   - deepseek-ai/deepseek-coder-33b-instruct (Strong code model)
+AGENT4_MODEL=mistralai/Codestral-22B-v0.1
+
+# === Fallback Model (if primary fails) ===
+FALLBACK_MODEL=mistralai/Mistral-7B-Instruct-v0.3
+
+# -----------------------------------------------------------------------------
+# PRESET CONFIGURATIONS
+# -----------------------------------------------------------------------------
+
+# Uncomment ONE preset below, or configure individually above
+
+# --- PRESET: BUDGET (Free tier compatible) ---
+# AGENT2_MODEL=microsoft/Phi-3.5-mini-instruct
+# AGENT3_MODEL=mistralai/Mixtral-8x7B-Instruct-v0.1
+# AGENT4_MODEL=mistralai/Mistral-7B-Instruct-v0.3
+
+# --- PRESET: BALANCED (Pro tier) ---
+# AGENT2_MODEL=google/gemma-2-9b-it
+# AGENT3_MODEL=meta-llama/Llama-3.1-70B-Instruct
+# AGENT4_MODEL=mistralai/Codestral-22B-v0.1
+
+# --- PRESET: MAXIMUM QUALITY (Pro tier) ---
+# AGENT2_MODEL=google/gemma-2-27b-it
+# AGENT3_MODEL=meta-llama/Llama-3.1-405B-Instruct
+# AGENT4_MODEL=deepseek-ai/deepseek-coder-33b-instruct
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: Application Settings
+# -----------------------------------------------------------------------------
+
+DEBUG=false
+LOG_LEVEL=INFO
+MAX_PAGES=20
+MIN_PAGES=10
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: Browser Settings (Playwright)
+# -----------------------------------------------------------------------------
+
+BROWSER_TYPE=chromium
+BROWSER_HEADLESS=true
+BROWSER_TIMEOUT=30000
+NETWORK_IDLE_TIMEOUT=5000
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: Storage Settings
+# -----------------------------------------------------------------------------
+
+STORAGE_PATH=/data
+ENABLE_PERSISTENCE=true
+MAX_VERSIONS=10
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: Rate Limiting
+# -----------------------------------------------------------------------------
+
+CRAWL_DELAY_MS=1000
+MAX_CONCURRENT_CRAWLS=3
+RESPECT_ROBOTS_TXT=true
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: HuggingFace Inference Settings
+# -----------------------------------------------------------------------------
+
+USE_HF_INFERENCE_API=true
+HF_INFERENCE_TIMEOUT=120
+HF_MAX_NEW_TOKENS=2048
+HF_TEMPERATURE=0.3
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: UI Settings
+# -----------------------------------------------------------------------------
+
+SERVER_PORT=7860
+SHARE=false
+UI_THEME=soft
+
+# -----------------------------------------------------------------------------
+# OPTIONAL: Feature Flags
+# -----------------------------------------------------------------------------
+
+FEATURE_COLOR_RAMPS=true
+FEATURE_TYPE_SCALES=true
+FEATURE_A11Y_CHECKS=true
+FEATURE_PARALLEL_EXTRACTION=true

config/agents.yaml

@@ -0,0 +1,547 @@
+# Agent Personas & Configuration
+# Design System Extractor v2
+
+# =============================================================================
+# MODEL CONFIGURATION
+# =============================================================================
+# Model assignments for each agent based on task complexity
+# =============================================================================
+
+models:
+  # Agent 1: Crawler & Extractor — NO LLM NEEDED (Rule-based)
+  agent_crawler:
+    requires_llm: false
+    notes: "Pure rule-based extraction using Playwright + CSS parsing"
+  
+  # Agent 2: Normalizer — LIGHT MODEL
+  agent_normalizer:
+    requires_llm: true
+    primary_model: "Qwen/Qwen2.5-7B-Instruct"
+    fallback_model: "microsoft/Phi-3-mini-4k-instruct"
+    tasks:
+      - "Suggest semantic token names from values"
+      - "Explain duplicate detection rationale"
+      - "Infer naming conventions from class names"
+    temperature: 0.2  # Low temp for consistent naming
+    max_tokens: 1024
+    why_this_model: |
+      7B model is sufficient because:
+      - Tasks are mostly pattern matching and naming
+      - Doesn't require complex reasoning
+      - Fast inference, good for iterative workflows
+  
+  # Agent 3: Advisor — STRONG MODEL (Key agent)
+  agent_advisor:
+    requires_llm: true
+    primary_model: "Qwen/Qwen2.5-72B-Instruct"
+    fallback_model: "mistralai/Mixtral-8x7B-Instruct-v0.1"
+    budget_model: "Qwen/Qwen2.5-7B-Instruct"  # If quota constrained
+    tasks:
+      - "Analyze extracted system for patterns and anti-patterns"
+      - "Research and recommend type scales with rationale"
+      - "Suggest spacing systems based on detected patterns"
+      - "Identify accessibility issues and fixes"
+      - "Compare against industry design systems"
+    temperature: 0.4  # Moderate temp for creative recommendations
+    max_tokens: 4096
+    why_this_model: |
+      72B model is critical because:
+      - Needs deep understanding of design systems
+      - Must reason about trade-offs (pros/cons)
+      - Should know Material, Polaris, Carbon, etc.
+      - Quality of recommendations is user-facing
+  
+  # Agent 4: Generator — LIGHT MODEL
+  agent_generator:
+    requires_llm: true
+    primary_model: "Qwen/Qwen2.5-7B-Instruct"
+    fallback_model: "mistralai/Mistral-7B-Instruct-v0.3"
+    tasks:
+      - "Format tokens into Tokens Studio JSON"
+      - "Generate CSS custom properties"
+      - "Structure metadata correctly"
+    temperature: 0.1  # Very low temp for consistent formatting
+    max_tokens: 8192  # Larger for full JSON output
+    why_this_model: |
+      7B model is sufficient because:
+      - Structured output generation is formulaic
+      - JSON schema is well-defined
+      - Speed matters for export flow
+
+# =============================================================================
+# FREE TIER vs PRO RECOMMENDATIONS
+# =============================================================================
+
+tier_recommendations:
+  free_tier:
+    description: "For users without HF Pro subscription"
+    agent2: "Qwen/Qwen2.5-7B-Instruct"
+    agent3: "mistralai/Mixtral-8x7B-Instruct-v0.1"  # Best free option for reasoning
+    agent4: "Qwen/Qwen2.5-7B-Instruct"
+    notes: "Quality will be slightly lower for Agent 3 recommendations"
+  
+  pro_tier:
+    description: "For users with HF Pro subscription"
+    agent2: "Qwen/Qwen2.5-7B-Instruct"
+    agent3: "Qwen/Qwen2.5-72B-Instruct"  # Full quality
+    agent4: "Qwen/Qwen2.5-7B-Instruct"
+    notes: "Best quality, especially for design system recommendations"
+
+# =============================================================================
+# AGENT 1: Website Crawler & Extractor
+# =============================================================================
+agent_crawler:
+  name: "Design Archaeologist"
+  persona: |
+    You are a meticulous Design Archaeologist. Your job is to carefully excavate 
+    design decisions buried in website code. You approach each site with curiosity 
+    and systematic precision, documenting everything you find without judgment.
+    
+    You understand that live websites are often messy — accumulated decisions from 
+    multiple designers over years. Your role is to faithfully extract what exists, 
+    not to fix or improve it yet.
+  
+  responsibilities:
+    - Auto-discover pages from base URL (minimum 10)
+    - Include key templates: homepage, listing, detail, form, marketing, auth
+    - Scroll pages fully (above + below fold)
+    - Extract separately for Desktop (1440px) and Mobile (375px)
+  
+  extraction_targets:
+    colors:
+      - hex values
+      - rgb/rgba values
+      - usage frequency
+      - context (background, text, border, etc.)
+    typography:
+      - font families
+      - font sizes (px, rem, em)
+      - line heights
+      - font weights
+      - letter spacing
+    spacing:
+      - margin values
+      - padding values
+      - gap values
+      - infer base system (4px, 8px)
+    other:
+      - border radius
+      - box shadows
+      - layout signals (containers, grids)
+  
+  tools:
+    - playwright (crawling, scrolling, computed styles)
+    - css parsing (CSSOM)
+  
+  output_format:
+    - raw extracted tokens
+    - confidence score per token
+    - desktop/mobile separated
+    - errors and warnings logged
+
+  guardrails:
+    - Never modify or "fix" extracted values
+    - Always preserve original CSS values
+    - Log anomalies, don't hide them
+    - Respect robots.txt (configurable)
+
+# =============================================================================
+# AGENT 2: Token Normalizer & Structurer
+# =============================================================================
+agent_normalizer:
+  name: "Design System Librarian"
+  persona: |
+    You are a methodical Design System Librarian. Your expertise is in organizing 
+    chaotic information into structured, meaningful categories. You see patterns 
+    where others see noise.
+    
+    You are careful to distinguish between what you KNOW (detected) and what you 
+    THINK (inferred). You never overwrite the truth — you annotate it.
+  
+  responsibilities:
+    - Clean noisy extraction data
+    - Group and merge duplicates (with threshold tolerance)
+    - Infer naming patterns from class names
+    - Propose initial token names
+    - Tag confidence levels
+  
+  naming_conventions:
+    colors:
+      pattern: "color.{role}.{shade}"
+      example: "color.primary.500"
+      roles: 
+        - primary
+        - secondary
+        - neutral
+        - success
+        - warning
+        - error
+      shades:
+        - 50
+        - 100
+        - 200
+        - 300
+        - 400
+        - 500
+        - 600
+        - 700
+        - 800
+        - 900
+    typography:
+      pattern: "font.{category}.{size}"
+      example: "font.heading.lg"
+      categories:
+        - heading
+        - body
+        - label
+        - caption
+      sizes:
+        - xs
+        - sm
+        - md
+        - lg
+        - xl
+        - 2xl
+    spacing:
+      pattern: "space.{size}"
+      example: "space.4, space.8"
+      note: "based on pixel value / 4"
+  
+  tagging:
+    detected: "Directly found in CSS, high confidence"
+    inferred: "Derived from patterns, medium confidence"
+    low_confidence: "Appears rarely or inconsistently"
+  
+  duplicate_threshold:
+    colors: 3  # hex values within 3 steps are potential duplicates
+    spacing: 2  # pixel values within 2px are potential duplicates
+  
+  guardrails:
+    - Never overwrite extracted truth
+    - Always mark inferred vs detected
+    - Preserve original values alongside normalized
+    - Flag conflicts, don't resolve them silently
+
+# =============================================================================
+# AGENT 3: Design System Best Practices Advisor
+# =============================================================================
+agent_advisor:
+  name: "Senior Staff Design Systems Architect"
+  persona: |
+    You are a Senior Staff Design Systems Architect with 15+ years of experience 
+    building and scaling design systems at major tech companies. You've seen what 
+    works and what doesn't.
+    
+    Your role is to ADVISE, not DECIDE. You present options with clear rationale, 
+    letting humans make the final call. You respect existing decisions while 
+    offering paths to improvement.
+  
+  responsibilities:
+    - Analyze extracted system for patterns and anti-patterns
+    - Research modern design system best practices
+    - Propose upgrade OPTIONS (never auto-apply)
+    - Ensure accessibility compliance (AA minimum)
+  
+  research_sources:
+    - Material Design (Google)
+    - Polaris (Shopify)
+    - Carbon (IBM)
+    - Fluent (Microsoft)
+    - Primer (GitHub)
+    - Radix
+    - Tailwind CSS
+  
+  upgrade_categories:
+    typography_scales:
+      - name: "Minor Third"
+        ratio: 1.2
+        description: "Subtle progression, good for dense UIs"
+      - name: "Major Third"
+        ratio: 1.25
+        description: "Balanced, most popular choice"
+      - name: "Perfect Fourth"
+        ratio: 1.333
+        description: "Strong hierarchy, good for marketing"
+      - name: "Golden Ratio"
+        ratio: 1.618
+        description: "Dramatic, use sparingly"
+    
+    spacing_systems:
+      - name: "4px base"
+        scale: [4, 8, 12, 16, 20, 24, 32, 40, 48, 64]
+        description: "Fine-grained control"
+      - name: "8px base"
+        scale: [4, 8, 16, 24, 32, 48, 64, 80, 96]
+        description: "Industry standard, recommended"
+      - name: "Tailwind"
+        scale: [4, 8, 12, 16, 20, 24, 32, 40, 48, 56, 64, 80, 96]
+        description: "Utility-first, comprehensive"
+    
+    naming_conventions:
+      - name: "T-shirt sizes"
+        example: "xs, sm, md, lg, xl, 2xl"
+        pros: "Intuitive, easy to remember"
+        cons: "Limited granularity"
+      - name: "Numeric"
+        example: "100, 200, 300, 400, 500"
+        pros: "Extensible, precise"
+        cons: "Less intuitive"
+      - name: "Semantic"
+        example: "caption, body, subhead, title, display"
+        pros: "Meaningful, self-documenting"
+        cons: "Harder to extend"
+    
+    color_ramps:
+      shades: [50, 100, 200, 300, 400, 500, 600, 700, 800, 900]
+      aa_contrast_minimum: 4.5
+      aaa_contrast_minimum: 7.0
+  
+  output_format:
+    - option sets (never single recommendations)
+    - rationale for each option
+    - pros and cons
+    - accessibility impact
+    - migration effort estimate
+  
+  guardrails:
+    - Never auto-apply changes
+    - Always provide multiple options
+    - Respect existing system, suggest improvements
+    - Flag accessibility issues prominently
+
+# =============================================================================
+# AGENT 4: Plugin & JSON Generator
+# =============================================================================
+agent_generator:
+  name: "Automation Engineer"
+  persona: |
+    You are a precise Automation Engineer. Your job is to transform design 
+    decisions into machine-readable formats that tools can consume. You care 
+    deeply about compatibility, versioning, and clean output.
+    
+    You understand that your output will be used by Figma plugins, CSS 
+    preprocessors, and design tools — so format matters.
+  
+  responsibilities:
+    - Convert finalized tokens to standard formats
+    - Generate color ramps with AA compliance
+    - Maintain viewport separation (Desktop/Mobile)
+    - Version all outputs
+  
+  output_formats:
+    tokens_studio:
+      description: "Compatible with Tokens Studio Figma plugin"
+      extension: ".json"
+    figma_variables:
+      description: "Direct Figma Variables format"
+      extension: ".json"
+    css_variables:
+      description: "CSS custom properties"
+      extension: ".css"
+    tailwind_config:
+      description: "Tailwind CSS configuration"
+      extension: ".js"
+  
+  token_structure:
+    colors:
+      include_ramps: true
+      ramp_shades: [50, 100, 200, 300, 400, 500, 600, 700, 800, 900]
+      include_contrast: true
+    typography:
+      include_composite: true  # font-family + size + weight + line-height
+      include_individual: true
+    spacing:
+      base: 8
+      include_negative: false
+  
+  metadata_fields:
+    - source_url
+    - extracted_at
+    - version
+    - viewport
+    - token_source (detected/inferred/upgraded)
+  
+  guardrails:
+    - Always include metadata
+    - Validate JSON before output
+    - Preserve source attribution on each token
+    - Warn on potential conflicts
+
+# =============================================================================
+# LANGGRAPH WORKFLOW CONFIGURATION
+# =============================================================================
+workflow:
+  name: "design_system_extraction"
+  
+  checkpoints:
+    - id: "confirm_pages"
+      description: "Human confirms discovered pages before crawling"
+      required: true
+    - id: "review_extraction"
+      description: "Human reviews extracted tokens (Stage 1 UI)"
+      required: true
+    - id: "select_upgrades"
+      description: "Human selects upgrade options (Stage 2 UI)"
+      required: true
+    - id: "approve_export"
+      description: "Human approves final output (Stage 3 UI)"
+      required: true
+  
+  parallel_nodes:
+    - name: "viewport_extraction"
+      description: "Extract Desktop and Mobile in parallel"
+      agents: ["agent_crawler"]
+    - name: "research_and_advise"
+      description: "Agent 3 can research while human reviews Stage 1"
+      agents: ["agent_advisor"]
+  
+  error_handling:
+    retry_attempts: 3
+    retry_delay_seconds: 5
+    log_errors: true
+    show_in_ui: true
+
+# =============================================================================
+# UI CONFIGURATION
+# =============================================================================
+ui:
+  layout: "single_scroll"
+  
+  stages:
+    stage1:
+      name: "Extraction Review"
+      purpose: "Trust building — see what was found"
+      components:
+        - token_tables
+        - color_swatches
+        - typography_samples
+        - viewport_toggle
+        - confidence_indicators
+        - accept_reject_controls
+    
+    stage2:
+      name: "Upgrade Playground"
+      purpose: "Decision making through live visuals"
+      components:
+        - option_selector
+        - live_preview_iframe
+        - side_by_side_comparison
+        - existing_vs_upgraded_toggle
+    
+    stage3:
+      name: "Final Review & Export"
+      purpose: "Confidence before export"
+      components:
+        - token_preview_readonly
+        - json_tree_view
+        - diff_view
+        - viewport_tabs
+        - download_buttons
+        - version_labeling
+  
+  preview:
+    type: "iframe"
+    template: "specimen.html"
+    update_trigger: "on_selection"
+
+# =============================================================================
+# EXTRACTION SETTINGS
+# =============================================================================
+extraction:
+  viewports:
+    desktop:
+      width: 1440
+      height: 900
+      name: "Desktop"
+    mobile:
+      width: 375
+      height: 812
+      name: "Mobile"
+  
+  crawling:
+    max_pages: 20
+    min_pages: 10
+    scroll_behavior: "smooth"
+    wait_for_network_idle: true
+    network_idle_timeout_ms: 5000
+    skip_infinite_scroll: true
+    respect_robots_txt: true
+  
+  page_templates:
+    required:
+      - homepage
+      - listing
+      - detail
+    optional:
+      - form
+      - marketing
+      - auth
+      - checkout
+      - about
+      - contact
+
+# =============================================================================
+# COLOR PROCESSING
+# =============================================================================
+color_processing:
+  ramp_generation:
+    enabled: true
+    shades: [50, 100, 200, 300, 400, 500, 600, 700, 800, 900]
+    method: "oklch"  # or "hsl" or "lab"
+  
+  accessibility:
+    check_contrast: true
+    minimum_standard: "AA"  # or "AAA"
+    contrast_pairs:
+      - ["text", "background"]
+      - ["button-text", "button-background"]
+  
+  duplicate_detection:
+    enabled: true
+    threshold_delta_e: 3  # CIE Delta E threshold
+
+# =============================================================================
+# TYPOGRAPHY PROCESSING
+# =============================================================================
+typography_processing:
+  scale_options:
+    - name: "Minor Third"
+      ratio: 1.2
+    - name: "Major Third"
+      ratio: 1.25
+    - name: "Perfect Fourth"
+      ratio: 1.333
+  
+  base_size: 16  # px
+  
+  text_styles:
+    - display
+    - heading-xl
+    - heading-lg
+    - heading-md
+    - heading-sm
+    - body-lg
+    - body-md
+    - body-sm
+    - caption
+    - label
+
+# =============================================================================
+# SPACING PROCESSING
+# =============================================================================
+spacing_processing:
+  base: 8  # px
+  scale: [0, 4, 8, 12, 16, 24, 32, 48, 64, 80, 96]
+  
+  names:
+    0: "none"
+    4: "xs"
+    8: "sm"
+    12: "sm-md"
+    16: "md"
+    24: "lg"
+    32: "xl"
+    48: "2xl"
+    64: "3xl"
+    80: "4xl"
+    96: "5xl"

config/settings.py

@@ -0,0 +1,217 @@
+"""
+Application Settings
+Design System Extractor v2
+
+Loads configuration from environment variables and YAML files.
+"""
+
+import os
+from pathlib import Path
+from typing import Optional
+from dataclasses import dataclass, field
+from dotenv import load_dotenv
+import yaml
+
+# Load environment variables from .env file
+env_path = Path(__file__).parent / ".env"
+if env_path.exists():
+    load_dotenv(env_path)
+else:
+    # Try loading from parent directory (for development)
+    load_dotenv(Path(__file__).parent.parent / ".env")
+
+
+@dataclass
+class HFSettings:
+    """Hugging Face configuration."""
+    hf_token: str = field(default_factory=lambda: os.getenv("HF_TOKEN", ""))
+    hf_space_name: str = field(default_factory=lambda: os.getenv("HF_SPACE_NAME", ""))
+    use_inference_api: bool = field(default_factory=lambda: os.getenv("USE_HF_INFERENCE_API", "true").lower() == "true")
+    inference_timeout: int = field(default_factory=lambda: int(os.getenv("HF_INFERENCE_TIMEOUT", "120")))
+    max_new_tokens: int = field(default_factory=lambda: int(os.getenv("HF_MAX_NEW_TOKENS", "2048")))
+    temperature: float = field(default_factory=lambda: float(os.getenv("HF_TEMPERATURE", "0.3")))
+
+
+@dataclass
+class ModelSettings:
+    """Model configuration for each agent — Diverse providers."""
+    # Agent 1: Rule-based, no LLM needed
+    
+    # Agent 2 (Normalizer): Fast structured output
+    # Default: Microsoft Phi (fast, great structured output)
+    agent2_model: str = field(default_factory=lambda: os.getenv("AGENT2_MODEL", "microsoft/Phi-3.5-mini-instruct"))
+    
+    # Agent 3 (Advisor): Strong reasoning - MOST IMPORTANT
+    # Default: Meta Llama 70B (excellent reasoning)
+    agent3_model: str = field(default_factory=lambda: os.getenv("AGENT3_MODEL", "meta-llama/Llama-3.1-70B-Instruct"))
+    
+    # Agent 4 (Generator): Code/JSON specialist
+    # Default: Mistral Codestral (code specialist)
+    agent4_model: str = field(default_factory=lambda: os.getenv("AGENT4_MODEL", "mistralai/Codestral-22B-v0.1"))
+    
+    # Fallback
+    fallback_model: str = field(default_factory=lambda: os.getenv("FALLBACK_MODEL", "mistralai/Mistral-7B-Instruct-v0.3"))
+
+
+@dataclass
+class APISettings:
+    """API key configuration (optional alternatives)."""
+    anthropic_api_key: str = field(default_factory=lambda: os.getenv("ANTHROPIC_API_KEY", ""))
+    openai_api_key: str = field(default_factory=lambda: os.getenv("OPENAI_API_KEY", ""))
+
+
+@dataclass
+class BrowserSettings:
+    """Playwright browser configuration."""
+    browser_type: str = field(default_factory=lambda: os.getenv("BROWSER_TYPE", "chromium"))
+    headless: bool = field(default_factory=lambda: os.getenv("BROWSER_HEADLESS", "true").lower() == "true")
+    timeout: int = field(default_factory=lambda: int(os.getenv("BROWSER_TIMEOUT", "30000")))
+    network_idle_timeout: int = field(default_factory=lambda: int(os.getenv("NETWORK_IDLE_TIMEOUT", "5000")))
+
+
+@dataclass
+class CrawlSettings:
+    """Website crawling configuration."""
+    max_pages: int = field(default_factory=lambda: int(os.getenv("MAX_PAGES", "20")))
+    min_pages: int = field(default_factory=lambda: int(os.getenv("MIN_PAGES", "10")))
+    crawl_delay_ms: int = field(default_factory=lambda: int(os.getenv("CRAWL_DELAY_MS", "1000")))
+    max_concurrent: int = field(default_factory=lambda: int(os.getenv("MAX_CONCURRENT_CRAWLS", "3")))
+    respect_robots_txt: bool = field(default_factory=lambda: os.getenv("RESPECT_ROBOTS_TXT", "true").lower() == "true")
+
+
+@dataclass
+class ViewportSettings:
+    """Viewport configuration for extraction."""
+    desktop_width: int = 1440
+    desktop_height: int = 900
+    mobile_width: int = 375
+    mobile_height: int = 812
+
+
+@dataclass
+class StorageSettings:
+    """Persistent storage configuration."""
+    storage_path: str = field(default_factory=lambda: os.getenv("STORAGE_PATH", "/data"))
+    enable_persistence: bool = field(default_factory=lambda: os.getenv("ENABLE_PERSISTENCE", "true").lower() == "true")
+    max_versions: int = field(default_factory=lambda: int(os.getenv("MAX_VERSIONS", "10")))
+
+
+@dataclass
+class UISettings:
+    """UI configuration."""
+    server_port: int = field(default_factory=lambda: int(os.getenv("SERVER_PORT", "7860")))
+    share: bool = field(default_factory=lambda: os.getenv("SHARE", "false").lower() == "true")
+    theme: str = field(default_factory=lambda: os.getenv("UI_THEME", "soft"))
+
+
+@dataclass
+class FeatureFlags:
+    """Feature toggles."""
+    color_ramps: bool = field(default_factory=lambda: os.getenv("FEATURE_COLOR_RAMPS", "true").lower() == "true")
+    type_scales: bool = field(default_factory=lambda: os.getenv("FEATURE_TYPE_SCALES", "true").lower() == "true")
+    a11y_checks: bool = field(default_factory=lambda: os.getenv("FEATURE_A11Y_CHECKS", "true").lower() == "true")
+    parallel_extraction: bool = field(default_factory=lambda: os.getenv("FEATURE_PARALLEL_EXTRACTION", "true").lower() == "true")
+
+
+@dataclass
+class Settings:
+    """Main settings container."""
+    debug: bool = field(default_factory=lambda: os.getenv("DEBUG", "false").lower() == "true")
+    log_level: str = field(default_factory=lambda: os.getenv("LOG_LEVEL", "INFO"))
+    
+    hf: HFSettings = field(default_factory=HFSettings)
+    models: ModelSettings = field(default_factory=ModelSettings)
+    api: APISettings = field(default_factory=APISettings)
+    browser: BrowserSettings = field(default_factory=BrowserSettings)
+    crawl: CrawlSettings = field(default_factory=CrawlSettings)
+    viewport: ViewportSettings = field(default_factory=ViewportSettings)
+    storage: StorageSettings = field(default_factory=StorageSettings)
+    ui: UISettings = field(default_factory=UISettings)
+    features: FeatureFlags = field(default_factory=FeatureFlags)
+    
+    # Agent configuration loaded from YAML
+    agents_config: dict = field(default_factory=dict)
+    
+    def __post_init__(self):
+        """Load agent configuration from YAML after initialization."""
+        self.load_agents_config()
+    
+    def load_agents_config(self):
+        """Load agent personas and settings from YAML file."""
+        yaml_path = Path(__file__).parent / "agents.yaml"
+        if yaml_path.exists():
+            with open(yaml_path, "r") as f:
+                self.agents_config = yaml.safe_load(f)
+        else:
+            print(f"Warning: agents.yaml not found at {yaml_path}")
+            self.agents_config = {}
+    
+    def get_agent_persona(self, agent_name: str) -> str:
+        """Get persona string for an agent."""
+        agent_key = f"agent_{agent_name}"
+        if agent_key in self.agents_config:
+            return self.agents_config[agent_key].get("persona", "")
+        return ""
+    
+    def get_agent_config(self, agent_name: str) -> dict:
+        """Get full configuration for an agent."""
+        agent_key = f"agent_{agent_name}"
+        return self.agents_config.get(agent_key, {})
+    
+    def get_model_for_agent(self, agent_name: str) -> str:
+        """Get the model ID for a specific agent."""
+        model_map = {
+            "normalizer": self.models.agent2_model,
+            "advisor": self.models.agent3_model,
+            "generator": self.models.agent4_model,
+        }
+        return model_map.get(agent_name, self.models.fallback_model)
+    
+    def validate(self) -> list[str]:
+        """Validate settings and return list of errors."""
+        errors = []
+        
+        if not self.hf.hf_token:
+            errors.append("HF_TOKEN is required for model inference")
+        
+        if self.crawl.max_pages < self.crawl.min_pages:
+            errors.append("MAX_PAGES must be >= MIN_PAGES")
+        
+        return errors
+
+
+# Global settings instance
+settings = Settings()
+
+
+def get_settings() -> Settings:
+    """Get the global settings instance."""
+    return settings
+
+
+def reload_settings() -> Settings:
+    """Reload settings from environment and config files."""
+    global settings
+    settings = Settings()
+    return settings
+
+
+# Convenience functions
+def is_debug() -> bool:
+    """Check if debug mode is enabled."""
+    return settings.debug
+
+
+def get_hf_token() -> str:
+    """Get HuggingFace token."""
+    return settings.hf.hf_token
+
+
+def get_agent_persona(agent_name: str) -> str:
+    """Get persona for an agent."""
+    return settings.get_agent_persona(agent_name)
+
+
+def get_model_for_agent(agent_name: str) -> str:
+    """Get model ID for an agent."""
+    return settings.get_model_for_agent(agent_name)

core/__init__.py

@@ -0,0 +1,61 @@
+"""
+Core utilities for Design System Extractor v2.
+"""
+
+from core.token_schema import (
+    TokenSource,
+    Confidence,
+    Viewport,
+    PageType,
+    ColorToken,
+    TypographyToken,
+    SpacingToken,
+    RadiusToken,
+    ShadowToken,
+    ExtractedTokens,
+    NormalizedTokens,
+    FinalTokens,
+    WorkflowState,
+)
+
+from core.color_utils import (
+    parse_color,
+    normalize_hex,
+    get_contrast_ratio,
+    check_wcag_compliance,
+    generate_color_ramp,
+    generate_accessible_ramp,
+    categorize_color,
+    suggest_color_name,
+)
+
+# HF Inference is imported lazily to avoid circular imports
+# Use: from core.hf_inference import get_inference_client
+
+__all__ = [
+    # Enums
+    "TokenSource",
+    "Confidence",
+    "Viewport",
+    "PageType",
+    # Token models
+    "ColorToken",
+    "TypographyToken",
+    "SpacingToken",
+    "RadiusToken",
+    "ShadowToken",
+    # Result models
+    "ExtractedTokens",
+    "NormalizedTokens",
+    "FinalTokens",
+    "WorkflowState",
+    # Color utilities
+    "parse_color",
+    "normalize_hex",
+    "get_contrast_ratio",
+    "check_wcag_compliance",
+    "generate_color_ramp",
+    "generate_accessible_ramp",
+    "categorize_color",
+    "suggest_color_name",
+]

core/color_utils.py

@@ -0,0 +1,462 @@
+"""
+Color Utilities
+Design System Extractor v2
+
+Functions for color analysis, contrast calculation, and ramp generation.
+"""
+
+import re
+import colorsys
+from typing import Optional
+from dataclasses import dataclass
+
+
+# =============================================================================
+# COLOR PARSING
+# =============================================================================
+
+@dataclass
+class ParsedColor:
+    """Parsed color with multiple representations."""
+    hex: str
+    rgb: tuple[int, int, int]
+    hsl: tuple[float, float, float]
+    oklch: Optional[tuple[float, float, float]] = None
+
+
+def hex_to_rgb(hex_color: str) -> tuple[int, int, int]:
+    """Convert hex color to RGB tuple."""
+    hex_color = hex_color.lstrip("#")
+    if len(hex_color) == 3:
+        hex_color = "".join([c * 2 for c in hex_color])
+    return tuple(int(hex_color[i:i+2], 16) for i in (0, 2, 4))
+
+
+def rgb_to_hex(r: int, g: int, b: int) -> str:
+    """Convert RGB to hex color."""
+    return f"#{r:02x}{g:02x}{b:02x}"
+
+
+def rgb_to_hsl(r: int, g: int, b: int) -> tuple[float, float, float]:
+    """Convert RGB to HSL."""
+    r_norm, g_norm, b_norm = r / 255, g / 255, b / 255
+    h, l, s = colorsys.rgb_to_hls(r_norm, g_norm, b_norm)
+    return (h * 360, s * 100, l * 100)
+
+
+def hsl_to_rgb(h: float, s: float, l: float) -> tuple[int, int, int]:
+    """Convert HSL to RGB."""
+    h_norm, s_norm, l_norm = h / 360, s / 100, l / 100
+    r, g, b = colorsys.hls_to_rgb(h_norm, l_norm, s_norm)
+    return (int(r * 255), int(g * 255), int(b * 255))
+
+
+def parse_color(color_string: str) -> Optional[ParsedColor]:
+    """
+    Parse any CSS color format to ParsedColor.
+    
+    Supports:
+    - Hex: #fff, #ffffff
+    - RGB: rgb(255, 255, 255)
+    - RGBA: rgba(255, 255, 255, 0.5)
+    - HSL: hsl(0, 100%, 50%)
+    """
+    color_string = color_string.strip().lower()
+    
+    # Hex format
+    if color_string.startswith("#"):
+        hex_color = color_string
+        if len(hex_color) == 4:
+            hex_color = f"#{hex_color[1]*2}{hex_color[2]*2}{hex_color[3]*2}"
+        try:
+            rgb = hex_to_rgb(hex_color)
+            hsl = rgb_to_hsl(*rgb)
+            return ParsedColor(hex=hex_color, rgb=rgb, hsl=hsl)
+        except ValueError:
+            return None
+    
+    # RGB/RGBA format
+    rgb_match = re.match(r"rgba?\s*\(\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)", color_string)
+    if rgb_match:
+        r, g, b = int(rgb_match.group(1)), int(rgb_match.group(2)), int(rgb_match.group(3))
+        hex_color = rgb_to_hex(r, g, b)
+        hsl = rgb_to_hsl(r, g, b)
+        return ParsedColor(hex=hex_color, rgb=(r, g, b), hsl=hsl)
+    
+    # HSL format
+    hsl_match = re.match(r"hsl\s*\(\s*(\d+)\s*,\s*(\d+)%?\s*,\s*(\d+)%?", color_string)
+    if hsl_match:
+        h, s, l = float(hsl_match.group(1)), float(hsl_match.group(2)), float(hsl_match.group(3))
+        rgb = hsl_to_rgb(h, s, l)
+        hex_color = rgb_to_hex(*rgb)
+        return ParsedColor(hex=hex_color, rgb=rgb, hsl=(h, s, l))
+    
+    return None
+
+
+def normalize_hex(color: str) -> str:
+    """Normalize hex color to lowercase 6-digit format."""
+    parsed = parse_color(color)
+    return parsed.hex if parsed else color
+
+
+# =============================================================================
+# CONTRAST CALCULATIONS (WCAG)
+# =============================================================================
+
+def get_luminance(r: int, g: int, b: int) -> float:
+    """
+    Calculate relative luminance according to WCAG 2.1.
+    
+    Formula: L = 0.2126 * R + 0.7152 * G + 0.0722 * B
+    where R, G, B are linearized values.
+    """
+    def linearize(c: int) -> float:
+        c_norm = c / 255
+        if c_norm <= 0.04045:
+            return c_norm / 12.92
+        return ((c_norm + 0.055) / 1.055) ** 2.4
+    
+    return 0.2126 * linearize(r) + 0.7152 * linearize(g) + 0.0722 * linearize(b)
+
+
+def get_contrast_ratio(color1: str, color2: str) -> float:
+    """
+    Calculate WCAG contrast ratio between two colors.
+    
+    Returns ratio from 1:1 to 21:1
+    """
+    parsed1 = parse_color(color1)
+    parsed2 = parse_color(color2)
+    
+    if not parsed1 or not parsed2:
+        return 1.0
+    
+    l1 = get_luminance(*parsed1.rgb)
+    l2 = get_luminance(*parsed2.rgb)
+    
+    lighter = max(l1, l2)
+    darker = min(l1, l2)
+    
+    return (lighter + 0.05) / (darker + 0.05)
+
+
+def check_wcag_compliance(foreground: str, background: str) -> dict:
+    """
+    Check WCAG compliance for a color pair.
+    
+    Returns dict with AA and AAA compliance for normal and large text.
+    """
+    ratio = get_contrast_ratio(foreground, background)
+    
+    return {
+        "contrast_ratio": round(ratio, 2),
+        "aa_normal_text": ratio >= 4.5,      # AA for normal text
+        "aa_large_text": ratio >= 3.0,       # AA for large text (18pt+ or 14pt+ bold)
+        "aaa_normal_text": ratio >= 7.0,     # AAA for normal text
+        "aaa_large_text": ratio >= 4.5,      # AAA for large text
+    }
+
+
+def get_contrast_with_white(color: str) -> float:
+    """Get contrast ratio against white."""
+    return get_contrast_ratio(color, "#ffffff")
+
+
+def get_contrast_with_black(color: str) -> float:
+    """Get contrast ratio against black."""
+    return get_contrast_ratio(color, "#000000")
+
+
+def get_best_text_color(background: str) -> str:
+    """Determine whether white or black text works better on a background."""
+    white_contrast = get_contrast_with_white(background)
+    black_contrast = get_contrast_with_black(background)
+    return "#ffffff" if white_contrast > black_contrast else "#000000"
+
+
+# =============================================================================
+# COLOR SIMILARITY & DEDUPLICATION
+# =============================================================================
+
+def color_distance(color1: str, color2: str) -> float:
+    """
+    Calculate perceptual distance between two colors.
+    
+    Uses simple Euclidean distance in RGB space.
+    For more accuracy, consider using CIE Lab Delta E.
+    """
+    parsed1 = parse_color(color1)
+    parsed2 = parse_color(color2)
+    
+    if not parsed1 or not parsed2:
+        return float("inf")
+    
+    r1, g1, b1 = parsed1.rgb
+    r2, g2, b2 = parsed2.rgb
+    
+    return ((r1 - r2) ** 2 + (g1 - g2) ** 2 + (b1 - b2) ** 2) ** 0.5
+
+
+def are_colors_similar(color1: str, color2: str, threshold: float = 10.0) -> bool:
+    """Check if two colors are perceptually similar."""
+    return color_distance(color1, color2) < threshold
+
+
+def find_duplicate_colors(colors: list[str], threshold: float = 10.0) -> list[tuple[str, str]]:
+    """
+    Find pairs of colors that are potentially duplicates.
+    
+    Returns list of (color1, color2) tuples.
+    """
+    duplicates = []
+    normalized = [normalize_hex(c) for c in colors]
+    
+    for i, color1 in enumerate(normalized):
+        for color2 in normalized[i + 1:]:
+            if are_colors_similar(color1, color2, threshold):
+                duplicates.append((color1, color2))
+    
+    return duplicates
+
+
+def deduplicate_colors(colors: list[str], threshold: float = 10.0) -> list[str]:
+    """
+    Remove duplicate colors, keeping the first occurrence.
+    """
+    result = []
+    normalized = [normalize_hex(c) for c in colors]
+    
+    for color in normalized:
+        is_duplicate = False
+        for existing in result:
+            if are_colors_similar(color, existing, threshold):
+                is_duplicate = True
+                break
+        if not is_duplicate:
+            result.append(color)
+    
+    return result
+
+
+# =============================================================================
+# COLOR RAMP GENERATION
+# =============================================================================
+
+def generate_color_ramp(
+    base_color: str,
+    shades: list[int] = None,
+    method: str = "hsl"
+) -> dict[str, str]:
+    """
+    Generate a color ramp from a base color.
+    
+    Args:
+        base_color: The base color (typically becomes 500)
+        shades: List of shade values (default: [50, 100, 200, 300, 400, 500, 600, 700, 800, 900])
+        method: "hsl" (simple) or "oklch" (perceptually uniform, not implemented)
+    
+    Returns:
+        Dict mapping shade to hex color.
+    """
+    if shades is None:
+        shades = [50, 100, 200, 300, 400, 500, 600, 700, 800, 900]
+    
+    parsed = parse_color(base_color)
+    if not parsed:
+        return {}
+    
+    h, s, l = parsed.hsl
+    ramp = {}
+    
+    # Base color is 500
+    base_shade = 500
+    
+    for shade in shades:
+        if shade == base_shade:
+            ramp[str(shade)] = parsed.hex
+            continue
+        
+        # Calculate lightness adjustment
+        # Lighter shades (50-400): increase lightness
+        # Darker shades (600-900): decrease lightness
+        if shade < base_shade:
+            # Lighter: interpolate toward white
+            factor = (base_shade - shade) / base_shade
+            new_l = l + (100 - l) * factor * 0.85
+            # Slightly reduce saturation for very light shades
+            new_s = s * (1 - factor * 0.3)
+        else:
+            # Darker: interpolate toward black
+            factor = (shade - base_shade) / (900 - base_shade)
+            new_l = l * (1 - factor * 0.85)
+            # Increase saturation slightly for dark shades
+            new_s = min(100, s * (1 + factor * 0.2))
+        
+        new_rgb = hsl_to_rgb(h, new_s, new_l)
+        ramp[str(shade)] = rgb_to_hex(*new_rgb)
+    
+    return ramp
+
+
+def generate_accessible_ramp(
+    base_color: str,
+    background: str = "#ffffff"
+) -> dict[str, dict]:
+    """
+    Generate a color ramp with accessibility information.
+    
+    Returns dict with color values and their contrast ratios.
+    """
+    ramp = generate_color_ramp(base_color)
+    result = {}
+    
+    for shade, color in ramp.items():
+        compliance = check_wcag_compliance(color, background)
+        result[shade] = {
+            "value": color,
+            "contrast_ratio": compliance["contrast_ratio"],
+            "aa_text": compliance["aa_normal_text"],
+            "best_text_color": get_best_text_color(color),
+        }
+    
+    return result
+
+
+# =============================================================================
+# COLOR CATEGORIZATION
+# =============================================================================
+
+def categorize_color(color: str) -> str:
+    """
+    Categorize a color by its general hue.
+    
+    Returns: "red", "orange", "yellow", "green", "cyan", "blue", "purple", "pink", "neutral"
+    """
+    parsed = parse_color(color)
+    if not parsed:
+        return "unknown"
+    
+    h, s, l = parsed.hsl
+    
+    # Neutrals (low saturation or extreme lightness)
+    if s < 10 or l < 5 or l > 95:
+        return "neutral"
+    
+    # Categorize by hue
+    if h < 15 or h >= 345:
+        return "red"
+    elif h < 45:
+        return "orange"
+    elif h < 70:
+        return "yellow"
+    elif h < 150:
+        return "green"
+    elif h < 190:
+        return "cyan"
+    elif h < 260:
+        return "blue"
+    elif h < 290:
+        return "purple"
+    else:
+        return "pink"
+
+
+def suggest_color_name(color: str, role: str = None) -> str:
+    """
+    Suggest a semantic name for a color.
+    
+    Args:
+        color: The color value
+        role: Optional role hint ("primary", "background", "text", etc.)
+    
+    Returns suggested name like "blue-500" or "neutral-100"
+    """
+    parsed = parse_color(color)
+    if not parsed:
+        return "unknown"
+    
+    category = categorize_color(color)
+    h, s, l = parsed.hsl
+    
+    # Determine shade level based on lightness
+    if l >= 95:
+        shade = "50"
+    elif l >= 85:
+        shade = "100"
+    elif l >= 75:
+        shade = "200"
+    elif l >= 65:
+        shade = "300"
+    elif l >= 55:
+        shade = "400"
+    elif l >= 45:
+        shade = "500"
+    elif l >= 35:
+        shade = "600"
+    elif l >= 25:
+        shade = "700"
+    elif l >= 15:
+        shade = "800"
+    else:
+        shade = "900"
+    
+    return f"{category}-{shade}"
+
+
+def group_colors_by_category(colors: list[str]) -> dict[str, list[str]]:
+    """
+    Group colors by their category.
+    
+    Returns dict mapping category to list of colors.
+    """
+    groups: dict[str, list[str]] = {}
+    
+    for color in colors:
+        category = categorize_color(color)
+        if category not in groups:
+            groups[category] = []
+        groups[category].append(normalize_hex(color))
+    
+    return groups
+
+
+# =============================================================================
+# UTILITY FUNCTIONS
+# =============================================================================
+
+def sort_colors_by_hue(colors: list[str]) -> list[str]:
+    """Sort colors by hue, then by lightness."""
+    def sort_key(color: str):
+        parsed = parse_color(color)
+        if not parsed:
+            return (0, 0, 0)
+        h, s, l = parsed.hsl
+        # Neutrals (low saturation) go at the end
+        if s < 10:
+            return (360 + l, s, l)
+        return (h, s, l)
+    
+    return sorted([normalize_hex(c) for c in colors], key=sort_key)
+
+
+def sort_colors_by_lightness(colors: list[str]) -> list[str]:
+    """Sort colors from light to dark."""
+    def sort_key(color: str):
+        parsed = parse_color(color)
+        return -parsed.hsl[2] if parsed else 0
+    
+    return sorted([normalize_hex(c) for c in colors], key=sort_key)
+
+
+def is_dark_color(color: str) -> bool:
+    """Check if a color is considered dark (for text on backgrounds)."""
+    parsed = parse_color(color)
+    if not parsed:
+        return False
+    return parsed.hsl[2] < 50
+
+
+def is_light_color(color: str) -> bool:
+    """Check if a color is considered light."""
+    return not is_dark_color(color)

core/hf_inference.py

@@ -0,0 +1,580 @@
+"""
+HuggingFace Inference Client
+Design System Extractor v2
+
+Handles all LLM inference calls using HuggingFace Inference API.
+Supports diverse models from different providers for specialized tasks.
+"""
+
+import os
+from typing import Optional, AsyncGenerator
+from dataclasses import dataclass
+from huggingface_hub import InferenceClient, AsyncInferenceClient
+
+from config.settings import get_settings
+
+
+@dataclass
+class ModelInfo:
+    """Information about a model."""
+    model_id: str
+    provider: str
+    context_length: int
+    strengths: list[str]
+    best_for: str
+    tier: str  # "free", "pro", "pro+"
+
+
+# =============================================================================
+# COMPREHENSIVE MODEL REGISTRY — Organized by Provider
+# =============================================================================
+
+AVAILABLE_MODELS = {
+    # =========================================================================
+    # META — Llama Family (Best for reasoning)
+    # =========================================================================
+    "meta-llama/Llama-3.1-405B-Instruct": ModelInfo(
+        model_id="meta-llama/Llama-3.1-405B-Instruct",
+        provider="Meta",
+        context_length=128000,
+        strengths=["Best reasoning", "Massive knowledge", "Complex analysis"],
+        best_for="Agent 3 (Advisor) — PREMIUM CHOICE",
+        tier="pro+"
+    ),
+    "meta-llama/Llama-3.1-70B-Instruct": ModelInfo(
+        model_id="meta-llama/Llama-3.1-70B-Instruct",
+        provider="Meta",
+        context_length=128000,
+        strengths=["Excellent reasoning", "Long context", "Design knowledge"],
+        best_for="Agent 3 (Advisor) — RECOMMENDED",
+        tier="pro"
+    ),
+    "meta-llama/Llama-3.1-8B-Instruct": ModelInfo(
+        model_id="meta-llama/Llama-3.1-8B-Instruct",
+        provider="Meta",
+        context_length=128000,
+        strengths=["Fast", "Good reasoning for size", "Long context"],
+        best_for="Budget Agent 3 fallback",
+        tier="free"
+    ),
+    
+    # =========================================================================
+    # MISTRAL — European Excellence
+    # =========================================================================
+    "mistralai/Mixtral-8x22B-Instruct-v0.1": ModelInfo(
+        model_id="mistralai/Mixtral-8x22B-Instruct-v0.1",
+        provider="Mistral",
+        context_length=65536,
+        strengths=["Large MoE", "Strong reasoning", "Efficient"],
+        best_for="Agent 3 (Advisor) — Pro alternative",
+        tier="pro"
+    ),
+    "mistralai/Mixtral-8x7B-Instruct-v0.1": ModelInfo(
+        model_id="mistralai/Mixtral-8x7B-Instruct-v0.1",
+        provider="Mistral",
+        context_length=32768,
+        strengths=["Good MoE efficiency", "Solid reasoning"],
+        best_for="Agent 3 (Advisor) — Free tier option",
+        tier="free"
+    ),
+    "mistralai/Mistral-7B-Instruct-v0.3": ModelInfo(
+        model_id="mistralai/Mistral-7B-Instruct-v0.3",
+        provider="Mistral",
+        context_length=32768,
+        strengths=["Fast", "Good instruction following"],
+        best_for="General fallback",
+        tier="free"
+    ),
+    "mistralai/Codestral-22B-v0.1": ModelInfo(
+        model_id="mistralai/Codestral-22B-v0.1",
+        provider="Mistral",
+        context_length=32768,
+        strengths=["Code specialist", "JSON generation", "Structured output"],
+        best_for="Agent 4 (Generator) — RECOMMENDED",
+        tier="pro"
+    ),
+    
+    # =========================================================================
+    # COHERE — Command R Family (Analysis & Retrieval)
+    # =========================================================================
+    "CohereForAI/c4ai-command-r-plus": ModelInfo(
+        model_id="CohereForAI/c4ai-command-r-plus",
+        provider="Cohere",
+        context_length=128000,
+        strengths=["Excellent analysis", "RAG optimized", "Long context"],
+        best_for="Agent 3 (Advisor) — Great for research tasks",
+        tier="pro"
+    ),
+    "CohereForAI/c4ai-command-r-v01": ModelInfo(
+        model_id="CohereForAI/c4ai-command-r-v01",
+        provider="Cohere",
+        context_length=128000,
+        strengths=["Good analysis", "Efficient"],
+        best_for="Agent 3 budget option",
+        tier="free"
+    ),
+    
+    # =========================================================================
+    # GOOGLE — Gemma Family
+    # =========================================================================
+    "google/gemma-2-27b-it": ModelInfo(
+        model_id="google/gemma-2-27b-it",
+        provider="Google",
+        context_length=8192,
+        strengths=["Strong instruction following", "Good balance"],
+        best_for="Agent 2 (Normalizer) — Quality option",
+        tier="pro"
+    ),
+    "google/gemma-2-9b-it": ModelInfo(
+        model_id="google/gemma-2-9b-it",
+        provider="Google",
+        context_length=8192,
+        strengths=["Fast", "Good instruction following"],
+        best_for="Agent 2 (Normalizer) — Balanced",
+        tier="free"
+    ),
+    
+    # =========================================================================
+    # MICROSOFT — Phi Family (Small but Mighty)
+    # =========================================================================
+    "microsoft/Phi-3.5-mini-instruct": ModelInfo(
+        model_id="microsoft/Phi-3.5-mini-instruct",
+        provider="Microsoft",
+        context_length=128000,
+        strengths=["Very fast", "Great structured output", "Long context"],
+        best_for="Agent 2 (Normalizer) — RECOMMENDED",
+        tier="free"
+    ),
+    "microsoft/Phi-3-medium-4k-instruct": ModelInfo(
+        model_id="microsoft/Phi-3-medium-4k-instruct",
+        provider="Microsoft",
+        context_length=4096,
+        strengths=["Fast", "Good for simple tasks"],
+        best_for="Simple naming tasks",
+        tier="free"
+    ),
+    
+    # =========================================================================
+    # QWEN — Alibaba Family
+    # =========================================================================
+    "Qwen/Qwen2.5-72B-Instruct": ModelInfo(
+        model_id="Qwen/Qwen2.5-72B-Instruct",
+        provider="Alibaba",
+        context_length=32768,
+        strengths=["Strong reasoning", "Multilingual", "Good design knowledge"],
+        best_for="Agent 3 (Advisor) — Alternative",
+        tier="pro"
+    ),
+    "Qwen/Qwen2.5-32B-Instruct": ModelInfo(
+        model_id="Qwen/Qwen2.5-32B-Instruct",
+        provider="Alibaba",
+        context_length=32768,
+        strengths=["Good balance", "Multilingual"],
+        best_for="Medium-tier option",
+        tier="pro"
+    ),
+    "Qwen/Qwen2.5-Coder-32B-Instruct": ModelInfo(
+        model_id="Qwen/Qwen2.5-Coder-32B-Instruct",
+        provider="Alibaba",
+        context_length=32768,
+        strengths=["Code specialist", "JSON/structured output"],
+        best_for="Agent 4 (Generator) — Alternative",
+        tier="pro"
+    ),
+    "Qwen/Qwen2.5-7B-Instruct": ModelInfo(
+        model_id="Qwen/Qwen2.5-7B-Instruct",
+        provider="Alibaba",
+        context_length=32768,
+        strengths=["Fast", "Good all-rounder"],
+        best_for="General fallback",
+        tier="free"
+    ),
+    
+    # =========================================================================
+    # DEEPSEEK — Code Specialists
+    # =========================================================================
+    "deepseek-ai/deepseek-coder-33b-instruct": ModelInfo(
+        model_id="deepseek-ai/deepseek-coder-33b-instruct",
+        provider="DeepSeek",
+        context_length=16384,
+        strengths=["Excellent code generation", "JSON specialist"],
+        best_for="Agent 4 (Generator) — Code focused",
+        tier="pro"
+    ),
+    "deepseek-ai/DeepSeek-V2.5": ModelInfo(
+        model_id="deepseek-ai/DeepSeek-V2.5",
+        provider="DeepSeek",
+        context_length=32768,
+        strengths=["Strong reasoning", "Good code"],
+        best_for="Multi-purpose",
+        tier="pro"
+    ),
+    
+    # =========================================================================
+    # BIGCODE — StarCoder Family
+    # =========================================================================
+    "bigcode/starcoder2-15b-instruct-v0.1": ModelInfo(
+        model_id="bigcode/starcoder2-15b-instruct-v0.1",
+        provider="BigCode",
+        context_length=16384,
+        strengths=["Code generation", "Multiple languages"],
+        best_for="Agent 4 (Generator) — Open source code model",
+        tier="free"
+    ),
+}
+
+
+# =============================================================================
+# RECOMMENDED CONFIGURATIONS BY TIER
+# =============================================================================
+
+MODEL_PRESETS = {
+    "budget": {
+        "name": "Budget (Free Tier)",
+        "description": "Best free models for each task",
+        "agent2": "microsoft/Phi-3.5-mini-instruct",
+        "agent3": "mistralai/Mixtral-8x7B-Instruct-v0.1",
+        "agent4": "bigcode/starcoder2-15b-instruct-v0.1",
+        "fallback": "mistralai/Mistral-7B-Instruct-v0.3",
+    },
+    "balanced": {
+        "name": "Balanced (Pro Tier)",
+        "description": "Good quality/cost balance",
+        "agent2": "google/gemma-2-9b-it",
+        "agent3": "meta-llama/Llama-3.1-70B-Instruct",
+        "agent4": "mistralai/Codestral-22B-v0.1",
+        "fallback": "Qwen/Qwen2.5-7B-Instruct",
+    },
+    "quality": {
+        "name": "Maximum Quality (Pro+)",
+        "description": "Best models regardless of cost",
+        "agent2": "google/gemma-2-27b-it",
+        "agent3": "meta-llama/Llama-3.1-405B-Instruct",
+        "agent4": "deepseek-ai/deepseek-coder-33b-instruct",
+        "fallback": "meta-llama/Llama-3.1-8B-Instruct",
+    },
+    "diverse": {
+        "name": "Diverse Providers",
+        "description": "One model from each major provider",
+        "agent2": "microsoft/Phi-3.5-mini-instruct",  # Microsoft
+        "agent3": "CohereForAI/c4ai-command-r-plus",  # Cohere
+        "agent4": "mistralai/Codestral-22B-v0.1",     # Mistral
+        "fallback": "meta-llama/Llama-3.1-8B-Instruct",  # Meta
+    },
+}
+
+
+# =============================================================================
+# AGENT-SPECIFIC RECOMMENDATIONS
+# =============================================================================
+
+AGENT_MODEL_RECOMMENDATIONS = {
+    "crawler": {
+        "requires_llm": False,
+        "notes": "Pure rule-based extraction using Playwright + CSS parsing"
+    },
+    "extractor": {
+        "requires_llm": False,
+        "notes": "Pure rule-based extraction using Playwright + CSS parsing"
+    },
+    "normalizer": {
+        "requires_llm": True,
+        "task": "Token naming, duplicate detection, pattern inference",
+        "needs": ["Fast inference", "Good instruction following", "Structured output"],
+        "recommended": [
+            ("microsoft/Phi-3.5-mini-instruct", "BEST — Fast, great structured output"),
+            ("google/gemma-2-9b-it", "Good balance of speed and quality"),
+            ("Qwen/Qwen2.5-7B-Instruct", "Reliable all-rounder"),
+        ],
+        "temperature": 0.2,
+    },
+    "advisor": {
+        "requires_llm": True,
+        "task": "Design system analysis, best practice recommendations",
+        "needs": ["Strong reasoning", "Design knowledge", "Creative suggestions"],
+        "recommended": [
+            ("meta-llama/Llama-3.1-70B-Instruct", "BEST — Excellent reasoning"),
+            ("CohereForAI/c4ai-command-r-plus", "Great for analysis tasks"),
+            ("Qwen/Qwen2.5-72B-Instruct", "Strong alternative"),
+            ("mistralai/Mixtral-8x7B-Instruct-v0.1", "Best free option"),
+        ],
+        "temperature": 0.4,
+    },
+    "generator": {
+        "requires_llm": True,
+        "task": "Generate JSON tokens, CSS variables, structured output",
+        "needs": ["Code generation", "JSON formatting", "Schema adherence"],
+        "recommended": [
+            ("mistralai/Codestral-22B-v0.1", "BEST — Mistral's code model"),
+            ("deepseek-ai/deepseek-coder-33b-instruct", "Excellent code specialist"),
+            ("Qwen/Qwen2.5-Coder-32B-Instruct", "Strong code model"),
+            ("bigcode/starcoder2-15b-instruct-v0.1", "Best free option"),
+        ],
+        "temperature": 0.1,
+    },
+}
+
+
+# =============================================================================
+# INFERENCE CLIENT
+# =============================================================================
+
+class HFInferenceClient:
+    """
+    Wrapper around HuggingFace Inference API.
+    
+    Handles model selection, retries, and fallbacks.
+    """
+    
+    def __init__(self):
+        self.settings = get_settings()
+        self.token = self.settings.hf.hf_token
+        
+        if not self.token:
+            raise ValueError("HF_TOKEN is required for inference")
+        
+        # Create clients
+        self.sync_client = InferenceClient(token=self.token)
+        self.async_client = AsyncInferenceClient(token=self.token)
+    
+    def get_model_for_agent(self, agent_name: str) -> str:
+        """Get the appropriate model for an agent."""
+        return self.settings.get_model_for_agent(agent_name)
+    
+    def get_temperature_for_agent(self, agent_name: str) -> float:
+        """Get recommended temperature for an agent."""
+        temps = {
+            "normalizer": 0.2,  # Consistent naming
+            "advisor": 0.4,    # Creative recommendations
+            "generator": 0.1,  # Precise formatting
+        }
+        return temps.get(agent_name, 0.3)
+    
+    def _build_messages(
+        self,
+        system_prompt: str,
+        user_message: str,
+        examples: list[dict] = None
+    ) -> list[dict]:
+        """Build message list for chat completion."""
+        messages = []
+        
+        if system_prompt:
+            messages.append({"role": "system", "content": system_prompt})
+        
+        if examples:
+            for example in examples:
+                messages.append({"role": "user", "content": example["user"]})
+                messages.append({"role": "assistant", "content": example["assistant"]})
+        
+        messages.append({"role": "user", "content": user_message})
+        
+        return messages
+    
+    def complete(
+        self,
+        agent_name: str,
+        system_prompt: str,
+        user_message: str,
+        examples: list[dict] = None,
+        max_tokens: int = None,
+        temperature: float = None,
+        json_mode: bool = False,
+    ) -> str:
+        """
+        Synchronous completion.
+        
+        Args:
+            agent_name: Which agent is making the call (for model selection)
+            system_prompt: System instructions
+            user_message: User input
+            examples: Optional few-shot examples
+            max_tokens: Max tokens to generate
+            temperature: Sampling temperature (uses agent default if not specified)
+            json_mode: If True, instruct model to output JSON
+        
+        Returns:
+            Generated text
+        """
+        model = self.get_model_for_agent(agent_name)
+        max_tokens = max_tokens or self.settings.hf.max_new_tokens
+        temperature = temperature or self.get_temperature_for_agent(agent_name)
+        
+        # Build messages
+        if json_mode:
+            system_prompt = f"{system_prompt}\n\nYou must respond with valid JSON only. No markdown, no explanation, just JSON."
+        
+        messages = self._build_messages(system_prompt, user_message, examples)
+        
+        try:
+            response = self.sync_client.chat_completion(
+                model=model,
+                messages=messages,
+                max_tokens=max_tokens,
+                temperature=temperature,
+            )
+            return response.choices[0].message.content
+            
+        except Exception as e:
+            # Try fallback model
+            fallback = self.settings.models.fallback_model
+            if fallback != model:
+                print(f"Primary model {model} failed, trying fallback: {fallback}")
+                response = self.sync_client.chat_completion(
+                    model=fallback,
+                    messages=messages,
+                    max_tokens=max_tokens,
+                    temperature=temperature,
+                )
+                return response.choices[0].message.content
+            raise e
+    
+    async def complete_async(
+        self,
+        agent_name: str,
+        system_prompt: str,
+        user_message: str,
+        examples: list[dict] = None,
+        max_tokens: int = None,
+        temperature: float = None,
+        json_mode: bool = False,
+    ) -> str:
+        """
+        Asynchronous completion.
+        
+        Same parameters as complete().
+        """
+        model = self.get_model_for_agent(agent_name)
+        max_tokens = max_tokens or self.settings.hf.max_new_tokens
+        temperature = temperature or self.get_temperature_for_agent(agent_name)
+        
+        if json_mode:
+            system_prompt = f"{system_prompt}\n\nYou must respond with valid JSON only. No markdown, no explanation, just JSON."
+        
+        messages = self._build_messages(system_prompt, user_message, examples)
+        
+        try:
+            response = await self.async_client.chat_completion(
+                model=model,
+                messages=messages,
+                max_tokens=max_tokens,
+                temperature=temperature,
+            )
+            return response.choices[0].message.content
+            
+        except Exception as e:
+            fallback = self.settings.models.fallback_model
+            if fallback != model:
+                print(f"Primary model {model} failed, trying fallback: {fallback}")
+                response = await self.async_client.chat_completion(
+                    model=fallback,
+                    messages=messages,
+                    max_tokens=max_tokens,
+                    temperature=temperature,
+                )
+                return response.choices[0].message.content
+            raise e
+    
+    async def stream_async(
+        self,
+        agent_name: str,
+        system_prompt: str,
+        user_message: str,
+        max_tokens: int = None,
+        temperature: float = None,
+    ) -> AsyncGenerator[str, None]:
+        """
+        Async streaming completion.
+        
+        Yields tokens as they are generated.
+        """
+        model = self.get_model_for_agent(agent_name)
+        max_tokens = max_tokens or self.settings.hf.max_new_tokens
+        temperature = temperature or self.get_temperature_for_agent(agent_name)
+        
+        messages = self._build_messages(system_prompt, user_message)
+        
+        async for chunk in await self.async_client.chat_completion(
+            model=model,
+            messages=messages,
+            max_tokens=max_tokens,
+            temperature=temperature,
+            stream=True,
+        ):
+            if chunk.choices[0].delta.content:
+                yield chunk.choices[0].delta.content
+
+
+# =============================================================================
+# SINGLETON & CONVENIENCE FUNCTIONS
+# =============================================================================
+
+_client: Optional[HFInferenceClient] = None
+
+
+def get_inference_client() -> HFInferenceClient:
+    """Get or create the inference client singleton."""
+    global _client
+    if _client is None:
+        _client = HFInferenceClient()
+    return _client
+
+
+def complete(
+    agent_name: str,
+    system_prompt: str,
+    user_message: str,
+    **kwargs
+) -> str:
+    """Convenience function for sync completion."""
+    client = get_inference_client()
+    return client.complete(agent_name, system_prompt, user_message, **kwargs)
+
+
+async def complete_async(
+    agent_name: str,
+    system_prompt: str,
+    user_message: str,
+    **kwargs
+) -> str:
+    """Convenience function for async completion."""
+    client = get_inference_client()
+    return await client.complete_async(agent_name, system_prompt, user_message, **kwargs)
+
+
+def get_model_info(model_id: str) -> dict:
+    """Get information about a specific model."""
+    if model_id in AVAILABLE_MODELS:
+        info = AVAILABLE_MODELS[model_id]
+        return {
+            "model_id": info.model_id,
+            "provider": info.provider,
+            "context_length": info.context_length,
+            "strengths": info.strengths,
+            "best_for": info.best_for,
+            "tier": info.tier,
+        }
+    return {"model_id": model_id, "provider": "unknown"}
+
+
+def get_models_by_provider() -> dict[str, list[str]]:
+    """Get all models grouped by provider."""
+    by_provider = {}
+    for model_id, info in AVAILABLE_MODELS.items():
+        if info.provider not in by_provider:
+            by_provider[info.provider] = []
+        by_provider[info.provider].append(model_id)
+    return by_provider
+
+
+def get_models_by_tier(tier: str) -> list[str]:
+    """Get all models for a specific tier (free, pro, pro+)."""
+    return [
+        model_id for model_id, info in AVAILABLE_MODELS.items()
+        if info.tier == tier
+    ]
+
+
+def get_preset_config(preset_name: str) -> dict:
+    """Get a preset model configuration."""
+    return MODEL_PRESETS.get(preset_name, MODEL_PRESETS["balanced"])

core/token_schema.py

@@ -0,0 +1,473 @@
+"""
+Token Schema Definitions
+Design System Extractor v2
+
+Pydantic models for all token types and extraction results.
+These are the core data structures used throughout the application.
+"""
+
+from datetime import datetime
+from enum import Enum
+from typing import Optional, Any
+from pydantic import BaseModel, Field, field_validator
+
+
+# =============================================================================
+# ENUMS
+# =============================================================================
+
+class TokenSource(str, Enum):
+    """Origin of a token value."""
+    DETECTED = "detected"       # Directly found in CSS
+    INFERRED = "inferred"       # Derived from patterns
+    UPGRADED = "upgraded"       # User-selected improvement
+    MANUAL = "manual"           # User manually added
+
+
+class Confidence(str, Enum):
+    """Confidence level for extracted tokens."""
+    HIGH = "high"               # 10+ occurrences, consistent usage
+    MEDIUM = "medium"           # 3-9 occurrences
+    LOW = "low"                 # 1-2 occurrences or conflicting
+
+
+class Viewport(str, Enum):
+    """Viewport type."""
+    DESKTOP = "desktop"         # 1440px width
+    MOBILE = "mobile"           # 375px width
+
+
+class PageType(str, Enum):
+    """Type of page template."""
+    HOMEPAGE = "homepage"
+    LISTING = "listing"
+    DETAIL = "detail"
+    FORM = "form"
+    MARKETING = "marketing"
+    AUTH = "auth"
+    CHECKOUT = "checkout"
+    ABOUT = "about"
+    CONTACT = "contact"
+    OTHER = "other"
+
+
+# =============================================================================
+# BASE TOKEN MODEL
+# =============================================================================
+
+class BaseToken(BaseModel):
+    """Base class for all tokens."""
+    source: TokenSource = TokenSource.DETECTED
+    confidence: Confidence = Confidence.MEDIUM
+    frequency: int = 0
+    suggested_name: Optional[str] = None
+    
+    # For tracking user decisions
+    accepted: bool = True
+    flagged: bool = False
+    notes: Optional[str] = None
+
+
+# =============================================================================
+# COLOR TOKENS
+# =============================================================================
+
+class ColorToken(BaseToken):
+    """Extracted color token."""
+    value: str  # hex value (e.g., "#007bff")
+    value_rgb: Optional[str] = None  # "rgb(0, 123, 255)"
+    value_hsl: Optional[str] = None  # "hsl(211, 100%, 50%)"
+    
+    # Context information
+    contexts: list[str] = Field(default_factory=list)  # ["background", "text", "border"]
+    elements: list[str] = Field(default_factory=list)  # ["button", "header", "link"]
+    css_properties: list[str] = Field(default_factory=list)  # ["background-color", "color"]
+    
+    # Accessibility
+    contrast_white: Optional[float] = None  # Contrast ratio against white
+    contrast_black: Optional[float] = None  # Contrast ratio against black
+    wcag_aa_large_text: bool = False
+    wcag_aa_small_text: bool = False
+    wcag_aaa_large_text: bool = False
+    wcag_aaa_small_text: bool = False
+    
+    @field_validator("value")
+    @classmethod
+    def validate_hex(cls, v: str) -> str:
+        """Ensure hex color is properly formatted."""
+        v = v.strip().lower()
+        if not v.startswith("#"):
+            v = f"#{v}"
+        # Convert 3-digit hex to 6-digit
+        if len(v) == 4:
+            v = f"#{v[1]}{v[1]}{v[2]}{v[2]}{v[3]}{v[3]}"
+        return v
+
+
+class ColorRamp(BaseModel):
+    """Generated color ramp with shades."""
+    base_color: str  # Original extracted color
+    name: str  # e.g., "primary", "neutral"
+    shades: dict[str, str] = Field(default_factory=dict)  # {"50": "#e6f2ff", "500": "#007bff", ...}
+    source: TokenSource = TokenSource.UPGRADED
+
+
+# =============================================================================
+# TYPOGRAPHY TOKENS
+# =============================================================================
+
+class TypographyToken(BaseToken):
+    """Extracted typography token."""
+    font_family: str
+    font_size: str  # "16px" or "1rem"
+    font_size_px: Optional[float] = None  # Computed px value
+    font_weight: int = 400
+    line_height: str = "1.5"  # "1.5" or "24px"
+    line_height_computed: Optional[float] = None  # Computed ratio
+    letter_spacing: Optional[str] = None
+    text_transform: Optional[str] = None  # "uppercase", "lowercase", etc.
+    
+    # Context
+    elements: list[str] = Field(default_factory=list)  # ["h1", "p", "button"]
+    css_selectors: list[str] = Field(default_factory=list)  # [".heading", ".body-text"]
+
+
+class TypeScale(BaseModel):
+    """Typography scale configuration."""
+    name: str  # "Major Third", "Perfect Fourth"
+    ratio: float  # 1.25, 1.333
+    base_size: int = 16  # px
+    sizes: dict[str, str] = Field(default_factory=dict)  # {"xs": "12px", "sm": "14px", ...}
+    source: TokenSource = TokenSource.UPGRADED
+
+
+class FontFamily(BaseModel):
+    """Font family information."""
+    name: str  # "Inter"
+    fallbacks: list[str] = Field(default_factory=list)  # ["system-ui", "sans-serif"]
+    category: str = "sans-serif"  # "serif", "sans-serif", "monospace"
+    frequency: int = 0
+    usage: str = "primary"  # "primary", "secondary", "accent", "monospace"
+
+
+# =============================================================================
+# SPACING TOKENS
+# =============================================================================
+
+class SpacingToken(BaseToken):
+    """Extracted spacing token."""
+    value: str  # "16px"
+    value_px: int  # 16
+    
+    # Context
+    contexts: list[str] = Field(default_factory=list)  # ["margin", "padding", "gap"]
+    properties: list[str] = Field(default_factory=list)  # ["margin-top", "padding-left"]
+    
+    # Analysis
+    fits_base_4: bool = False  # Divisible by 4
+    fits_base_8: bool = False  # Divisible by 8
+    is_outlier: bool = False  # Doesn't fit common patterns
+
+
+class SpacingScale(BaseModel):
+    """Spacing scale configuration."""
+    name: str  # "8px base"
+    base: int  # 8
+    scale: list[int] = Field(default_factory=list)  # [4, 8, 16, 24, 32, 48, 64]
+    names: dict[int, str] = Field(default_factory=dict)  # {4: "xs", 8: "sm", 16: "md"}
+    source: TokenSource = TokenSource.UPGRADED
+
+
+# =============================================================================
+# BORDER RADIUS TOKENS
+# =============================================================================
+
+class RadiusToken(BaseToken):
+    """Extracted border radius token."""
+    value: str  # "8px" or "50%"
+    value_px: Optional[int] = None  # If px value
+    
+    # Context
+    elements: list[str] = Field(default_factory=list)  # ["button", "card", "input"]
+    
+    # Analysis
+    fits_base_4: bool = False
+    fits_base_8: bool = False
+
+
+# =============================================================================
+# SHADOW TOKENS
+# =============================================================================
+
+class ShadowToken(BaseToken):
+    """Extracted box shadow token."""
+    value: str  # Full CSS shadow value
+    
+    # Parsed components
+    offset_x: Optional[str] = None
+    offset_y: Optional[str] = None
+    blur: Optional[str] = None
+    spread: Optional[str] = None
+    color: Optional[str] = None
+    inset: bool = False
+    
+    # Context
+    elements: list[str] = Field(default_factory=list)
+
+
+# =============================================================================
+# PAGE & CRAWL MODELS
+# =============================================================================
+
+class DiscoveredPage(BaseModel):
+    """A page discovered during crawling."""
+    url: str
+    title: Optional[str] = None
+    page_type: PageType = PageType.OTHER
+    depth: int = 0  # Distance from homepage
+    selected: bool = True  # User can deselect pages
+    
+    # Crawl status
+    crawled: bool = False
+    error: Optional[str] = None
+
+
+class CrawlResult(BaseModel):
+    """Result of crawling a single page."""
+    url: str
+    viewport: Viewport
+    success: bool
+    
+    # Timing
+    started_at: datetime
+    completed_at: Optional[datetime] = None
+    duration_ms: Optional[int] = None
+    
+    # Results
+    colors_found: int = 0
+    typography_found: int = 0
+    spacing_found: int = 0
+    
+    # Errors
+    error: Optional[str] = None
+    warnings: list[str] = Field(default_factory=list)
+
+
+# =============================================================================
+# EXTRACTION RESULT
+# =============================================================================
+
+class ExtractedTokens(BaseModel):
+    """Complete extraction result for one viewport."""
+    viewport: Viewport
+    source_url: str
+    pages_crawled: list[str] = Field(default_factory=list)
+    
+    # Extracted tokens
+    colors: list[ColorToken] = Field(default_factory=list)
+    typography: list[TypographyToken] = Field(default_factory=list)
+    spacing: list[SpacingToken] = Field(default_factory=list)
+    radius: list[RadiusToken] = Field(default_factory=list)
+    shadows: list[ShadowToken] = Field(default_factory=list)
+    
+    # Detected patterns
+    font_families: list[FontFamily] = Field(default_factory=list)
+    base_font_size: Optional[str] = None
+    spacing_base: Optional[int] = None  # Detected: 4 or 8
+    naming_convention: Optional[str] = None  # "bem", "utility", "none"
+    
+    # Metadata
+    extraction_timestamp: datetime = Field(default_factory=datetime.now)
+    extraction_duration_ms: Optional[int] = None
+    
+    # Quality indicators
+    total_elements_analyzed: int = 0
+    unique_colors: int = 0
+    unique_font_sizes: int = 0
+    unique_spacing_values: int = 0
+    
+    # Issues
+    errors: list[str] = Field(default_factory=list)
+    warnings: list[str] = Field(default_factory=list)
+    
+    def summary(self) -> dict:
+        """Get extraction summary."""
+        return {
+            "viewport": self.viewport.value,
+            "pages_crawled": len(self.pages_crawled),
+            "colors": len(self.colors),
+            "typography": len(self.typography),
+            "spacing": len(self.spacing),
+            "radius": len(self.radius),
+            "shadows": len(self.shadows),
+            "font_families": len(self.font_families),
+            "errors": len(self.errors),
+            "warnings": len(self.warnings),
+        }
+
+
+# =============================================================================
+# NORMALIZED TOKENS (Agent 2 Output)
+# =============================================================================
+
+class NormalizedTokens(BaseModel):
+    """Normalized and structured tokens from Agent 2."""
+    viewport: Viewport
+    source_url: str
+    
+    # Normalized tokens with suggested names
+    colors: dict[str, ColorToken] = Field(default_factory=dict)  # {"primary-500": ColorToken, ...}
+    typography: dict[str, TypographyToken] = Field(default_factory=dict)
+    spacing: dict[str, SpacingToken] = Field(default_factory=dict)
+    radius: dict[str, RadiusToken] = Field(default_factory=dict)
+    shadows: dict[str, ShadowToken] = Field(default_factory=dict)
+    
+    # Detected info
+    font_families: list[FontFamily] = Field(default_factory=list)
+    detected_spacing_base: Optional[int] = None
+    detected_naming_convention: Optional[str] = None
+    
+    # Duplicates & conflicts
+    duplicate_colors: list[tuple[str, str]] = Field(default_factory=list)  # [("#1a1a1a", "#1b1b1b"), ...]
+    conflicting_tokens: list[str] = Field(default_factory=list)
+    
+    # Metadata
+    normalized_at: datetime = Field(default_factory=datetime.now)
+
+
+# =============================================================================
+# UPGRADE OPTIONS (Agent 3 Output)
+# =============================================================================
+
+class UpgradeOption(BaseModel):
+    """A single upgrade option."""
+    id: str
+    name: str
+    description: str
+    category: str  # "typography", "spacing", "colors", "naming"
+    
+    # The actual values
+    values: dict[str, Any] = Field(default_factory=dict)
+    
+    # Metadata
+    pros: list[str] = Field(default_factory=list)
+    cons: list[str] = Field(default_factory=list)
+    effort: str = "low"  # "low", "medium", "high"
+    recommended: bool = False
+    
+    # Selection state
+    selected: bool = False
+
+
+class UpgradeRecommendations(BaseModel):
+    """All upgrade recommendations from Agent 3."""
+    
+    # Options by category
+    typography_scales: list[UpgradeOption] = Field(default_factory=list)
+    spacing_systems: list[UpgradeOption] = Field(default_factory=list)
+    color_ramps: list[UpgradeOption] = Field(default_factory=list)
+    naming_conventions: list[UpgradeOption] = Field(default_factory=list)
+    
+    # Accessibility fixes
+    accessibility_issues: list[dict] = Field(default_factory=list)
+    accessibility_fixes: list[UpgradeOption] = Field(default_factory=list)
+    
+    # Metadata
+    generated_at: datetime = Field(default_factory=datetime.now)
+
+
+# =============================================================================
+# FINAL OUTPUT (Agent 4 Output)
+# =============================================================================
+
+class TokenMetadata(BaseModel):
+    """Metadata for exported tokens."""
+    source_url: str
+    extracted_at: datetime
+    version: str
+    viewport: Viewport
+    generator: str = "Design System Extractor v2"
+
+
+class FinalTokens(BaseModel):
+    """Final exported token set."""
+    metadata: TokenMetadata
+    
+    # Token collections
+    colors: dict[str, dict] = Field(default_factory=dict)
+    typography: dict[str, dict] = Field(default_factory=dict)
+    spacing: dict[str, dict] = Field(default_factory=dict)
+    radius: dict[str, dict] = Field(default_factory=dict)
+    shadows: dict[str, dict] = Field(default_factory=dict)
+    
+    def to_tokens_studio_format(self) -> dict:
+        """Convert to Tokens Studio compatible format."""
+        return {
+            "$metadata": {
+                "source": self.metadata.source_url,
+                "version": self.metadata.version,
+            },
+            "color": self.colors,
+            "typography": self.typography,
+            "spacing": self.spacing,
+            "borderRadius": self.radius,
+            "boxShadow": self.shadows,
+        }
+    
+    def to_css_variables(self) -> str:
+        """Convert to CSS custom properties."""
+        lines = [":root {"]
+        
+        for name, data in self.colors.items():
+            value = data.get("value", data) if isinstance(data, dict) else data
+            lines.append(f"  --color-{name}: {value};")
+        
+        for name, data in self.spacing.items():
+            value = data.get("value", data) if isinstance(data, dict) else data
+            lines.append(f"  --space-{name}: {value};")
+        
+        lines.append("}")
+        return "\n".join(lines)
+
+
+# =============================================================================
+# LANGGRAPH STATE
+# =============================================================================
+
+class WorkflowState(BaseModel):
+    """LangGraph workflow state."""
+    
+    # Input
+    base_url: str
+    
+    # Discovery phase
+    discovered_pages: list[DiscoveredPage] = Field(default_factory=list)
+    confirmed_pages: list[str] = Field(default_factory=list)
+    
+    # Extraction phase
+    desktop_tokens: Optional[ExtractedTokens] = None
+    mobile_tokens: Optional[ExtractedTokens] = None
+    
+    # Normalization phase
+    desktop_normalized: Optional[NormalizedTokens] = None
+    mobile_normalized: Optional[NormalizedTokens] = None
+    
+    # Upgrade phase
+    upgrade_recommendations: Optional[UpgradeRecommendations] = None
+    selected_upgrades: dict[str, str] = Field(default_factory=dict)  # {"typography_scale": "major_third", ...}
+    
+    # Generation phase
+    desktop_final: Optional[FinalTokens] = None
+    mobile_final: Optional[FinalTokens] = None
+    
+    # Workflow status
+    current_stage: str = "init"  # "init", "discover", "confirm", "extract", "normalize", "review", "upgrade", "generate", "export"
+    errors: list[str] = Field(default_factory=list)
+    warnings: list[str] = Field(default_factory=list)
+    
+    # Timestamps
+    started_at: Optional[datetime] = None
+    completed_at: Optional[datetime] = None
+    
+    class Config:
+        arbitrary_types_allowed = True

docs/CONTEXT.md

@@ -0,0 +1,402 @@
+# Design System Extractor v2 — Master Context File
+
+> **Upload this file to refresh Claude's context when continuing work on this project.**
+
+---
+
+## 🎯 Project Goal
+
+Build a **semi-automated, human-in-the-loop agentic system** that:
+1. Reverse-engineers a design system from a live website
+2. Reconstructs and upgrades it into a modern, scalable design system
+3. Outputs production-ready JSON tokens
+
+**Philosophy:** This is a design-aware co-pilot, NOT a magic button. Humans decide, agents propose.
+
+---
+
+## 🏗️ Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              TECH STACK                                     │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  Frontend:        Gradio (long-scroll, sectioned UI with live preview)      │
+│  Orchestration:   LangGraph (agent state management & workflow)             │
+│  Models:          HuggingFace Inference API (see model assignments below)   │
+│  Hosting:         Hugging Face Spaces                                       │
+│  Storage:         HF Spaces persistent storage                              │
+│  Output:          Platform-agnostic JSON tokens                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 🧠 Model Assignments
+
+| Agent | Role | Model | Provider | Why |
+|-------|------|-------|----------|-----|
+| **Agent 1** | Crawler & Extractor | None (Rule-based) | — | Pure CSS extraction, no LLM needed |
+| **Agent 2** | Normalizer | `microsoft/Phi-3.5-mini-instruct` | Microsoft | Fast, great structured output |
+| **Agent 3** | Advisor | `meta-llama/Llama-3.1-70B-Instruct` | Meta | Excellent reasoning, design knowledge |
+| **Agent 4** | Generator | `mistralai/Codestral-22B-v0.1` | Mistral | Code specialist, JSON formatting |
+
+### Model Presets
+
+| Preset | Agent 2 | Agent 3 | Agent 4 |
+|--------|---------|---------|---------|
+| **Budget (Free)** | Phi-3.5-mini | Mixtral-8x7B | StarCoder2-15B |
+| **Balanced (Pro)** | Gemma-2-9b | Llama-3.1-70B | Codestral-22B |
+| **Quality (Pro+)** | Gemma-2-27b | Llama-3.1-405B | DeepSeek-Coder-33B |
+| **Diverse** | Microsoft Phi | Cohere Command R+ | Mistral Codestral |
+
+### Available Providers
+- **Meta**: Llama 3.1 family (8B, 70B, 405B)
+- **Mistral**: Mixtral, Mistral, Codestral
+- **Cohere**: Command R, Command R+
+- **Google**: Gemma 2 family
+- **Microsoft**: Phi 3.5 family
+- **Alibaba**: Qwen 2.5 family
+- **DeepSeek**: DeepSeek Coder, V2.5
+- **BigCode**: StarCoder2
+
+---
+
+## 🤖 Agent Personas (4 Agents)
+
+### Agent 1: Website Crawler & Extractor
+- **Persona:** Meticulous Design Archaeologist
+- **Tool:** Playwright
+- **Job:** 
+  - Auto-discover 10+ pages from base URL
+  - Crawl Desktop (1440px) + Mobile (375px) separately
+  - Scroll to bottom + wait for network idle
+  - Extract: colors, typography, spacing, radius, shadows
+- **Output:** Raw tokens with frequency, context, confidence
+
+### Agent 2: Token Normalizer & Structurer
+- **Persona:** Design System Librarian
+- **Job:**
+  - Clean noisy extraction, dedupe
+  - Infer naming patterns
+  - Tag tokens as: `detected` | `inferred` | `low-confidence`
+- **Output:** Structured token sets with metadata
+
+### Agent 3: Design System Best Practices Advisor
+- **Persona:** Senior Staff Design Systems Architect
+- **Job:**
+  - Research modern DS patterns (Material, Polaris, Carbon, etc.)
+  - Propose upgrade OPTIONS (not decisions)
+  - Suggest: type scales (3 options), spacing (8px), color ramps (AA compliant), naming conventions
+- **Output:** Option sets with rationale
+
+### Agent 4: Plugin & JSON Generator
+- **Persona:** Automation Engineer
+- **Job:**
+  - Convert finalized tokens to Figma-compatible JSON
+  - Generate: typography, color (with tints/shades), spacing variables
+  - Maintain Desktop + Mobile + version metadata
+- **Output:** Production-ready JSON
+
+---
+
+## 🖥️ UI Stages (3 Stages)
+
+### Stage 1: Extraction Review
+- **Purpose:** Trust building
+- **Shows:** Token tables, color swatches, font previews, confidence indicators
+- **Human Actions:** Accept/reject tokens, flag anomalies, toggle Desktop↔Mobile
+
+### Stage 2: Upgrade Playground (MOST IMPORTANT)
+- **Purpose:** Decision-making through live visuals
+- **Shows:** Side-by-side option selector + live preview
+- **Human Actions:** Select type scale A/B/C, spacing system, color ramps — preview updates instantly
+
+### Stage 3: Final Review & Export
+- **Purpose:** Confidence before export
+- **Shows:** Token preview, JSON tree, diff view (original vs final)
+- **Human Actions:** Download JSON, save version, label version
+
+---
+
+## 📁 Project Structure
+
+```
+design-system-extractor/
+├── app.py                          # Gradio main entry point
+├── requirements.txt
+├── README.md
+│
+├── config/
+│   ├── .env.example                # Environment variables template
+│   ├── agents.yaml                 # Agent personas & configurations
+│   └── settings.py                 # Application settings
+│
+├── agents/
+│   ├── __init__.py
+│   ├── state.py                    # LangGraph state definitions
+│   ├── graph.py                    # LangGraph workflow orchestration
+│   ├── crawler.py                  # Agent 1: Website crawler
+│   ├── extractor.py                # Agent 1: Token extraction
+│   ├── normalizer.py               # Agent 2: Token normalization
+│   ├── advisor.py                  # Agent 3: Best practices
+│   └── generator.py                # Agent 4: JSON generator
+│
+├── core/
+│   ├── __init__.py
+│   ├── browser.py                  # Playwright browser management
+│   ├── css_parser.py               # CSS/computed style extraction
+│   ├── color_utils.py              # Color analysis, contrast, ramps
+│   ├── typography_utils.py         # Type scale detection & generation
+│   ├── spacing_utils.py            # Spacing pattern detection
+│   └── token_schema.py             # Token data structures (Pydantic)
+│
+├── ui/
+│   ├── __init__.py
+│   ├── components.py               # Reusable Gradio components
+│   ├── stage1_extraction.py        # Stage 1 UI
+│   ├── stage2_upgrade.py           # Stage 2 UI
+│   ├── stage3_export.py            # Stage 3 UI
+│   └── preview_generator.py        # HTML preview generation
+│
+├── templates/
+│   ├── preview.html                # Live preview base template
+│   └── specimen.html               # Design system specimen template
+│
+├── storage/
+│   └── persistence.py              # HF Spaces storage management
+│
+├── tests/
+│   ├── test_crawler.py
+│   ├── test_extractor.py
+│   └── test_normalizer.py
+│
+└── docs/
+    ├── CONTEXT.md                  # THIS FILE - upload for context refresh
+    └── API.md                      # API documentation
+```
+
+---
+
+## 🔧 Key Technical Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Viewports | Fixed 1440px + 375px | Simplicity, covers main use cases |
+| Scrolling | Bottom + network idle | Captures lazy-loaded content |
+| Infinite scroll | Skip | Avoid complexity |
+| Modals | Manual trigger | User decides what to capture |
+| Color ramps | 5-10 shades, AA compliant | Industry standard |
+| Type scales | 3 options (1.25, 1.333, 1.414) | User selects |
+| Spacing | 8px base system | Modern standard |
+| ML models | Minimal, rule-based preferred | Simplicity, reliability |
+| Versioning | HF Spaces persistent storage | Built-in, free |
+| Preview | Gradio + iframe (best for dynamic) | Smooth updates |
+
+---
+
+## 📊 Token Schema (Core Data Structures)
+
+```python
+class TokenSource(Enum):
+    DETECTED = "detected"       # Directly found in CSS
+    INFERRED = "inferred"       # Derived from patterns
+    UPGRADED = "upgraded"       # User-selected improvement
+    
+class Confidence(Enum):
+    HIGH = "high"               # 10+ occurrences
+    MEDIUM = "medium"           # 3-9 occurrences
+    LOW = "low"                 # 1-2 occurrences
+
+class Viewport(Enum):
+    DESKTOP = "desktop"         # 1440px
+    MOBILE = "mobile"           # 375px
+```
+
+### Token Types:
+- **ColorToken:** value, frequency, contexts, elements, contrast ratios
+- **TypographyToken:** family, size, weight, line-height, elements
+- **SpacingToken:** value, frequency, contexts, fits_base_8
+- **RadiusToken:** value, frequency, elements
+- **ShadowToken:** value, frequency, elements
+
+---
+
+## 🔄 LangGraph Workflow
+
+```
+                    ┌─────────────┐
+                    │   START     │
+                    └──────┬──────┘
+                           │
+                           ▼
+                    ┌─────────────┐
+                    │ URL Input   │
+                    └──────┬──────┘
+                           │
+                           ▼
+              ┌────────────────────────┐
+              │  Agent 1: Discover     │
+              │  (find pages)          │
+              └───────────┬────────────┘
+                          │
+                          ▼
+              ┌────────────────────────┐
+              │  HUMAN: Confirm pages  │◄─── Checkpoint 1
+              └───────────┬────────────┘
+                          │
+                          ▼
+              ┌────────────────────────┐
+              │  Agent 1: Extract      │
+              │  (crawl & extract)     │
+              └───────────┬────────────┘
+                          │
+                          ▼
+              ┌────────────────────────┐
+              │  Agent 2: Normalize    │
+              └───────────┬────────────┘
+                          │
+                          ▼
+              ┌────────────────────────┐
+              │  HUMAN: Review tokens  │◄─── Checkpoint 2 (Stage 1 UI)
+              └───────────┬────────────┘
+                          │
+          ┌───────────────┴───────────────┐
+          │                               │
+          ▼                               ▼
+┌──────────────────┐            ┌──────────────────┐
+│ Agent 3: Advise  │            │  (parallel)      │
+│ (best practices) │            │                  │
+└────────┬─────────┘            └──────────────────┘
+         │
+         ▼
+┌────────────────────────┐
+│  HUMAN: Select options │◄─── Checkpoint 3 (Stage 2 UI)
+└───────────┬────────────┘
+            │
+            ▼
+┌────────────────────────┐
+│  Agent 4: Generate     │
+│  (final JSON)          │
+└───────────┬────────────┘
+            │
+            ▼
+┌────────────────────────┐
+│  HUMAN: Export         │◄─── Checkpoint 4 (Stage 3 UI)
+└───────────┬────────────┘
+            │
+            ▼
+      ┌─────────┐
+      │   END   │
+      └─────────┘
+```
+
+---
+
+## 🚦 Human-in-the-Loop Rules
+
+1. **No irreversible automation**
+2. **Agents propose → Humans decide**
+3. **Every auto action must be:**
+   - Visible
+   - Reversible
+   - Previewed
+
+---
+
+## 📦 Output JSON Format
+
+```json
+{
+  "metadata": {
+    "source_url": "https://example.com",
+    "extracted_at": "2025-01-23T10:00:00Z",
+    "version": "v1-recovered",
+    "viewport": "desktop"
+  },
+  "colors": {
+    "primary": {
+      "50": { "value": "#e6f2ff", "source": "upgraded" },
+      "500": { "value": "#007bff", "source": "detected" },
+      "900": { "value": "#001a33", "source": "upgraded" }
+    }
+  },
+  "typography": {
+    "heading-xl": {
+      "fontFamily": "Inter",
+      "fontSize": "32px",
+      "fontWeight": 700,
+      "lineHeight": "1.2",
+      "source": "detected"
+    }
+  },
+  "spacing": {
+    "xs": { "value": "4px", "source": "upgraded" },
+    "sm": { "value": "8px", "source": "detected" },
+    "md": { "value": "16px", "source": "detected" }
+  }
+}
+```
+
+---
+
+## 🛠️ Implementation Phases
+
+### Phase 1 (Current)
+- [x] Project structure
+- [x] Configuration files
+- [ ] Token schema (Pydantic models)
+- [ ] Agent 1: Crawler
+- [ ] Agent 1: Extractor
+- [ ] Agent 2: Normalizer
+- [ ] Stage 1 UI
+- [ ] LangGraph basic workflow
+
+### Phase 2
+- [ ] Agent 3: Advisor
+- [ ] Stage 2 UI (Upgrade Playground)
+- [ ] Live preview system
+
+### Phase 3
+- [ ] Agent 4: Generator
+- [ ] Stage 3 UI
+- [ ] Export functionality
+
+### Phase 4
+- [ ] Full LangGraph orchestration
+- [ ] HF Spaces deployment
+- [ ] Persistent storage
+
+---
+
+## 🔑 Environment Variables
+
+```env
+# Required
+HF_TOKEN=your_huggingface_token
+
+# Model Configuration (defaults shown — diverse providers)
+AGENT2_MODEL=microsoft/Phi-3.5-mini-instruct      # Microsoft - Fast naming
+AGENT3_MODEL=meta-llama/Llama-3.1-70B-Instruct    # Meta - Strong reasoning
+AGENT4_MODEL=mistralai/Codestral-22B-v0.1         # Mistral - Code/JSON
+
+# Optional
+DEBUG=true
+LOG_LEVEL=INFO
+```
+
+---
+
+## 📝 Notes for Claude
+
+When continuing this project:
+1. **Check current phase** in Implementation Phases section
+2. **Review agent personas** in agents.yaml for consistent behavior
+3. **Follow token schema** defined in core/token_schema.py
+4. **Maintain LangGraph state** consistency across agents
+5. **Use Gradio components** from ui/components.py for consistency
+6. **Test with** real websites before deployment
+
+---
+
+*Last updated: 2025-01-23*

requirements.txt

@@ -0,0 +1,71 @@
+# =============================================================================
+# Design System Extractor v2 — Dependencies
+# =============================================================================
+
+# -----------------------------------------------------------------------------
+# Core Framework
+# -----------------------------------------------------------------------------
+gradio>=4.44.0
+langgraph>=0.2.0
+langchain>=0.3.0
+
+# -----------------------------------------------------------------------------
+# HuggingFace (Primary LLM Provider)
+# -----------------------------------------------------------------------------
+huggingface-hub>=0.25.0
+transformers>=4.40.0
+
+# -----------------------------------------------------------------------------
+# Data Validation & Configuration
+# -----------------------------------------------------------------------------
+pydantic>=2.0.0
+pydantic-settings>=2.0.0
+python-dotenv>=1.0.0
+PyYAML>=6.0.0
+
+# -----------------------------------------------------------------------------
+# Web Crawling & Browser Automation
+# -----------------------------------------------------------------------------
+playwright>=1.40.0
+beautifulsoup4>=4.12.0
+lxml>=5.0.0
+httpx>=0.25.0
+
+# -----------------------------------------------------------------------------
+# CSS & Color Processing
+# -----------------------------------------------------------------------------
+cssutils>=2.9.0
+colormath>=3.0.0
+colour>=0.1.5
+
+# -----------------------------------------------------------------------------
+# Data Processing
+# -----------------------------------------------------------------------------
+numpy>=1.24.0
+pandas>=2.0.0
+
+# -----------------------------------------------------------------------------
+# Async Support
+# -----------------------------------------------------------------------------
+aiofiles>=23.0.0
+
+# -----------------------------------------------------------------------------
+# Utilities
+# -----------------------------------------------------------------------------
+rich>=13.0.0
+tqdm>=4.66.0
+python-slugify>=8.0.0
+
+# -----------------------------------------------------------------------------
+# Testing (development only)
+# -----------------------------------------------------------------------------
+pytest>=7.4.0
+pytest-asyncio>=0.21.0
+pytest-cov>=4.1.0
+
+# -----------------------------------------------------------------------------
+# Type Checking (development only)
+# -----------------------------------------------------------------------------
+mypy>=1.5.0
+types-PyYAML>=6.0.0
+types-beautifulsoup4>=4.12.0