Upload 5 files

README.md

@@ -1,13 +1,129 @@
 ---
-title: Silicon Sampling Dashboard
-emoji: 🏢
+title: COGbot Silicon Sampling Dashboard
+emoji: 🤖
 colorFrom: blue
-colorTo: yellow
-sdk: gradio
-sdk_version: 6.0.1
-app_file: app.py
+colorTo: purple
+sdk: streamlit
+sdk_version: 1.28.0
+app_file: dashboard.py
 pinned: false
 license: mit
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🤖 COGbot Dashboard - Silicon Sampling
+
+Generate synthetic survey responses using AI-powered persona simulation.
+
+## 🚀 Quick Start
+
+1. **Choose your AI model** (Claude or ChatGPT)
+2. **Enter your API key** (get one from the links in the sidebar)
+3. **Write your survey question**
+4. **Generate responses** from 2,204 ESS personas
+5. **Download results** as CSV
+
+## 💡 What is Silicon Sampling?
+
+Silicon sampling uses AI to generate synthetic survey responses based on real demographic personas. Each persona is built from European Social Survey (ESS) data and includes:
+- Age, gender, education, occupation
+- Political ideology, religious attendance
+- Income, household composition
+- Regional and ethnic background
+
+## ✨ Features
+
+### Response Generation Mode
+- Generate synthetic survey responses
+- Multiple formats: Scale (0-10), Scale (1-5), Multiple Choice, Yes/No, Open Text
+- Statistical summaries (mean, median, std dev)
+- Automated thematic analysis for open text
+- Download as CSV
+
+### Question Testing Mode
+- Test draft survey questions for clarity
+- Identify ambiguous wording
+- Get improvement suggestions
+- Validate questions before real fielding
+
+## 💰 Cost
+
+This tool requires your own API key from either:
+- **Claude** (Anthropic): ~$0.015 per 50 responses [Get key →](https://console.anthropic.com/settings/keys)
+- **ChatGPT** (OpenAI): ~$0.01 per 50 responses [Get key →](https://platform.openai.com/api-keys)
+
+**Example costs:**
+- 50 responses: ~$0.01-0.015
+- 100 responses: ~$0.02-0.03
+- 500 responses: ~$0.10-0.15
+
+Your API key is only used for your session and is never stored.
+
+## 🎯 Use Cases
+
+- **Pilot Testing**: Test survey instruments before fielding
+- **Question Refinement**: Identify problematic wording
+- **Hypothesis Generation**: Explore potential response patterns
+- **Survey Methods Teaching**: Demonstrate questionnaire design
+- **Methodological Research**: Study survey question effects
+
+## 📊 Sample Data
+
+Based on **European Social Survey Round 9 UK data (2018)**:
+- 2,204 respondents
+- Representative UK demographics
+- Rich persona backstories
+
+## 🔒 Privacy & Security
+
+- API keys are never logged or stored
+- Used only for your current session
+- Data sent only to your chosen AI provider
+- No retention after session ends
+
+## 📚 How It Works
+
+1. **Persona Loading**: Each respondent has a detailed backstory
+2. **AI Prompting**: Backstory becomes the AI's "persona"
+3. **Question Answering**: AI responds as that persona would
+4. **Aggregation**: Responses collected and analyzed
+
+## 🎓 Citation
+
+Based on European Social Survey Round 9 UK data (2018).
+
+ESS Round 9: European Social Survey Round 9 Data (2018). Data file edition 3.1. Sikt - Norwegian Agency for Shared Services in Education and Research, Norway – Data Archive and distributor of ESS data for ESS ERIC. doi:10.21338/NSD-ESS9-2018.
+
+## 📖 Documentation
+
+- [Full Documentation](https://github.com/PatrickSturgis/Silicon_samples)
+- [Methodology Paper](https://github.com/PatrickSturgis/Silicon_samples)
+- [GitHub Repository](https://github.com/PatrickSturgis/Silicon_samples)
+
+## ⚠️ Important Notes
+
+- Synthetic responses are for research/testing purposes only
+- Should complement, not replace, real survey data
+- Best used for question development and pilot testing
+- Response quality depends on persona detail and AI model
+
+## 🛠️ Technical Details
+
+- Built with Streamlit
+- Supports Claude 3.5 Sonnet and GPT-4o-mini
+- Processes 50 responses in ~1-2 minutes
+- CSV export with all demographic variables
+
+## 📧 Contact & Support
+
+- **GitHub Issues**: [Report bugs or request features](https://github.com/PatrickSturgis/Silicon_samples/issues)
+- **Research Inquiries**: Via GitHub
+- **Educational Use**: Free for academic purposes
+
+## 📄 License
+
+MIT License - Free for research and educational use.
+
+---
+
+**Developed by**: Patrick Sturgis, LSE Department of Methodology  
+**Powered by**: Anthropic Claude & OpenAI GPT

dashboard.py

@@ -0,0 +1,726 @@
+#!/usr/bin/env python3
+"""
+Silicon Sampling Dashboard
+
+Interactive web interface for generating synthetic survey responses.
+Users can input custom questions and get silicon sample data without coding.
+
+Usage:
+    streamlit run dashboard.py
+"""
+
+import streamlit as st
+import pandas as pd
+from pathlib import Path
+import json
+from datetime import datetime
+import os
+from dashboard_backend import SiliconSampler, WinstonSampler, HuggingFaceSampler, OpenAISampler, AnthropicSampler
+
+# Check deployment mode (set PUBLIC_DEPLOYMENT=true for HuggingFace/public hosting)
+IS_PUBLIC = os.getenv('PUBLIC_DEPLOYMENT', 'false').lower() == 'true'
+
+# Page configuration
+st.set_page_config(
+    page_title="COGbot Dashboard",
+    page_icon="🤖",
+    layout="wide"
+)
+
+# Initialize session state
+if 'results' not in st.session_state:
+    st.session_state.results = None
+if 'processing' not in st.session_state:
+    st.session_state.processing = False
+if 'mode' not in st.session_state:
+    st.session_state.mode = "Response Generation"
+if 'question_text' not in st.session_state:
+    st.session_state.question_text = ""
+if 'response_options_text' not in st.session_state:
+    st.session_state.response_options_text = ""
+
+# Title and description
+st.title("🤖 COGbot Dashboard")
+st.markdown("""
+Generate synthetic survey responses using LLM-based persona simulation.
+Enter your question and response format - we'll handle the rest.
+""")
+
+# Sidebar - Logo and Configuration
+# Display LSE logo at top of sidebar
+logo_path = "LSE_logo.jpg"
+if Path(logo_path).exists():
+    st.sidebar.image(logo_path, width=180)
+    st.sidebar.markdown("---")
+
+st.sidebar.header("⚙️ Configuration")
+
+# Data source
+data_source = st.sidebar.radio(
+    "Data Source",
+    ["Default ESS UK (1,286 respondents)", "Upload CSV"]
+)
+
+if data_source == "Upload CSV":
+    uploaded_file = st.sidebar.file_uploader(
+        "Upload backstories CSV",
+        type=['csv'],
+        help="CSV must have 'backstory' column"
+    )
+    if uploaded_file:
+        df_backstories = pd.read_csv(uploaded_file)
+    else:
+        df_backstories = None
+else:
+    # Load default ESS data
+    default_path = Path("ess_uk_with_backstories.csv")
+    if default_path.exists():
+        df_backstories = pd.read_csv(default_path)
+    else:
+        df_backstories = None
+        st.sidebar.warning("⚠️ Default file not found: ess_uk_with_backstories.csv")
+
+# Show data info
+if df_backstories is not None:
+    st.sidebar.success(f"✅ Loaded {len(df_backstories):,} respondents")
+
+    # Sample size
+    max_size = len(df_backstories)
+    sample_size = st.sidebar.slider(
+        "Sample Size",
+        min_value=10,
+        max_value=max_size,
+        value=min(50, max_size),
+        step=10,
+        help="Start with small sample for testing"
+    )
+else:
+    sample_size = 0
+
+# Model settings
+st.sidebar.subheader("Model Settings")
+
+# Choose model options based on deployment mode
+if IS_PUBLIC:
+    # Public deployment: Only show API-based models
+    model_options = ["Claude (Claude 3.5 Sonnet)", "ChatGPT (GPT-4o-mini)"]
+    st.sidebar.info("""
+    💡 **About API Keys**
+
+    This tool uses AI models via API. You'll need to provide your own API key:
+    - **Claude**: ~$0.015 per 50 responses (recommended for quality)
+    - **ChatGPT**: ~$0.01 per 50 responses (faster, good quality)
+
+    Your API key is used only for your session and is never stored.
+    """)
+else:
+    # Local deployment: Show all options including local models
+    model_options = ["Claude (Claude 3.5 Sonnet)", "ChatGPT (GPT-4o-mini)", "Local (SmolLM2-1.7B)", "Winston (Qwen2.5-7B)"]
+
+model_option = st.sidebar.selectbox(
+    "Model",
+    model_options,
+    help="Choose your AI model. API models require your own API key."
+)
+
+# API key inputs based on selected model
+openai_api_key = None
+anthropic_api_key = None
+
+if "Claude" in model_option:
+    anthropic_api_key = st.sidebar.text_input(
+        "Anthropic API Key",
+        type="password",
+        help="Get your API key from https://console.anthropic.com/settings/keys"
+    )
+    if not anthropic_api_key:
+        st.sidebar.warning("⚠️ API key required for Claude")
+    else:
+        st.sidebar.success("✅ API key provided")
+        st.sidebar.markdown("[Get API key →](https://console.anthropic.com/settings/keys)")
+
+elif "ChatGPT" in model_option:
+    openai_api_key = st.sidebar.text_input(
+        "OpenAI API Key",
+        type="password",
+        help="Get your API key from https://platform.openai.com/api-keys"
+    )
+    if not openai_api_key:
+        st.sidebar.warning("⚠️ API key required for ChatGPT")
+    else:
+        st.sidebar.success("✅ API key provided")
+        st.sidebar.markdown("[Get API key →](https://platform.openai.com/api-keys)")
+
+temperature = st.sidebar.slider(
+    "Temperature",
+    min_value=0.0,
+    max_value=1.0,
+    value=0.7,
+    step=0.1,
+    help="Higher = more creative, Lower = more consistent"
+)
+
+# Main panel - Question configuration
+st.header("📋 Step 1: Configure Question")
+
+# Mode selection: Response Generation vs Question Testing
+mode = st.radio(
+    "Mode",
+    ["Response Generation", "Question Testing"],
+    help="Response Generation: Get synthetic survey responses. Question Testing: Get feedback on question quality."
+)
+
+col1, col2 = st.columns([2, 1])
+
+with col1:
+    question_text = st.text_area(
+        "Survey Question",
+        height=150,
+        placeholder="Enter your survey question here...",
+        help="The question your synthetic respondents will answer" if mode == "Response Generation" else "The draft question you want to test for clarity and quality"
+    )
+
+with col2:
+    if mode == "Response Generation":
+        response_format = st.selectbox(
+            "Response Format",
+            ["Scale (0-10)", "Scale (1-5)", "Multiple Choice", "Yes/No", "Open Text"]
+        )
+    else:  # Question Testing mode
+        response_format = "Open Text"
+        st.info("📝 Question Testing uses open text responses to gather feedback on question quality.")
+
+# Configure prompt based on mode
+# Initialize variables that will be used in preview
+mc_options = ""
+response_options_text = ""
+
+if mode == "Question Testing":
+    # Question Testing mode: Create critique prompt
+    st.subheader("Response Options/Instructions")
+    response_options_text = st.text_area(
+        "Response Options (if applicable)",
+        height=100,
+        placeholder="e.g., Scale from 0-10 where 0=Not at all, 10=Extremely, or Multiple choice options A, B, C, D",
+        help="Include any response options or scales that are part of the question being tested"
+    )
+
+    # Build the testing prompt
+    instructions = f"""Please provide feedback on this survey question. Comment on:
+
+1. Are there any parts of the question that are ambiguous or unclear?
+2. Are there any parts that are difficult to understand?
+3. Did you have any problems thinking about how to answer?
+4. Are the response options (if provided) appropriate and complete?
+
+Provide your feedback in 2-3 sentences, being specific about any issues you identify."""
+
+    # Automatically enable thematic coding for Question Testing
+    enable_thematic_coding = True
+    st.info("🔍 Thematic analysis will automatically run to identify common issues in the question.")
+
+else:
+    # Response Generation mode: Original behavior
+    # Scale anchor labels (if scale selected)
+    if "Scale" in response_format:
+        st.subheader("Scale Labels")
+
+        if "0-10" in response_format:
+            # 10-point scale: just endpoints
+            col_low, col_high = st.columns(2)
+            with col_low:
+                low_label = st.text_input(
+                    "0 means",
+                    value="Not at all",
+                    help="What does the lowest value mean?"
+                )
+            with col_high:
+                high_label = st.text_input(
+                    "10 means",
+                    value="Extremely",
+                    help="What does the highest value mean?"
+                )
+            instructions = f"Respond with a single integer from 0 to 10, where 0 means '{low_label}' and 10 means '{high_label}'. Only output the number."
+
+        else:  # 1-5 scale: label all 5 points
+            label_1 = st.text_input("1 means", value="Strongly disagree")
+            label_2 = st.text_input("2 means", value="Disagree")
+            label_3 = st.text_input("3 means", value="Neither agree nor disagree")
+            label_4 = st.text_input("4 means", value="Agree")
+            label_5 = st.text_input("5 means", value="Strongly agree")
+
+            instructions = f"""Respond with a single integer from 1 to 5 based on these labels:
+1 = {label_1}
+2 = {label_2}
+3 = {label_3}
+4 = {label_4}
+5 = {label_5}
+
+Only output the number."""
+    else:
+        # Non-scale formats
+        format_instructions = {
+            "Multiple Choice": "Choose one option and respond with only the letter (A, B, C, or D).",
+            "Yes/No": "Respond with only 'Yes' or 'No'.",
+            "Open Text": "Provide a brief 1-2 sentence response based on your persona."
+        }
+        instructions = format_instructions.get(response_format, "")
+
+    # Allow editing instructions
+    instructions = st.text_area(
+        "Instructions to Model",
+        value=instructions,
+        height=100,
+        help="How the model should format its response"
+    )
+
+    # Multiple choice options (if selected)
+    if response_format == "Multiple Choice":
+        st.subheader("Response Options")
+        col1, col2, col3, col4 = st.columns(4)
+        with col1:
+            option_a = st.text_input("Option A", "Strongly agree")
+        with col2:
+            option_b = st.text_input("Option B", "Agree")
+        with col3:
+            option_c = st.text_input("Option C", "Disagree")
+        with col4:
+            option_d = st.text_input("Option D", "Strongly disagree")
+
+        mc_options = f"\nA. {option_a}\nB. {option_b}\nC. {option_c}\nD. {option_d}"
+    else:
+        mc_options = ""
+
+    # Thematic coding option (if open text selected)
+    enable_thematic_coding = False
+    if response_format == "Open Text":
+        st.subheader("Thematic Coding")
+        enable_thematic_coding = st.checkbox(
+            "Perform automated thematic analysis after generating responses",
+            value=False,
+            help="Uses LLM to identify themes, counts, and percentages in open text responses. Runs automatically after response generation."
+        )
+
+# Preview full prompt
+with st.expander("🔍 Preview Full Prompt"):
+    st.markdown("**System Prompt:**")
+    st.code("""Adopt the following persona and answer only based on it.
+Do not invent details beyond the provided attributes.
+
+[Backstory will be inserted here for each respondent]""")
+
+    st.markdown("**User Prompt:**")
+    if mode == "Question Testing":
+        # Include response options in the question display for testing
+        full_question = f"Question: {question_text}\n"
+        if response_options_text.strip():
+            full_question += f"\nResponse Options: {response_options_text}\n"
+        full_question += f"\n{instructions}"
+    else:
+        full_question = question_text + mc_options + "\n\n" + instructions
+    st.code(full_question)
+
+# Generate button
+if mode == "Question Testing":
+    st.header("🧪 Step 2: Test Question")
+    button_text = "🧪 Test Question with Synthetic Respondents"
+else:
+    st.header("🚀 Step 2: Generate Responses")
+    button_text = "🎯 Generate Responses"
+
+can_generate = (
+    df_backstories is not None
+    and question_text.strip() != ""
+    and not st.session_state.processing
+    and (not ("Claude" in model_option) or anthropic_api_key)  # Require API key for Claude
+    and (not ("ChatGPT" in model_option) or openai_api_key)  # Require API key for ChatGPT
+)
+
+if st.button(
+    button_text,
+    disabled=not can_generate,
+    type="primary",
+    use_container_width=True
+):
+    st.session_state.processing = True
+    st.session_state.results = None
+    st.session_state.mode = mode  # Store mode for results display
+    st.session_state.question_text = question_text  # Store for thematic analysis
+    if mode == "Question Testing":
+        st.session_state.response_options_text = response_options_text  # Store for improved version
+
+    # Prepare configuration
+    config = {
+        "question": full_question,
+        "temperature": temperature,
+        "sample_size": sample_size
+    }
+
+    # Create sampler based on model selection
+    if "Claude" in model_option:
+        config["model_type"] = "anthropic"
+        config["anthropic_api_key"] = anthropic_api_key
+        sampler = AnthropicSampler(config)
+    elif "ChatGPT" in model_option:
+        config["model_type"] = "openai"
+        config["openai_api_key"] = openai_api_key
+        sampler = OpenAISampler(config)
+    elif "Winston" in model_option:
+        config["model_type"] = "winston"
+        sampler = WinstonSampler(config)
+    else:  # Local
+        config["model_type"] = "local"
+        sampler = SiliconSampler(config)
+
+    # Progress bar
+    progress_bar = st.progress(0)
+    status_text = st.empty()
+
+    # Sample backstories (random sample)
+    df_sample = df_backstories.sample(n=sample_size, random_state=42).copy()
+
+    # Process
+    try:
+        results = sampler.generate_responses(
+            df_sample,
+            progress_callback=lambda i, total: (
+                progress_bar.progress(i / total),
+                status_text.text(f"Processing: {i}/{total} respondents ({100*i/total:.1f}%)")
+            )
+        )
+
+        st.session_state.results = results
+        st.session_state.processing = False
+        st.success(f"✅ Generated {len(results)} responses!")
+        st.rerun()
+
+    except Exception as e:
+        st.error(f"❌ Error: {str(e)}")
+        st.session_state.processing = False
+
+# Show results
+if st.session_state.results is not None:
+    st.header("📊 Step 3: Results")
+
+    results_df = st.session_state.results
+
+    # Summary stats
+    col1, col2, col3 = st.columns(3)
+    with col1:
+        st.metric("Total Responses", len(results_df))
+    with col2:
+        valid_responses = results_df['response'].notna().sum()
+        st.metric("Valid Responses", valid_responses)
+    with col3:
+        completion_rate = 100 * valid_responses / len(results_df)
+        st.metric("Completion Rate", f"{completion_rate:.1f}%")
+
+    # Preview
+    st.subheader("Preview (First 10 rows)")
+    st.dataframe(results_df.head(10), use_container_width=True)
+
+    # Download
+    st.subheader("Download Results")
+
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    filename = f"silicon_sample_{timestamp}.csv"
+
+    csv = results_df.to_csv(index=False)
+    st.download_button(
+        label="📥 Download CSV",
+        data=csv,
+        file_name=filename,
+        mime="text/csv",
+        use_container_width=True
+    )
+
+    # Response distribution and statistics
+    if response_format in ["Scale (0-10)", "Scale (1-5)", "Yes/No", "Multiple Choice"]:
+        st.subheader(f"Response Distribution: {question_text}")
+        try:
+            # For numeric formats, convert to numbers
+            if response_format.startswith("Scale"):
+                numeric_responses = pd.to_numeric(results_df['response'], errors='coerce')
+                valid_responses = numeric_responses.dropna()
+            elif response_format == "Yes/No":
+                # For Yes/No, show frequency distribution
+                valid_responses = results_df['response'].dropna()
+            elif response_format == "Multiple Choice":
+                # For Multiple Choice, show frequency distribution
+                valid_responses = results_df['response'].dropna()
+
+            if len(valid_responses) > 0:
+                # Show statistics for numeric scales
+                if response_format.startswith("Scale"):
+                    col1, col2, col3, col4, col5 = st.columns(5)
+
+                    with col1:
+                        st.metric("Mean", f"{valid_responses.mean():.2f}")
+                    with col2:
+                        st.metric("Median", f"{valid_responses.median():.2f}")
+                    with col3:
+                        st.metric("Std Dev", f"{valid_responses.std():.2f}")
+                    with col4:
+                        mode_val = valid_responses.mode()
+                        mode_display = f"{mode_val.iloc[0]:.0f}" if len(mode_val) > 0 else "N/A"
+                        st.metric("Mode", mode_display)
+                    with col5:
+                        st.metric("Valid N", f"{len(valid_responses)}")
+
+                    # Distribution chart
+                    st.bar_chart(pd.to_numeric(results_df['response'], errors='coerce').value_counts().sort_index())
+
+                # Show frequency counts for categorical
+                else:
+                    value_counts = valid_responses.value_counts()
+
+                    # Display as metrics
+                    cols = st.columns(min(len(value_counts), 5))
+                    for idx, (value, count) in enumerate(value_counts.items()):
+                        if idx < 5:  # Limit to 5 columns
+                            with cols[idx]:
+                                pct = 100 * count / len(valid_responses)
+                                st.metric(f"{value}", f"{count} ({pct:.1f}%)")
+
+                    # Also show total N
+                    st.metric("Total Valid N", f"{len(valid_responses)}")
+
+                    # Distribution chart
+                    st.bar_chart(value_counts)
+            else:
+                st.info("No valid responses to analyze")
+        except Exception as e:
+            st.info(f"Could not generate statistics: {str(e)}")
+
+    # Thematic coding for open text responses
+    elif response_format == "Open Text" and enable_thematic_coding:
+        # Get the stored mode and question text
+        stored_mode = st.session_state.get('mode', 'Response Generation')
+        stored_question = st.session_state.get('question_text', question_text)
+
+        # Different heading based on mode
+        if stored_mode == "Question Testing":
+            st.subheader(f"Question Testing Results: {stored_question}")
+        else:
+            st.subheader(f"Thematic Analysis: {stored_question}")
+
+        # Get valid text responses
+        valid_responses = results_df['response'].dropna()
+        valid_responses = valid_responses[valid_responses.str.strip() != ""]
+
+        if len(valid_responses) > 0:
+            st.info(f"Analyzing {len(valid_responses)} open text responses...")
+
+            # Automatically run thematic coding
+            if True:  # Changed from button to automatic
+                with st.spinner("Analyzing themes with LLM..."):
+                    try:
+                        # Prepare responses for analysis
+                        responses_text = "\n\n".join([f"Response {i+1}: {resp}" for i, resp in enumerate(valid_responses)])
+
+                        # Create thematic analysis prompt - different for Question Testing
+                        if stored_mode == "Question Testing":
+                            coding_prompt = f"""You are a survey methodology expert analyzing feedback from respondents who tested a draft survey question.
+
+Question being tested: "{stored_question}"
+
+Here is the feedback from respondents:
+
+{responses_text}
+
+Task:
+1. Identify the main issues and concerns raised about the question (aim for 4-8 distinct issues)
+2. For each issue, provide:
+   - Issue name (2-4 words, e.g., "Ambiguous wording", "Unclear scale", "Missing context")
+   - Brief description (1 sentence explaining the specific problem)
+   - Count of how many respondents mentioned this issue
+   - Percentage of total respondents
+
+Format your response as:
+ISSUE: [Name]
+DESCRIPTION: [Description]
+COUNT: [Number]
+PERCENTAGE: [Percentage]
+
+[Repeat for each issue]
+
+After listing all issues, provide a brief summary of the most critical problems that should be addressed."""
+                        else:
+                            coding_prompt = f"""You are a qualitative researcher conducting thematic analysis on open-ended survey responses.
+
+Question asked: "{stored_question}"
+
+Here are all the responses:
+
+{responses_text}
+
+Task:
+1. Identify the main themes present in these responses (aim for 4-8 themes)
+2. For each theme, provide:
+   - Theme name (2-4 words)
+   - Brief description (1 sentence)
+   - Count of how many responses express this theme
+   - Percentage of total responses
+
+Format your response as:
+THEME: [Name]
+DESCRIPTION: [Description]
+COUNT: [Number]
+PERCENTAGE: [Percentage]
+
+[Repeat for each theme]"""
+
+                        # Send to LLM for coding
+                        if "Claude" in model_option:
+                            # Use Anthropic sampler
+                            from dashboard_backend import AnthropicSampler
+                            temp_config = {
+                                "temperature": 0.3,  # Lower temp for more consistent coding
+                                "model_type": "anthropic",
+                                "anthropic_api_key": anthropic_api_key
+                            }
+                            temp_sampler = AnthropicSampler(temp_config)
+
+                            st.info("Sending to Claude for analysis...")
+
+                            # Query Anthropic
+                            analysis_result = temp_sampler.query_single(
+                                "You are a qualitative research expert analyzing survey responses.",
+                                coding_prompt
+                            )
+
+                        elif "ChatGPT" in model_option:
+                            # Use OpenAI sampler
+                            from dashboard_backend import OpenAISampler
+                            temp_config = {
+                                "temperature": 0.3,  # Lower temp for more consistent coding
+                                "model_type": "openai",
+                                "openai_api_key": openai_api_key
+                            }
+                            temp_sampler = OpenAISampler(temp_config)
+
+                            st.info("Sending to ChatGPT for analysis...")
+
+                            # Query OpenAI
+                            analysis_result = temp_sampler.query_single(
+                                "You are a qualitative research expert analyzing survey responses.",
+                                coding_prompt
+                            )
+
+                        elif "Winston" in model_option:
+                            # Use Winston sampler with single query method
+                            from dashboard_backend import WinstonSampler
+                            temp_config = {
+                                "temperature": 0.3,  # Lower temp for more consistent coding
+                                "model_type": "winston"
+                            }
+                            temp_sampler = WinstonSampler(temp_config)
+
+                            st.info("Sending to Winston for analysis... This may take 1-2 minutes (includes model loading time).")
+
+                            # Query Winston
+                            analysis_result = temp_sampler.query_single(
+                                "You are a qualitative research expert analyzing survey responses.",
+                                coding_prompt
+                            )
+
+                        else:
+                            # Use local model
+                            from dashboard_backend import SiliconSampler
+                            temp_config = {
+                                "question": coding_prompt,
+                                "temperature": 0.3,
+                                "model_type": "local"
+                            }
+                            temp_sampler = SiliconSampler(temp_config)
+                            temp_sampler._initialize_local_model()
+
+                            # Query with analysis prompt
+                            analysis_result = temp_sampler.query_llm(
+                                "You are a qualitative research expert analyzing survey responses.",
+                                coding_prompt
+                            )
+
+                        # Display results
+                        st.markdown("### Thematic Coding Results")
+                        st.text_area("Analysis", analysis_result, height=400)
+
+                        # For Question Testing mode, offer to suggest improved wording
+                        if stored_mode == "Question Testing":
+                            st.markdown("---")
+                            st.markdown("### Suggest Improved Question Wording")
+
+                            if st.button("✨ Generate Improved Question", type="secondary"):
+                                with st.spinner("Generating improved question wording..."):
+                                    try:
+                                        # Get response options if they exist
+                                        stored_options = st.session_state.get('response_options_text', '')
+
+                                        # Create improvement prompt
+                                        # Build the options section separately to avoid f-string backslash issue
+                                        options_section = f"\nOriginal Response Options: {stored_options}\n" if stored_options else ""
+                                        improved_options_section = "\n\nIMPROVED RESPONSE OPTIONS:\n[Your improved options]\n" if stored_options else ""
+
+                                        improvement_prompt = f"""You are a survey methodology expert. Based on the feedback analysis below, suggest an improved version of the survey question that addresses the identified issues.
+
+Original Question: "{stored_question}"{options_section}
+
+Issues Identified:
+{analysis_result}
+
+Task:
+1. Provide an improved version of the question that addresses the main issues
+2. If response options were provided, suggest improved response options as well
+3. Explain what changes you made and why they address the identified problems
+
+Format your response as:
+
+IMPROVED QUESTION:
+[Your improved question text]{improved_options_section}
+
+CHANGES MADE:
+[Brief explanation of what you changed and why]"""
+
+                                        # Send to same model that was used for analysis
+                                        if "Claude" in model_option:
+                                            improvement_result = temp_sampler.query_single(
+                                                "You are a survey methodology expert specializing in question wording and design.",
+                                                improvement_prompt
+                                            )
+                                        elif "ChatGPT" in model_option:
+                                            improvement_result = temp_sampler.query_single(
+                                                "You are a survey methodology expert specializing in question wording and design.",
+                                                improvement_prompt
+                                            )
+                                        elif "Winston" in model_option:
+                                            improvement_result = temp_sampler.query_single(
+                                                "You are a survey methodology expert specializing in question wording and design.",
+                                                improvement_prompt
+                                            )
+                                        else:
+                                            improvement_result = temp_sampler.query_llm(
+                                                "You are a survey methodology expert specializing in question wording and design.",
+                                                improvement_prompt
+                                            )
+
+                                        # Display improved version
+                                        st.markdown("### Improved Question Suggestion")
+                                        st.text_area("Suggested Improvements", improvement_result, height=300)
+
+                                        st.info("💡 Review the suggested improvements and adapt them as needed for your research context.")
+
+                                    except Exception as e:
+                                        st.error(f"Error generating improved question: {str(e)}")
+
+                    except Exception as e:
+                        st.error(f"Error during thematic analysis: {str(e)}")
+        else:
+            st.info("No valid open text responses to analyze")
+
+# Footer
+st.sidebar.markdown("---")
+st.sidebar.markdown("""
+**Need Help?**
+- [Documentation](WINSTON_README.md)
+- [GitHub](https://github.com/PatrickSturgis/Silicon_samples)
+""")

dashboard_backend.py

@@ -0,0 +1,871 @@
+#!/usr/bin/env python3
+"""
+Dashboard Backend - Silicon Sampling Processing
+
+Handles LLM querying and response generation for the dashboard.
+Supports both local (lightweight) and Winston (production) modes.
+"""
+
+import pandas as pd
+import torch
+from typing import Callable, Optional
+import time
+import os
+
+# Set HuggingFace cache to a writable location
+os.environ['HF_HOME'] = os.path.expanduser('~/Library/Caches/huggingface')
+os.environ['TRANSFORMERS_CACHE'] = os.path.expanduser('~/Library/Caches/huggingface')
+
+class SiliconSampler:
+    """
+    Silicon sampling backend for dashboard
+
+    Supports:
+    - Local mode: Quick testing with small models
+    - Winston mode: Production quality with Qwen2.5 (future)
+    """
+
+    def __init__(self, config: dict):
+        """
+        Initialize sampler
+
+        Args:
+            config: Dictionary with:
+                - question: Survey question text
+                - temperature: Sampling temperature
+                - sample_size: Number of respondents
+                - model_type: "local" or "winston"
+        """
+        self.config = config
+        self.llm = None
+        self.model = None
+        self.tokenizer = None
+        self.device = None
+        self.model_loaded = False
+
+        # Don't load model in __init__ - load lazily on first use
+
+    def _initialize_local_model(self):
+        """Initialize lightweight local model for testing"""
+        try:
+            from transformers import AutoTokenizer, AutoModelForCausalLM
+
+            # Use SmolLM2-1.7B-Instruct for better quality
+            model_name = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
+
+            print(f"Loading model: {model_name}")
+
+            self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+            self.device = "cuda" if torch.cuda.is_available() else "cpu"
+
+            self.model = AutoModelForCausalLM.from_pretrained(
+                model_name,
+                torch_dtype=torch.float32,  # Use float32 for CPU compatibility
+                low_cpu_mem_usage=True
+            )
+
+            self.model = self.model.to(self.device)
+
+            print(f"✅ Model loaded on {self.device}")
+
+        except Exception as e:
+            print(f"Error loading model: {e}")
+            raise
+
+    def query_llm(self, backstory: str, question: str) -> str:
+        """
+        Query LLM with backstory and question
+
+        Args:
+            backstory: Persona backstory text
+            question: Survey question
+
+        Returns:
+            Model response
+        """
+        # Lazy load model on first query
+        if not self.model_loaded and self.config['model_type'] == 'local':
+            self._initialize_local_model()
+            self.model_loaded = True
+
+        messages = [
+            {
+                "role": "system",
+                "content": (
+                    "Adopt the following persona and answer only based on it. "
+                    "Do not invent details beyond the provided attributes.\n\n"
+                    f"{backstory}"
+                )
+            },
+            {
+                "role": "user",
+                "content": question
+            }
+        ]
+
+        # Format using chat template (same as working job assessment code)
+        formatted_prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            add_generation_prompt=True
+        )
+
+        # Tokenize
+        inputs = self.tokenizer(
+            formatted_prompt,
+            return_tensors="pt",
+            truncation=True,
+            max_length=2048
+        ).to(self.device)
+
+        # Generate (matching working parameters)
+        with torch.no_grad():
+            outputs = self.model.generate(
+                **inputs,
+                max_new_tokens=100,
+                temperature=self.config['temperature'],
+                top_p=1.0,
+                do_sample=True if self.config['temperature'] > 0 else False,
+                pad_token_id=self.tokenizer.pad_token_id,
+                eos_token_id=self.tokenizer.eos_token_id
+            )
+
+        # Decode
+        generated_tokens = outputs[0][inputs['input_ids'].shape[1]:]
+        response = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()
+
+        return response
+
+    def generate_responses(
+        self,
+        df: pd.DataFrame,
+        progress_callback: Optional[Callable[[int, int], None]] = None
+    ) -> pd.DataFrame:
+        """
+        Generate responses for all backstories in DataFrame
+
+        Args:
+            df: DataFrame with 'backstory' column
+            progress_callback: Optional function(current, total) for progress updates
+
+        Returns:
+            DataFrame with original columns plus 'response' column
+        """
+        if 'backstory' not in df.columns:
+            raise ValueError("DataFrame must have 'backstory' column")
+
+        results = df.copy()
+        results['response'] = ""
+
+        question = self.config['question']
+        total = len(df)
+
+        for i, (idx, row) in enumerate(df.iterrows()):
+            backstory = row['backstory']
+
+            # Skip empty backstories
+            if pd.isna(backstory) or str(backstory).strip() == "":
+                results.loc[idx, 'response'] = "[EMPTY]"
+                continue
+
+            try:
+                # Query LLM
+                response = self.query_llm(str(backstory), question)
+                results.loc[idx, 'response'] = response
+
+            except Exception as e:
+                results.loc[idx, 'response'] = f"[ERROR: {str(e)[:50]}]"
+
+            # Progress callback
+            if progress_callback:
+                progress_callback(i + 1, total)
+
+            # Small delay to prevent overheating on CPU
+            if self.device == "cpu":
+                time.sleep(0.1)
+
+        return results
+
+
+class HuggingFaceSampler:
+    """
+    Hugging Face Inference API sampler
+
+    Uses HF's free Inference API to access larger models without local compute.
+    Requires HF_TOKEN environment variable or passed in config.
+    """
+
+    def __init__(self, config: dict):
+        self.config = config
+        self.api_token = config.get('hf_token') or os.getenv('HF_TOKEN')
+        # Use Meta's Llama 3.2 which is freely accessible via Inference API
+        self.model_name = config.get('hf_model', 'meta-llama/Llama-3.2-3B-Instruct')
+
+        if not self.api_token:
+            raise ValueError(
+                "Hugging Face API token required. Set HF_TOKEN environment variable "
+                "or pass 'hf_token' in config. Get token from: https://huggingface.co/settings/tokens"
+            )
+
+    def query_llm(self, backstory: str, question: str) -> str:
+        """Query HF Inference API using direct HTTP requests"""
+        import requests
+
+        # Format the prompt for the model
+        prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+Adopt the following persona and answer only based on it. Do not invent details beyond the provided attributes.
+
+{backstory}<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+"""
+
+        # Use the new serverless inference API endpoint
+        api_url = f"https://api-inference.huggingface.co/models/{self.model_name}"
+
+        headers = {
+            "Authorization": f"Bearer {self.api_token}",
+            "Content-Type": "application/json"
+        }
+
+        payload = {
+            "inputs": prompt,
+            "parameters": {
+                "max_new_tokens": 100,
+                "temperature": self.config['temperature'],
+                "return_full_text": False
+            }
+        }
+
+        try:
+            response = requests.post(api_url, headers=headers, json=payload, timeout=30)
+
+            if response.status_code == 200:
+                result = response.json()
+                if isinstance(result, list) and len(result) > 0:
+                    return result[0].get('generated_text', '').strip()
+                else:
+                    return str(result).strip()
+            else:
+                return f"[API_ERROR: {response.status_code} - {response.text[:100]}]"
+
+        except Exception as e:
+            return f"[API_ERROR: {str(e)[:100]}]"
+
+    def generate_responses(
+        self,
+        df: pd.DataFrame,
+        progress_callback: Optional[Callable[[int, int], None]] = None
+    ) -> pd.DataFrame:
+        """Generate responses using HF Inference API"""
+
+        if 'backstory' not in df.columns:
+            raise ValueError("DataFrame must have 'backstory' column")
+
+        results = df.copy()
+        results['response'] = ""
+
+        question = self.config['question']
+        total = len(df)
+
+        for i, (idx, row) in enumerate(df.iterrows()):
+            backstory = row['backstory']
+
+            if pd.isna(backstory) or str(backstory).strip() == "":
+                results.loc[idx, 'response'] = "[EMPTY]"
+                continue
+
+            try:
+                response = self.query_llm(str(backstory), question)
+                results.loc[idx, 'response'] = response
+
+            except Exception as e:
+                results.loc[idx, 'response'] = f"[ERROR: {str(e)[:50]}]"
+
+            if progress_callback:
+                progress_callback(i + 1, total)
+
+            # Small delay to avoid rate limiting
+            time.sleep(0.5)
+
+        return results
+
+
+class OpenAISampler:
+    """
+    OpenAI API sampler (ChatGPT)
+
+    Uses OpenAI's API to access GPT models.
+    Requires OPENAI_API_KEY environment variable or passed in config.
+    """
+
+    def __init__(self, config: dict):
+        self.config = config
+        self.api_key = config.get('openai_api_key') or os.getenv('OPENAI_API_KEY')
+        # Use GPT-4o-mini by default (fast and cost-effective)
+        self.model_name = config.get('openai_model', 'gpt-4o-mini')
+
+        if not self.api_key:
+            raise ValueError(
+                "OpenAI API key required. Set OPENAI_API_KEY environment variable "
+                "or pass 'openai_api_key' in config. Get key from: https://platform.openai.com/api-keys"
+            )
+
+    def query_llm(self, backstory: str, question: str) -> str:
+        """Query OpenAI API"""
+        import requests
+
+        api_url = "https://api.openai.com/v1/chat/completions"
+
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json"
+        }
+
+        messages = [
+            {
+                "role": "system",
+                "content": (
+                    "Adopt the following persona and answer only based on it. "
+                    "Do not invent details beyond the provided attributes.\n\n"
+                    f"{backstory}"
+                )
+            },
+            {
+                "role": "user",
+                "content": question
+            }
+        ]
+
+        payload = {
+            "model": self.model_name,
+            "messages": messages,
+            "temperature": self.config['temperature'],
+            "max_tokens": 150
+        }
+
+        try:
+            response = requests.post(api_url, headers=headers, json=payload, timeout=30)
+
+            if response.status_code == 200:
+                result = response.json()
+                return result['choices'][0]['message']['content'].strip()
+            else:
+                return f"[API_ERROR: {response.status_code} - {response.text[:100]}]"
+
+        except Exception as e:
+            return f"[API_ERROR: {str(e)[:100]}]"
+
+    def query_single(self, backstory: str, question: str) -> str:
+        """
+        Query OpenAI with a single request (e.g., for thematic analysis)
+
+        Args:
+            backstory: System prompt / context
+            question: Query text
+
+        Returns:
+            LLM response text
+        """
+        # For OpenAI, we can just use the regular query_llm method
+        # but with higher max_tokens for longer analysis
+        import requests
+
+        api_url = "https://api.openai.com/v1/chat/completions"
+
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json"
+        }
+
+        messages = [
+            {
+                "role": "system",
+                "content": backstory
+            },
+            {
+                "role": "user",
+                "content": question
+            }
+        ]
+
+        payload = {
+            "model": self.model_name,
+            "messages": messages,
+            "temperature": self.config.get('temperature', 0.3),
+            "max_tokens": 1000  # More tokens for thematic analysis
+        }
+
+        try:
+            response = requests.post(api_url, headers=headers, json=payload, timeout=60)
+
+            if response.status_code == 200:
+                result = response.json()
+                return result['choices'][0]['message']['content'].strip()
+            else:
+                raise Exception(f"API returned {response.status_code}: {response.text[:200]}")
+
+        except Exception as e:
+            raise Exception(f"OpenAI API error: {str(e)}")
+
+    def generate_responses(
+        self,
+        df: pd.DataFrame,
+        progress_callback: Optional[Callable[[int, int], None]] = None
+    ) -> pd.DataFrame:
+        """Generate responses using OpenAI API"""
+
+        if 'backstory' not in df.columns:
+            raise ValueError("DataFrame must have 'backstory' column")
+
+        results = df.copy()
+        results['response'] = ""
+
+        question = self.config['question']
+        total = len(df)
+
+        for i, (idx, row) in enumerate(df.iterrows()):
+            backstory = row['backstory']
+
+            if pd.isna(backstory) or str(backstory).strip() == "":
+                results.loc[idx, 'response'] = "[EMPTY]"
+                continue
+
+            try:
+                response = self.query_llm(str(backstory), question)
+                results.loc[idx, 'response'] = response
+
+            except Exception as e:
+                results.loc[idx, 'response'] = f"[ERROR: {str(e)[:50]}]"
+
+            if progress_callback:
+                progress_callback(i + 1, total)
+
+            # Small delay to avoid rate limiting
+            time.sleep(0.2)
+
+        return results
+
+
+class AnthropicSampler:
+    """
+    Anthropic API sampler (Claude)
+
+    Uses Anthropic's API to access Claude models.
+    Requires ANTHROPIC_API_KEY environment variable or passed in config.
+    """
+
+    def __init__(self, config: dict):
+        self.config = config
+        self.api_key = config.get('anthropic_api_key') or os.getenv('ANTHROPIC_API_KEY')
+        # Use Claude 3.5 Sonnet by default (best balance of quality and cost)
+        self.model_name = config.get('anthropic_model', 'claude-3-5-sonnet-20241022')
+
+        if not self.api_key:
+            raise ValueError(
+                "Anthropic API key required. Set ANTHROPIC_API_KEY environment variable "
+                "or pass 'anthropic_api_key' in config. Get key from: https://console.anthropic.com/settings/keys"
+            )
+
+    def query_llm(self, backstory: str, question: str) -> str:
+        """Query Anthropic API"""
+        import requests
+
+        api_url = "https://api.anthropic.com/v1/messages"
+
+        headers = {
+            "x-api-key": self.api_key,
+            "anthropic-version": "2023-06-01",
+            "Content-Type": "application/json"
+        }
+
+        payload = {
+            "model": self.model_name,
+            "max_tokens": 150,
+            "temperature": self.config['temperature'],
+            "system": (
+                "Adopt the following persona and answer only based on it. "
+                "Do not invent details beyond the provided attributes.\n\n"
+                f"{backstory}"
+            ),
+            "messages": [
+                {
+                    "role": "user",
+                    "content": question
+                }
+            ]
+        }
+
+        try:
+            response = requests.post(api_url, headers=headers, json=payload, timeout=30)
+
+            if response.status_code == 200:
+                result = response.json()
+                return result['content'][0]['text'].strip()
+            else:
+                return f"[API_ERROR: {response.status_code} - {response.text[:100]}]"
+
+        except Exception as e:
+            return f"[API_ERROR: {str(e)[:100]}]"
+
+    def query_single(self, backstory: str, question: str) -> str:
+        """
+        Query Anthropic with a single request (e.g., for thematic analysis)
+
+        Args:
+            backstory: System prompt / context
+            question: Query text
+
+        Returns:
+            LLM response text
+        """
+        import requests
+
+        api_url = "https://api.anthropic.com/v1/messages"
+
+        headers = {
+            "x-api-key": self.api_key,
+            "anthropic-version": "2023-06-01",
+            "Content-Type": "application/json"
+        }
+
+        payload = {
+            "model": self.model_name,
+            "max_tokens": 1000,  # More tokens for thematic analysis
+            "temperature": self.config.get('temperature', 0.3),
+            "system": backstory,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": question
+                }
+            ]
+        }
+
+        try:
+            response = requests.post(api_url, headers=headers, json=payload, timeout=60)
+
+            if response.status_code == 200:
+                result = response.json()
+                return result['content'][0]['text'].strip()
+            else:
+                raise Exception(f"API returned {response.status_code}: {response.text[:200]}")
+
+        except Exception as e:
+            raise Exception(f"Anthropic API error: {str(e)}")
+
+    def generate_responses(
+        self,
+        df: pd.DataFrame,
+        progress_callback: Optional[Callable[[int, int], None]] = None
+    ) -> pd.DataFrame:
+        """Generate responses using Anthropic API"""
+
+        if 'backstory' not in df.columns:
+            raise ValueError("DataFrame must have 'backstory' column")
+
+        results = df.copy()
+        results['response'] = ""
+
+        question = self.config['question']
+        total = len(df)
+
+        for i, (idx, row) in enumerate(df.iterrows()):
+            backstory = row['backstory']
+
+            if pd.isna(backstory) or str(backstory).strip() == "":
+                results.loc[idx, 'response'] = "[EMPTY]"
+                continue
+
+            try:
+                response = self.query_llm(str(backstory), question)
+                results.loc[idx, 'response'] = response
+
+            except Exception as e:
+                results.loc[idx, 'response'] = f"[ERROR: {str(e)[:50]}]"
+
+            if progress_callback:
+                progress_callback(i + 1, total)
+
+            # Small delay to avoid rate limiting
+            time.sleep(0.2)
+
+        return results
+
+
+class WinstonSampler:
+    """
+    Winston GPU server sampler using SSH commands
+
+    Requires:
+    - SSH key authentication to Winston (no password prompts)
+    - Winston files already set up (see WINSTON_README.md)
+    """
+
+    def __init__(self, config: dict):
+        self.config = config
+        self.winston_host = "sturgis@158.143.14.43"
+        self.winston_dir = "/home/sturgis/silicon_samples"
+
+    def query_single(self, backstory: str, question: str) -> str:
+        """
+        Query Winston with a single request (e.g., for thematic analysis)
+
+        Args:
+            backstory: System prompt / context
+            question: Query text
+
+        Returns:
+            LLM response text
+        """
+        import subprocess
+        import tempfile
+        from pathlib import Path
+
+        # Create single-row dataframe
+        df = pd.DataFrame({"backstory": [backstory]})
+
+        # Create temp files
+        temp_dir = Path(tempfile.mkdtemp())
+        local_input = temp_dir / "query_input.csv"
+        local_output = temp_dir / "query_output.csv"
+
+        df.to_csv(local_input, index=False)
+
+        remote_input = f"{self.winston_dir}/temp_query_input.csv"
+        remote_output = f"{self.winston_dir}/temp_query_output.csv"
+
+        try:
+            # Upload
+            subprocess.run(
+                ["scp", str(local_input), f"{self.winston_host}:{remote_input}"],
+                check=True,
+                capture_output=True
+            )
+
+            # Update config with question
+            # Use JSON to safely pass the question text
+            import json as json_lib
+            temp_val = self.config.get('temperature', 0.3)
+
+            # Create Python script that uses json.dumps to handle escaping
+            config_update_script = f"""
+import json
+with open('{self.winston_dir}/config_winston_silicon.json') as f:
+    config = json.load(f)
+config['question'] = {json_lib.dumps(question)}
+config['processing']['temperature'] = {temp_val}
+config['processing']['max_tokens'] = 500
+with open('{self.winston_dir}/config_winston_silicon.json', 'w') as f:
+    json.dump(config, f, indent=2)
+"""
+
+            # Write script to temp file, upload, execute, then delete
+            local_script = temp_dir / "update_config.py"
+            with open(local_script, 'w') as f:
+                f.write(config_update_script)
+
+            remote_script = f"{self.winston_dir}/temp_update_config.py"
+
+            subprocess.run(
+                ["scp", str(local_script), f"{self.winston_host}:{remote_script}"],
+                check=True,
+                capture_output=True
+            )
+
+            subprocess.run(
+                ["ssh", self.winston_host, f"python3 {remote_script}"],
+                check=True,
+                capture_output=True
+            )
+
+            subprocess.run(
+                ["ssh", self.winston_host, f"rm {remote_script}"],
+                capture_output=True
+            )
+
+            # Run processing
+            cmd = (
+                f"cd {self.winston_dir} && "
+                f"bash -c 'source ~/miniconda3/bin/activate soc_env && "
+                f"python3 process_silicon_winston_simple.py {remote_input} {remote_output}'"
+            )
+
+            result = subprocess.run(
+                ["ssh", self.winston_host, cmd],
+                capture_output=True,
+                text=True,
+                timeout=120
+            )
+
+            if result.returncode != 0:
+                raise Exception(f"Winston query failed: {result.stderr}")
+
+            # Download result
+            subprocess.run(
+                ["scp", f"{self.winston_host}:{remote_output}", str(local_output)],
+                check=True,
+                capture_output=True
+            )
+
+            # Read response
+            results_df = pd.read_csv(local_output)
+
+            if 'LLM_response' in results_df.columns:
+                response = results_df['LLM_response'].iloc[0]
+            elif 'response' in results_df.columns:
+                response = results_df['response'].iloc[0]
+            else:
+                response = "[No response column found]"
+
+            # Cleanup remote
+            subprocess.run(
+                ["ssh", self.winston_host, f"rm -f {remote_input} {remote_output}"],
+                capture_output=True
+            )
+
+            return response
+
+        except Exception as e:
+            raise Exception(f"Winston query error: {str(e)}")
+        finally:
+            # Cleanup local files
+            local_input.unlink(missing_ok=True)
+            local_output.unlink(missing_ok=True)
+            if 'local_script' in locals():
+                local_script.unlink(missing_ok=True)
+            # Remove temp directory (will only work if empty)
+            try:
+                temp_dir.rmdir()
+            except:
+                # If directory not empty, use shutil
+                import shutil
+                shutil.rmtree(temp_dir, ignore_errors=True)
+
+    def generate_responses(
+        self,
+        df: pd.DataFrame,
+        progress_callback: Optional[Callable[[int, int], None]] = None
+    ) -> pd.DataFrame:
+        """
+        Generate responses using Winston GPU server
+
+        This is a synchronous operation that:
+        1. Uploads sample data to Winston
+        2. Runs processing script directly (not via Slurm)
+        3. Downloads results
+
+        Args:
+            df: DataFrame with 'backstory' column
+            progress_callback: Optional function(current, total) for progress updates
+
+        Returns:
+            DataFrame with original columns plus 'response' column
+        """
+        import subprocess
+        import tempfile
+        from pathlib import Path
+
+        if 'backstory' not in df.columns:
+            raise ValueError("DataFrame must have 'backstory' column")
+
+        # Create temp files
+        temp_dir = Path(tempfile.mkdtemp())
+        local_input = temp_dir / "input.csv"
+        local_output = temp_dir / "output.csv"
+
+        # Save input data
+        df.to_csv(local_input, index=False)
+
+        # Remote paths
+        remote_input = f"{self.winston_dir}/temp_dashboard_input.csv"
+        remote_output = f"{self.winston_dir}/temp_dashboard_output.csv"
+
+        try:
+            # Step 1: Upload input file
+            print("📤 Uploading data to Winston...")
+            subprocess.run(
+                ["scp", str(local_input), f"{self.winston_host}:{remote_input}"],
+                check=True,
+                capture_output=True
+            )
+
+            # Step 2: Create question config on Winston
+            question_text = self.config['question']
+            temp_val = self.config['temperature']
+
+            # Update config remotely with our question
+            config_update = f"""
+import json
+with open('{self.winston_dir}/config_winston_silicon.json') as f:
+    config = json.load(f)
+config['question'] = '''{question_text}'''
+config['processing']['temperature'] = {temp_val}
+config['processing']['max_tokens'] = 100
+with open('{self.winston_dir}/config_winston_silicon.json', 'w') as f:
+    json.dump(config, f, indent=2)
+"""
+
+            subprocess.run(
+                ["ssh", self.winston_host, f"python3 -c \"{config_update}\""],
+                check=True,
+                capture_output=True
+            )
+
+            # Step 3: Run processing on Winston
+            print("🚀 Processing on Winston with Qwen2.5...")
+            print("   This may take several minutes...")
+
+            cmd = (
+                f"cd {self.winston_dir} && "
+                f"bash -c 'source ~/miniconda3/bin/activate soc_env && "
+                f"python3 process_silicon_winston_simple.py {remote_input} {remote_output}'"
+            )
+
+            result = subprocess.run(
+                ["ssh", self.winston_host, cmd],
+                capture_output=True,
+                text=True
+            )
+
+            if result.returncode != 0:
+                raise Exception(f"Winston processing failed: {result.stderr}")
+
+            # Show progress (we can't get real-time updates, so just show completion)
+            if progress_callback:
+                progress_callback(len(df), len(df))
+
+            # Step 4: Download results
+            print("📥 Downloading results...")
+            subprocess.run(
+                ["scp", f"{self.winston_host}:{remote_output}", str(local_output)],
+                check=True,
+                capture_output=True
+            )
+
+            # Step 5: Load and process results
+            results_df = pd.read_csv(local_output)
+
+            # Rename LLM_response column to response for consistency with dashboard
+            if 'LLM_response' in results_df.columns:
+                results_df['response'] = results_df['LLM_response']
+                results_df = results_df.drop(columns=['LLM_response'])
+
+            # Clean up remote files
+            subprocess.run(
+                ["ssh", self.winston_host, f"rm -f {remote_input} {remote_output}"],
+                capture_output=True
+            )
+
+            return results_df
+
+        except subprocess.CalledProcessError as e:
+            raise Exception(f"SSH/SCP command failed: {e.stderr if hasattr(e, 'stderr') else str(e)}")
+        finally:
+            # Clean up local temp files
+            local_input.unlink(missing_ok=True)
+            local_output.unlink(missing_ok=True)
+            temp_dir.rmdir()

requirements.txt

@@ -0,0 +1,3 @@
+streamlit>=1.28.0
+pandas>=2.0.0
+requests>=2.31.0