Upload 4 files

CHANGELOG.md

@@ -10,8 +10,9 @@ All notable changes to ConversAI will be documented in this file.
   - **No API endpoint issues** - everything runs on your Space
   - **Faster after first load** - models cached in memory
   - **100% private** - all processing happens locally
-  - Default model: **google/flan-t5-large** (1.2GB, good quality)
+  - Default model: **google/flan-t5-xl** (3GB, excellent quality)
   - Supports all Flan-T5 variants (base, large, xl, xxl)
+  - **Important**: XL or larger required for quality results; smaller models produce poor output
 
 ### Added
 - **New dependencies**: transformers, torch, accelerate, sentencepiece
@@ -43,11 +44,12 @@ All notable changes to ConversAI will be documented in this file.
   - Added model caching to keep models in memory
   - Auto-detects CUDA/CPU and optimizes accordingly
 
-- **Default model**: `google/flan-t5-large` (line 84)
+- **Default model**: `google/flan-t5-xl` (line 84)
   - Changed from API-based to local transformers
-  - 1.2GB model provides good balance of quality and speed
-  - Better at JSON generation than smaller models
-  - User can upgrade to xl/xxl or downgrade to base via LLM_MODEL env var
+  - 3GB model required for acceptable quality
+  - Testing showed base/large models produce generic, irrelevant questions
+  - XL provides good balance of quality and resource usage
+  - User can upgrade to xxl or downgrade to large/base via LLM_MODEL env var (not recommended)
 
 - **Complete rewrite of survey generation** in `survey_generator.py`:
   - **Changed approach**: No longer asks model to generate JSON (T5 models struggle with structured output)

README.md

@@ -1,170 +1,170 @@
----
-title: ProjectEcho - Qualitative Research Assistant
-emoji: 🔬
-colorFrom: blue
-colorTo: purple
-sdk: gradio
-sdk_version: 5.49.1
-app_file: app.py
-pinned: false
-license: mit
----
-
-# ConversAI - AI-Powered Qualitative Research Assistant
-
-Battle the blank page, reach global audiences, and uncover insights with AI assistance.
-
----
-
-> **✨ UPDATED (Nov 2025):** Now uses **local transformers** with **Google Flan-T5** models - Fast, reliable, and **completely FREE**! No API dependencies, runs directly on HuggingFace Spaces.
-
----
-
-## 🌟 Features
-
-### 📝 Survey Generation
-- Generate professional surveys from simple outlines
-- Follow industry best practices automatically
-- Choose from qualitative, quantitative, or mixed methods
-- Customize number of questions and target audience
-
-### 🌍 Survey Translation
-- Translate surveys to 18+ languages
-- Maintain cultural appropriateness and meaning
-- Reach global audiences effortlessly
-- Batch translation support
-
-### 📊 Data Analysis
-- AI-assisted thematic analysis
-- Sentiment analysis and emotional insights
-- Automatic pattern and trend detection
-- Generate actionable insights and recommendations
-- Export detailed analysis reports
-
-## 🚀 Quick Start
-
-**On HuggingFace Spaces:** Works immediately with zero configuration! Uses the free HF Inference API.
-
-**Workflow:**
-1. **Generate a Survey**: Start with an outline or topic description
-2. **Translate**: Select target languages to reach global audiences
-3. **Collect Responses**: Use the generated survey with your participants
-4. **Analyze**: Upload responses to uncover key findings and trends
-
-## 🔧 Configuration
-
-### Default: Local Transformers (Completely FREE!)
-
-**✨ Zero configuration needed!** ConversAI works out-of-the-box on HuggingFace Spaces using local model loading.
-
-**Default Model:** google/flan-t5-large
-- ✅ **100% Free** - No API keys, no costs, ever
-- ✅ **Good quality** - 1.2GB model, excellent at following instructions
-- ✅ **Fast after loading** - Typically 3-8 seconds per request after initial load
-- ✅ **No API dependencies** - Runs entirely on your Space's compute
-- ✅ **Private** - All processing happens locally, nothing sent to external APIs
-- ✅ **Reliable** - Google's instruction-tuned model, battle-tested
-
-**Setup for HuggingFace Spaces:**
-- Just deploy - models download automatically on first run
-- **No API keys or tokens required!**
-- Models are cached after first download for faster subsequent loads
-
-### Alternative Free Models
-
-You can try different free models by setting the `LLM_MODEL` environment variable:
-
-**Recommended Free Models (Local Transformers):**
-
-| Model | Best For | Speed | Quality | Model Size |
-|-------|----------|-------|---------|------------|
-| **google/flan-t5-base** | Testing - fastest | ⚡⚡⚡ Very Fast | ⭐⭐ Basic | 250MB |
-| **google/flan-t5-large** (default) | **Recommended** - balanced | ⚡⚡ Fast | ⭐⭐⭐ Good | 1.2GB |
-| **google/flan-t5-xl** | Better quality | ⚡ Medium | ⭐⭐⭐⭐ Excellent | 3GB |
-| **google/flan-t5-xxl** | Maximum quality | ⚡ Slower | ⭐⭐⭐⭐⭐ Best | 11GB |
-
-**Note:** Flan-T5 models are Google's instruction-tuned models, specifically designed for following instructions. They run locally with transformers library.
-
-**To change model:**
-```bash
-# In Space Settings → Variables
-LLM_MODEL=google/flan-t5-large  # Better quality
-
-# Or for maximum quality (requires more memory)
-LLM_MODEL=google/flan-t5-xl
-```
-
-**Why Local Transformers?**
-- ✅ **No API dependencies** - runs entirely on your Space
-- ✅ **No 404 errors** - no network issues
-- ✅ **Fast after loading** - models cached in memory
-- ✅ **Instruction-tuned** - designed for following prompts
-- ✅ **Privacy** - all processing happens locally
-
-### Tips for Best Performance with Local Models
-
-1. **Default model (flan-t5-large) is recommended** - Good balance of quality and speed
-2. **First load takes time** - Model downloads and loads (~2-3 minutes for large)
-3. **Subsequent requests are fast** - Model stays in memory (3-8 seconds)
-4. **For simple testing** - Use flan-t5-base (faster loading)
-5. **For best quality** - Use flan-t5-xl or xxl (requires more memory)
-6. **Keep prompts clear** - Simpler outlines work better with smaller models
-
-## 📦 Installation
-
-```bash
-# Install dependencies
-pip install -r requirements.txt
-
-# Check environment setup (optional but recommended)
-python check_env.py
-
-# Run the app
-python app.py
-```
-
-## 🏗️ Architecture
-
-ConversAI is built with a modular architecture:
-
-- **llm_backend.py** - Unified LLM interface supporting multiple providers
-- **survey_generator.py** - AI-powered survey generation
-- **survey_translator.py** - Multi-language translation engine
-- **data_analyzer.py** - Qualitative data analysis and insights
-- **app.py** - Gradio-based web interface
-- **export_utils.py** - Export to JSON, CSV, Markdown
-
-## 📄 Data Privacy
-
-- All processing is done through your configured LLM provider
-- No data is stored permanently by this application
-- Survey data and responses remain in your control
-- Suitable for sensitive research projects
-
-## 🤝 Contributing
-
-Contributions are welcome! This is a production-grade application designed for real-world qualitative research.
-
-## 📝 License
-
-MIT License - Feel free to use for research and commercial purposes.
-
----
-
-## 📚 Documentation
-
-**New to ConversAI?** Start with **[USER_GUIDE.md](USER_GUIDE.md)** for a complete walkthrough.
-
-**Quick Links:**
-- 📖 [Complete User Guide](USER_GUIDE.md) - How to use ConversAI (START HERE)
-- ⚡ [Quick Start for HF Spaces](QUICK_START_HF_SPACES.md) - 5-minute deployment
-- 🔧 [Troubleshooting](TROUBLESHOOTING.md) - Common issues and solutions
-- 🆓 [Free Models Guide](FREE_MODELS.md) - Best free models to use
-
-**Diagnostic Tools:**
-- Run `python check_env.py` - Check your environment setup
-- Run `python test_hf_backend.py` - Test HuggingFace connection
-
----
-
-Built with ❤️ using Gradio and state-of-the-art open-source LLMs
+---
+title: ConversAI - Qualitative Research Assistant
+emoji: 🔬
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 5.45.0
+app_file: app.py
+pinned: false
+license: mit
+---
+
+# ConversAI - AI-Powered Qualitative Research Assistant
+
+Battle the blank page, reach global audiences, and uncover insights with AI assistance.
+
+---
+
+> **✨ UPDATED (Nov 2025):** Now uses **local transformers** with **Google Flan-T5** models - Fast, reliable, and **completely FREE**! No API dependencies, runs directly on HuggingFace Spaces.
+
+---
+
+## 🌟 Features
+
+### 📝 Survey Generation
+- Generate professional surveys from simple outlines
+- Follow industry best practices automatically
+- Choose from qualitative, quantitative, or mixed methods
+- Customize number of questions and target audience
+
+### 🌍 Survey Translation
+- Translate surveys to 18+ languages
+- Maintain cultural appropriateness and meaning
+- Reach global audiences effortlessly
+- Batch translation support
+
+### 📊 Data Analysis
+- AI-assisted thematic analysis
+- Sentiment analysis and emotional insights
+- Automatic pattern and trend detection
+- Generate actionable insights and recommendations
+- Export detailed analysis reports
+
+## 🚀 Quick Start
+
+**On HuggingFace Spaces:** Works immediately with zero configuration! Uses the free HF Inference API.
+
+**Workflow:**
+1. **Generate a Survey**: Start with an outline or topic description
+2. **Translate**: Select target languages to reach global audiences
+3. **Collect Responses**: Use the generated survey with your participants
+4. **Analyze**: Upload responses to uncover key findings and trends
+
+## 🔧 Configuration
+
+### Default: Local Transformers (Completely FREE!)
+
+**✨ Zero configuration needed!** ConversAI works out-of-the-box on HuggingFace Spaces using local model loading.
+
+**Default Model:** google/flan-t5-xl
+- ✅ **100% Free** - No API keys, no costs, ever
+- ✅ **High quality** - 3GB model, excellent at following complex instructions
+- ✅ **Good speed** - Typically 5-10 seconds per request after initial load
+- ✅ **No API dependencies** - Runs entirely on your Space's compute
+- ✅ **Private** - All processing happens locally, nothing sent to external APIs
+- ✅ **Reliable** - Google's instruction-tuned model, battle-tested
+
+**Setup for HuggingFace Spaces:**
+- Just deploy - models download automatically on first run
+- **No API keys or tokens required!**
+- Models are cached after first download for faster subsequent loads
+
+### Alternative Free Models
+
+You can try different free models by setting the `LLM_MODEL` environment variable:
+
+**Recommended Free Models (Local Transformers):**
+
+| Model | Best For | Speed | Quality | Model Size |
+|-------|----------|-------|---------|------------|
+| **google/flan-t5-base** | Quick testing only | ⚡⚡⚡ Very Fast | ⭐ Poor | 250MB |
+| **google/flan-t5-large** | Faster loading | ⚡⚡ Fast | ⭐⭐ Fair | 1.2GB |
+| **google/flan-t5-xl** (default) | **Recommended** - best balance | ⚡ Good | ⭐⭐⭐⭐ Excellent | 3GB |
+| **google/flan-t5-xxl** | Maximum quality | ⚡ Slower | ⭐⭐⭐⭐⭐ Best | 11GB |
+
+**Note:** Flan-T5 models are Google's instruction-tuned models, specifically designed for following instructions. They run locally with transformers library.
+
+**To change model:**
+```bash
+# In Space Settings → Variables
+LLM_MODEL=google/flan-t5-large  # Better quality
+
+# Or for maximum quality (requires more memory)
+LLM_MODEL=google/flan-t5-xl
+```
+
+**Why Local Transformers?**
+- ✅ **No API dependencies** - runs entirely on your Space
+- ✅ **No 404 errors** - no network issues
+- ✅ **Fast after loading** - models cached in memory
+- ✅ **Instruction-tuned** - designed for following prompts
+- ✅ **Privacy** - all processing happens locally
+
+### Tips for Best Performance with Local Models
+
+1. **Use flan-t5-xl (default)** - XL provides good quality, smaller models produce poor results
+2. **First load takes time** - Model downloads and loads (~3-5 minutes for XL)
+3. **Subsequent requests are fast** - Model stays in memory (5-10 seconds)
+4. **For maximum quality** - Use flan-t5-xxl (requires 16GB+ RAM)
+5. **Avoid smaller models** - Base and Large often produce generic or irrelevant questions
+6. **Be specific in outlines** - More detail helps model generate better questions
+
+## 📦 Installation
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Check environment setup (optional but recommended)
+python check_env.py
+
+# Run the app
+python app.py
+```
+
+## 🏗️ Architecture
+
+ConversAI is built with a modular architecture:
+
+- **llm_backend.py** - Unified LLM interface supporting multiple providers
+- **survey_generator.py** - AI-powered survey generation
+- **survey_translator.py** - Multi-language translation engine
+- **data_analyzer.py** - Qualitative data analysis and insights
+- **app.py** - Gradio-based web interface
+- **export_utils.py** - Export to JSON, CSV, Markdown
+
+## 📄 Data Privacy
+
+- All processing is done through your configured LLM provider
+- No data is stored permanently by this application
+- Survey data and responses remain in your control
+- Suitable for sensitive research projects
+
+## 🤝 Contributing
+
+Contributions are welcome! This is a production-grade application designed for real-world qualitative research.
+
+## 📝 License
+
+MIT License - Feel free to use for research and commercial purposes.
+
+---
+
+## 📚 Documentation
+
+**New to ConversAI?** Start with **[USER_GUIDE.md](USER_GUIDE.md)** for a complete walkthrough.
+
+**Quick Links:**
+- 📖 [Complete User Guide](USER_GUIDE.md) - How to use ConversAI (START HERE)
+- ⚡ [Quick Start for HF Spaces](QUICK_START_HF_SPACES.md) - 5-minute deployment
+- 🔧 [Troubleshooting](TROUBLESHOOTING.md) - Common issues and solutions
+- 🆓 [Free Models Guide](FREE_MODELS.md) - Best free models to use
+
+**Diagnostic Tools:**
+- Run `python check_env.py` - Check your environment setup
+- Run `python test_hf_backend.py` - Test HuggingFace connection
+
+---
+
+Built with ❤️ using Gradio and state-of-the-art open-source LLMs

llm_backend.py

@@ -78,10 +78,10 @@ class LLMBackend:
         defaults = {
             LLMProvider.OPENAI: "gpt-4o-mini",
             LLMProvider.ANTHROPIC: "claude-3-5-sonnet-20241022",
-            # Using Flan-T5-Large - good balance of size (1.2GB) and quality
-            # For smaller/faster: google/flan-t5-base (250MB)
-            # For better quality: google/flan-t5-xl (3GB) or google/flan-t5-xxl (11GB)
-            LLMProvider.HUGGINGFACE: "google/flan-t5-large",
+            # Using Flan-T5-XL - best balance for quality survey generation (3GB)
+            # For faster loading: google/flan-t5-large (1.2GB) - may have lower quality
+            # For maximum quality: google/flan-t5-xxl (11GB) - requires more memory
+            LLMProvider.HUGGINGFACE: "google/flan-t5-xl",
             LLMProvider.LM_STUDIO: "google/gemma-3-27b"
         }
         return os.getenv("LLM_MODEL", defaults[self.provider])

survey_generator.py

@@ -83,20 +83,21 @@ class SurveyGenerator:
 
     def _build_generation_prompt(self, outline, survey_type, num_questions, target_audience) -> str:
         """Build the user prompt for survey generation"""
-        # For T5 models, ask for simple numbered list instead of JSON
-        return f"""Generate {num_questions} survey questions about: {outline}
+        # For T5 models, be very specific and direct
+        return f"""Create {num_questions} professional survey questions.
 
-Target audience: {target_audience}
-Survey type: {survey_type}
+Topic: {outline}
+Audience: {target_audience}
 
-Create {num_questions} clear, professional questions. Write each question on a new line starting with a number.
+Write {num_questions} questions numbered 1-{num_questions}. Each question must be specific to the topic above.
 
-Example format:
-1. What is your overall experience with [topic]?
-2. How would you rate [specific aspect]?
-3. What improvements would you suggest?
+Examples:
+1. What is your experience with X?
+2. How would you rate Y?
+3. What challenges do you face with Z?
 
-Now generate {num_questions} questions:"""
+Your {num_questions} questions:
+1."""
 
     def _parse_survey_response(self, response: str) -> Dict:
         """Parse LLM response into survey structure"""
@@ -105,20 +106,36 @@ Now generate {num_questions} questions:"""
 
     def _parse_numbered_list(self, response: str) -> Dict:
         """Parse numbered list of questions into survey structure"""
-        lines = [line.strip() for line in response.split('\n') if line.strip()]
+        # First, try to split by numbered patterns (1., 2., etc.)
+        import re
+
+        # Pattern to match numbered questions: "1. Question" or "1) Question"
+        pattern = r'\d+[\.\)]\s+'
+
+        # Split by the pattern but keep what comes after each number
+        parts = re.split(pattern, response)
+
+        # Remove empty first element if exists
+        parts = [p.strip() for p in parts if p.strip()]
 
         questions = []
         question_id = 1
 
-        for line in lines:
-            # Skip empty lines or lines that are too short
-            if len(line) < 5:
+        for part in parts:
+            # Skip if too short
+            if len(part) < 10:
                 continue
 
-            # Remove leading numbers, bullets, dashes, etc.
-            clean_line = line.lstrip('0123456789.-) \t')
+            # Take only the first sentence/question if there are multiple
+            # Split by question mark or period
+            sentences = re.split(r'[?.!]\s+(?=\d+[\.\)]|\Z)', part)
+            clean_line = sentences[0].strip()
+
+            # Add question mark if missing
+            if not clean_line.endswith('?'):
+                clean_line += '?'
 
-            # Skip lines that don't look like questions
+            # Skip if still too short
             if len(clean_line) < 10:
                 continue