Fixed Intelligent Course Generator - Real Content Analysis

README.md

@@ -1,34 +1,41 @@
----
-title: Leadership Instructor Agent
-emoji: 🎯
-colorFrom: blue
-colorTo: purple
-sdk: gradio
-sdk_version: 4.44.0
-app_file: app.py
-pinned: false
-license: mit
----
+# 🧠 Intelligent Course Generator - FIXED
 
-# 🎯 Leadership Instructor Agent
+## ✅ What This Fixes
 
-Your personal AI leadership instructor powered by the Socratic method and your curated corpus of 44,854 leadership insights.
+1. **Real Topic Extraction** - Finds actual leadership topics from your corpus instead of generic "MicroLessons"
+2. **Content Analysis** - Shows exactly what can be built and what needs AI enhancement
+3. **Sufficiency Checking** - Determines if you have enough content for lesson/unit/course
+4. **Gap Identification** - Shows what percentage needs ChatGPT vs corpus content
+5. **Actual Content Generation** - Creates real lessons on real topics
 
-## Features
+## 🔍 How It Works
 
-- **Socratic Method Teaching**: Guides discovery through questions rather than lectures
-- **Personal Corpus**: Draws primarily from your curated leadership materials
-- **Smart Fallback**: Uses general leadership knowledge when corpus doesn't cover a topic
-- **Source Citations**: References specific materials from your corpus
-- **Copy-Paste Friendly**: Easy to save and reference responses
+### Step 1: Content Analysis
+- Enter broad search term (e.g., "communication", "leadership")
+- System extracts actual topics from your corpus
+- Shows buildability analysis for each topic
 
-## Usage
+### Step 2: Real Content Generation  
+- Use specific topics identified in analysis
+- Generates complete lessons with real content
+- Shows corpus % vs AI enhancement %
 
-Simply ask any leadership question and I'll help you explore the answer through thoughtful questioning and guidance.
+## 🚀 Features
 
-**Example Questions:**
-- "How should I handle a difficult team member?"
-- "What's the difference between management and leadership?"
-- "How do I build trust with my remote team?"
+- **Topic Extraction**: Finds real leadership topics in your corpus
+- **Sufficiency Analysis**: 50/50, 60/40, or 70/30 content standards
+- **Gap Filling**: Uses ChatGPT only for identified gaps
+- **Real Content**: Actual lessons, not placeholder text
+- **Quality Standards**: Meets your corpus utilization requirements
 
-Built with ❤️ using Gradio and Hugging Face
+## 📊 Content Standards
+
+- **70/30**: Ideal (70% corpus, 30% AI) 
+- **60/40**: Good (60% corpus, 40% AI)
+- **50/50**: Minimum (50% corpus, 50% AI)
+
+## 🎯 Usage
+
+1. **Analyze Content**: See what topics you can build
+2. **Generate Lessons**: Create real content on those topics
+3. **Quality Check**: Verify corpus vs AI percentages

app.py

@@ -18,159 +18,110 @@ model = None
 def safe_load_resources():
     """Safely load resources with error handling"""
     global dataset, model
-
+    
     try:
         if dataset is None:
-            print("🔄 Loading resources...")
+            print("Loading resources...")
             from datasets import load_dataset
             from sentence_transformers import SentenceTransformer
-
+            
             dataset = load_dataset("PhillyMac/Leadership_Corpus")['train']
             model = SentenceTransformer('all-MiniLM-L6-v2')
-            print(f"✅ Loaded {len(dataset)} insights")
-
+            print(f"✅ Loaded {len(dataset)} leadership insights")
+        
         return dataset, model, None
-
+        
     except Exception as e:
         error_msg = f"Resource loading error: {str(e)}"
         print(error_msg)
         return None, None, error_msg
 
-def get_chatgpt_with_references(topic, context_type="lesson", content_gap=""):
-    """Get ChatGPT content with forced academic references"""
-
+def extract_real_topics_from_corpus(search_term, max_topics=20):
+    """Extract actual leadership topics from corpus content"""
+    
     try:
-        import openai
-        api_key = os.getenv("OPENAI_API_KEY")
-
-        if not api_key:
-            return {
-                'content': f"ChatGPT enhancement not available for {topic}. Using foundational leadership principles.",
-                'references': ["ChatGPT unavailable - content based on established leadership theory"],
-                'error': "No API key"
-            }
-
-        openai.api_key = api_key
-
-        if context_type == "lesson":
-            prompt = f"""Create detailed content for {topic} leadership development lesson.
-
-CONTENT REQUIREMENTS:
-- Specific strategies and techniques for {topic}
-- Real-world application examples
-- Common challenges and solutions
-- Best practices from successful leaders
-
-GAP TO FILL: {content_gap}
-
-CRITICAL: You must provide specific academic references for all claims. Include:
-- Author names and years
-- Book titles or journal articles
-- Specific page numbers or sections where possible
-- Research studies with findings
-- Leadership expert quotes with attribution
-
-Format references in APA style. Minimum 5 credible sources required."""
-
-        elif context_type == "unit":
-            prompt = f"""Design comprehensive {topic} leadership unit content.
-
-CONTENT REQUIREMENTS:
-- Progressive skill development sequence
-- Integration strategies between lessons
-- Advanced application scenarios
-- Assessment frameworks
-
-GAP TO FILL: {content_gap}
-
-CRITICAL: Provide specific academic references for all methodologies and frameworks. Include:
-- Leadership development research citations
-- Organizational psychology studies
-- Management theory sources
-- Empirical studies on {topic} effectiveness
-
-Format in APA style. Minimum 8 credible sources required."""
-
-        else:  # course
-            prompt = f"""Create comprehensive {topic} leadership course framework.
-
-CONTENT REQUIREMENTS:
-- Complete course architecture
-- Learning progression design
-- Capstone project framework
-- Measurement and evaluation
-
-GAP TO FILL: {content_gap}
-
-CRITICAL: Provide extensive academic references for course design. Include:
-- Adult learning theory citations
-- Leadership development research
-- Course design methodology sources
-- Assessment validity studies
-
-Format in APA style. Minimum 12 credible sources required."""
-
-        response = openai.ChatCompletion.create(
-            model="gpt-4",
-            messages=[
-                {"role": "system", "content": "You are an expert in leadership development with access to academic research. Always provide specific, verifiable citations for all claims."},
-                {"role": "user", "content": prompt}
-            ],
-            max_tokens=1500,
-            temperature=0.3
-        )
-
-        content = response.choices[0].message.content
-        references = extract_references_from_content(content)
-
-        return {
-            'content': content,
-            'references': references,
-            'error': None
-        }
-
+        dataset, model, error = safe_load_resources()
+        if error:
+            return []
+        
+        # Search for content related to the term
+        search_embedding = model.encode([search_term])
+        sample_size = min(2000, len(dataset))
+        indices = np.random.choice(len(dataset), sample_size, replace=False)
+        
+        content_texts = [dataset[int(idx)]['content'] for idx in indices]
+        content_embeddings = model.encode(content_texts)
+        
+        similarities = cosine_similarity(search_embedding, content_embeddings)[0]
+        
+        # Get highly relevant content
+        relevant_indices = indices[similarities > 0.3]
+        relevant_content = [dataset[int(idx)]['content'] for idx in relevant_indices]
+        
+        # Extract actual topics using NLP patterns
+        topics = set()
+        
+        # Common leadership topic patterns
+        topic_patterns = [
+            r'\b([A-Z][a-z]+ [Ll]eadership)\b',  # "Strategic Leadership", "Transformational Leadership"
+            r'\b([A-Z][a-z]+ [Mm]anagement)\b',  # "Change Management", "Performance Management"
+            r'\b([A-Z][a-z]+ [Ss]kills?)\b',     # "Communication Skills", "Negotiation Skills"
+            r'\b([A-Z][a-z]+ [Dd]evelopment)\b', # "Team Development", "Leadership Development"
+            r'\b([A-Z][a-z]+ [Ii]ntelligence)\b', # "Emotional Intelligence"
+            r'\b([A-Z][a-z]+ [Bb]uilding)\b',    # "Team Building", "Trust Building"
+            r'\b([A-Z][a-z]+ [Rr]esolution)\b',  # "Conflict Resolution"
+            r'\bLeading ([A-Z][a-z]+)\b',        # "Leading Change", "Leading Teams"
+            r'\bManaging ([A-Z][a-z]+)\b',       # "Managing Conflict", "Managing Performance"
+        ]
+        
+        for content in relevant_content:
+            for pattern in topic_patterns:
+                matches = re.findall(pattern, content)
+                for match in matches:
+                    if isinstance(match, tuple):
+                        topic = ' '.join(match)
+                    else:
+                        topic = match
+                    
+                    # Clean up and validate topic
+                    topic = topic.strip()
+                    if len(topic.split()) <= 4 and len(topic) > 5:  # Reasonable length
+                        topics.add(topic)
+        
+        # Also extract key concepts that appear frequently
+        concept_words = []
+        for content in relevant_content:
+            words = re.findall(r'\b[A-Z][a-z]{4,}\b', content)  # Capitalized words 5+ letters
+            concept_words.extend(words)
+        
+        # Find most common concepts
+        word_counts = Counter(concept_words)
+        common_concepts = [word for word, count in word_counts.most_common(50) if count >= 3]
+        
+        # Create topics from common concepts
+        leadership_contexts = [
+            "Leadership", "Management", "Development", "Skills", "Strategy", 
+            "Communication", "Team Building", "Decision Making", "Innovation"
+        ]
+        
+        for concept in common_concepts:
+            for context in leadership_contexts:
+                potential_topic = f"{concept} {context}"
+                if len(potential_topic) < 50:  # Reasonable length
+                    topics.add(potential_topic)
+        
+        # Convert to list and sort by relevance (you could enhance this)
+        topic_list = list(topics)[:max_topics]
+        
+        return topic_list
+        
     except Exception as e:
-        return {
-            'content': f"AI enhancement unavailable: {str(e)}",
-            'references': [f"Error accessing ChatGPT: {str(e)}"],
-            'error': str(e)
-        }
-
-def extract_references_from_content(content):
-    """Extract academic references from ChatGPT content"""
-
-    references = []
-
-    # Look for common reference patterns
-    patterns = [
-        r'\([A-Z][a-zA-Z]+,\s*\d{4}\)',  # (Author, Year)
-        r'[A-Z][a-zA-Z]+\s*\(\d{4}\)',   # Author (Year)
-        r'[A-Z][a-zA-Z]+\s*et\s*al\.\s*\(\d{4}\)',  # Author et al. (Year)
-    ]
-
-    for pattern in patterns:
-        matches = re.findall(pattern, content)
-        references.extend(matches)
-
-    # Look for book titles in quotes or italics
-    book_patterns = [
-        r'"([A-Z][^"]*)"',  # "Book Title"
-        r'_([A-Z][^_]*)_',  # _Book Title_
-    ]
-
-    for pattern in book_patterns:
-        matches = re.findall(pattern, content)
-        references.extend([f'"{match}"' for match in matches if len(match) > 10])
-
-    # If no references found, add placeholder
-    if not references:
-        references = ["References not properly formatted in AI response - manual verification required"]
-
-    return list(set(references))  # Remove duplicates
-
-def analyze_corpus_sufficiency(topic):
-    """Analyze if corpus has sufficient content for lesson/unit/course generation"""
+        print(f"Topic extraction error: {str(e)}")
+        return []
 
+def analyze_content_sufficiency(topic):
+    """Analyze if corpus has sufficient content for different generation levels"""
+    
     try:
         dataset, model, error = safe_load_resources()
         if error:
@@ -178,962 +129,494 @@ def analyze_corpus_sufficiency(topic):
                 'status': 'error',
                 'message': error,
                 'corpus_percentage': 0,
-                'max_feasible': 'none'
+                'max_feasible': 'none',
+                'buildable_content': []
             }
-
-        print(f"🔍 Analyzing corpus sufficiency for: {topic}")
-
-        # Broader search for sufficiency analysis
+        
+        print(f"🔍 Analyzing content sufficiency for: {topic}")
+        
+        # Search corpus for topic-related content
         topic_embedding = model.encode([topic])
         sample_size = min(1500, len(dataset))
         indices = np.random.choice(len(dataset), sample_size, replace=False)
-
+        
         content_texts = [dataset[int(idx)]['content'] for idx in indices]
         content_embeddings = model.encode(content_texts)
-
+        
         similarities = cosine_similarity(topic_embedding, content_embeddings)[0]
-
-        # Classify content by relevance thresholds
-        high_relevance = similarities > 0.4    # Strong matches
-        medium_relevance = similarities > 0.3  # Good matches
-        low_relevance = similarities > 0.2     # Acceptable matches
-
+        
+        # Classify content by relevance
+        high_relevance = similarities > 0.4    # Strong matches - can build lessons
+        medium_relevance = similarities > 0.3  # Good matches - can contribute
+        low_relevance = similarities > 0.2     # Acceptable - background info
+        
         high_count = np.sum(high_relevance)
         medium_count = np.sum(medium_relevance) - high_count
         low_count = np.sum(low_relevance) - medium_count - high_count
-
-        total_relevant = high_count + medium_count + low_count
-
-        # Calculate content depth
+        
+        # Get actual content for analysis
         relevant_indices = indices[similarities > 0.2]
-        relevant_scores = similarities[similarities > 0.2]
-
-        content_analysis = []
-        themes = {}
-        source_documents = {}
-        total_content_length = 0
-
+        relevant_content = []
+        
         for i, idx in enumerate(relevant_indices):
+            actual_idx = int(idx)
+            content = dataset[actual_idx]['content']
+            score = similarities[similarities > 0.2][i]
+            
             try:
-                actual_idx = int(idx)
-                content = dataset[actual_idx]['content']
                 metadata = json.loads(dataset[actual_idx]['metadata'])
                 source_topic = metadata.get('topic', 'Leadership')
-                doc_id = dataset[actual_idx]['id']
-
-                content_analysis.append({
-                    'content': content,
-                    'topic': source_topic,
-                    'score': float(relevant_scores[i]),
-                    'id': doc_id,
-                    'length': len(content)
+            except:
+                source_topic = 'Leadership'
+            
+            relevant_content.append({
+                'content': content,
+                'score': float(score),
+                'source_topic': source_topic,
+                'length': len(content)
+            })
+        
+        # Sort by relevance
+        relevant_content.sort(key=lambda x: x['score'], reverse=True)
+        
+        # Calculate corpus percentage
+        total_relevant = high_count + medium_count + low_count
+        content_depth = sum(item['length'] for item in relevant_content[:10])
+        
+        # Scoring factors
+        relevance_score = (high_count * 3 + medium_count * 2 + low_count) / 50  # Max 50 points
+        depth_score = min(content_depth / 10000, 30)  # Max 30 points for depth
+        coverage_score = min(total_relevant / 20, 20)  # Max 20 points for coverage
+        
+        corpus_percentage = min(100, relevance_score + depth_score + coverage_score)
+        
+        # Determine what can be built
+        buildable_content = []
+        
+        if corpus_percentage >= 70:  # Strong corpus coverage
+            if high_count >= 8:
+                buildable_content.append({
+                    'type': 'course',
+                    'title': f"Comprehensive {topic} Leadership Course",
+                    'corpus_support': f"{corpus_percentage:.1f}%",
+                    'ai_needed': f"{100-corpus_percentage:.1f}%"
                 })
-
-                total_content_length += len(content)
-
-                # Track source documents
-                if source_topic not in source_documents:
-                    source_documents[source_topic] = []
-                source_documents[source_topic].append({
-                    'id': doc_id,
-                    'score': float(relevant_scores[i]),
-                    'content_length': len(content)
+            
+            if high_count >= 4:
+                buildable_content.append({
+                    'type': 'unit', 
+                    'title': f"Strategic {topic} Leadership Unit",
+                    'corpus_support': f"{corpus_percentage:.1f}%",
+                    'ai_needed': f"{100-corpus_percentage:.1f}%"
                 })
-
-                # Extract themes for lesson topics
-                content_lower = content.lower()
-                theme_keywords = {
-                    'communication': ['communicat', 'listen', 'feedback', 'conversation', 'dialogue'],
-                    'decision_making': ['decision', 'choice', 'judgment', 'analysis', 'evaluation'],
-                    'team_building': ['team', 'collaboration', 'group', 'collective', 'cooperation'],
-                    'conflict_resolution': ['conflict', 'disagreement', 'resolution', 'mediation'],
-                    'motivation': ['motivat', 'inspiration', 'engagement', 'drive'],
-                    'change_management': ['change', 'transformation', 'transition', 'adaptation'],
-                    'emotional_intelligence': ['emotional', 'empathy', 'awareness', 'regulation'],
-                    'delegation': ['delegat', 'responsibility', 'authority', 'empowerment'],
-                    'performance': ['performance', 'results', 'achievement', 'productivity'],
-                    'vision': ['vision', 'strategy', 'direction', 'future']
-                }
-
-                for theme, keywords in theme_keywords.items():
-                    for keyword in keywords:
-                        if keyword in content_lower:
-                            themes[theme] = themes.get(theme, 0) + relevant_scores[i]
-                            break
-
-            except Exception as e:
-                continue
-
-        # Sort content by relevance
-        content_analysis.sort(key=lambda x: x['score'], reverse=True)
-
-        # Generate lesson topics from themes
-        buildable_lessons = []
-        sorted_themes = sorted(themes.items(), key=lambda x: x[1], reverse=True)
-
-        for theme, score in sorted_themes:
-            if score > 0.5:  # Substantial theme presence
-                if theme == 'communication':
-                    buildable_lessons.append(f"Mastering {topic} Through Effective Communication")
-                elif theme == 'decision_making':
-                    buildable_lessons.append(f"{topic} in Critical Decision-Making")
-                elif theme == 'team_building':
-                    buildable_lessons.append(f"Building Team {topic} and Cohesion")
-                elif theme == 'conflict_resolution':
-                    buildable_lessons.append(f"Applying {topic} in Conflict Situations")
-                elif theme == 'motivation':
-                    buildable_lessons.append(f"Motivating Others Through {topic}")
-                elif theme == 'change_management':
-                    buildable_lessons.append(f"Leading {topic} During Change")
-                elif theme == 'emotional_intelligence':
-                    buildable_lessons.append(f"Emotional Intelligence and {topic}")
-                elif theme == 'delegation':
-                    buildable_lessons.append(f"Effective Delegation with {topic}")
-                elif theme == 'performance':
-                    buildable_lessons.append(f"Performance Excellence Through {topic}")
-                elif theme == 'vision':
-                    buildable_lessons.append(f"Strategic Vision and {topic} Leadership")
-
-        # Add foundational lessons if needed
-        if len(buildable_lessons) < 3:
-            buildable_lessons.extend([
-                f"Foundation Principles of {topic}",
-                f"Practical Applications of {topic}",
-                f"Advanced {topic} Leadership Techniques"
-            ])
-
-        # Calculate corpus percentage and determine feasibility
-        avg_relevance = np.mean([item['score'] for item in content_analysis]) if content_analysis else 0
-
-        # Corpus sufficiency calculation
-        corpus_strength = (high_count * 3 + medium_count * 2 + low_count * 1) / 10
-        content_depth = min(total_content_length / 50000, 2.0)
-        theme_diversity = len([t for t, s in themes.items() if s > 0.3]) / 5
-
-        corpus_percentage = min(100, (corpus_strength + content_depth + theme_diversity) * 20)
-
-        # Determine maximum feasible generation level
-        if corpus_percentage >= 70:  # 70/30 standard
-            if len(buildable_lessons) >= 12:
-                max_feasible = 'course'
-            elif len(buildable_lessons) >= 5:
-                max_feasible = 'unit'
-            else:
-                max_feasible = 'lesson'
-        elif corpus_percentage >= 60:  # 60/40 fallback
-            if len(buildable_lessons) >= 8:
-                max_feasible = 'unit'
-            else:
-                max_feasible = 'lesson'
-        else:
+            
+            buildable_content.append({
+                'type': 'lesson',
+                'title': f"Essential {topic} Leadership Principles",
+                'corpus_support': f"{corpus_percentage:.1f}%", 
+                'ai_needed': f"{100-corpus_percentage:.1f}%"
+            })
+            
+            max_feasible = 'course' if high_count >= 8 else 'unit' if high_count >= 4 else 'lesson'
+            
+        elif corpus_percentage >= 50:  # Moderate coverage
+            buildable_content.append({
+                'type': 'lesson',
+                'title': f"Foundational {topic} Leadership Concepts",
+                'corpus_support': f"{corpus_percentage:.1f}%",
+                'ai_needed': f"{100-corpus_percentage:.1f}%"
+            })
+            max_feasible = 'lesson'
+            
+        else:  # Insufficient coverage
             max_feasible = 'insufficient'
-
+        
         return {
             'status': 'success',
             'topic': topic,
-            'corpus_matches': len(content_analysis),
             'corpus_percentage': corpus_percentage,
             'max_feasible': max_feasible,
-            'buildable_lessons': buildable_lessons[:15],
-            'content_analysis': content_analysis[:10],
-            'source_documents': source_documents,
-            'themes': sorted_themes[:10],
-            'total_content_length': total_content_length,
-            'standards_met': '70/30' if corpus_percentage >= 70 else '60/40' if corpus_percentage >= 60 else 'insufficient'
+            'buildable_content': buildable_content,
+            'content_samples': relevant_content[:5],  # Top 5 most relevant
+            'high_relevance_count': int(high_count),
+            'medium_relevance_count': int(medium_count),
+            'total_relevant_documents': len(relevant_content)
         }
-
+        
     except Exception as e:
         return {
-            'status': 'error',
+            'status': 'error', 
             'message': f"Analysis error: {str(e)}",
             'corpus_percentage': 0,
             'max_feasible': 'error'
         }
 
-def generate_complete_lesson(topic, analysis=None):
-    """Generate complete 5-8 minute lesson with full content"""
-
-    if not analysis:
-        analysis = analyze_corpus_sufficiency(topic)
-
-    if analysis['status'] == 'error':
-        return f"❌ Content Analysis Failed: {analysis['message']}"
-
-    if analysis['max_feasible'] == 'insufficient':
-        return f"""❌ Insufficient Corpus Content for {topic}
-
-**Analysis Results:**
-- Corpus coverage: {analysis['corpus_percentage']:.1f}%
-- Required minimum: 60% corpus content
-- Recommendation: Add more {topic} content to your corpus
-
-**Available Content:** {analysis['corpus_matches']} relevant documents found, but insufficient depth for lesson generation meeting 60/40 standard."""
-
-    # Use first buildable lesson
-    lesson_topic = analysis['buildable_lessons'][0] if analysis['buildable_lessons'] else f"Core {topic} Leadership Principles"
-    corpus_content = analysis['content_analysis'][:5]
-
-    # Get AI enhancement for gaps
-    ai_enhancement = get_chatgpt_with_references(
-        lesson_topic,
-        "lesson",
-        f"Fill knowledge gaps for {lesson_topic} lesson. Corpus provides {analysis['corpus_percentage']:.1f}% coverage."
-    )
-
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")
-
-    lesson = f"""# 🎮 COMPLETE INTELLIGENT LESSON: {lesson_topic}
-
-**Topic Generated From:** {topic}
-**Corpus Coverage:** {analysis['corpus_percentage']:.1f}% | Standard: {analysis['standards_met']}
-**Duration:** 5-8 minutes of complete content
-**Generated:** {timestamp}
-
----
-
-## 📊 Content Standards Verification
-
-✅ **Meets {analysis['standards_met']} Standard**
-- **Corpus Content:** {analysis['corpus_percentage']:.1f}%
-- **AI Enhancement:** {100 - analysis['corpus_percentage']:.1f}%
-- **Source Documents:** {len(analysis['source_documents'])} corpus documents
-- **Evidence-Based:** All content supported by corpus evidence or academic research
-
----
-
-## 🎯 Learning Objectives
-
-By completing this lesson on **{lesson_topic}**, you will:
-
-1. **Apply** evidence-based principles of {lesson_topic} in complex leadership scenarios
-2. **Analyze** the impact of {lesson_topic} on team dynamics, performance, and organizational culture
-3. **Integrate** {lesson_topic} strategies with your existing leadership competencies and style
-4. **Evaluate** the effectiveness of different {lesson_topic} approaches across various contexts and situations
-
----
-
-## 🧠 Opening Reflection & Personal Assessment (1 minute)
-
-### Avatar Instructor Introduction
-*"Welcome to this lesson on {lesson_topic}. I'm here to guide you through an evidence-based exploration of how this critical leadership capability can transform your effectiveness. Let's start with honest self-reflection."*
-
-### Personal Leadership Assessment
-**Current Confidence Scale:** Rate your current effectiveness with {lesson_topic} (1-10): ___
-
-**Reflection Prompt:**
-*Think of a recent leadership situation where {lesson_topic} was important. Take 30 seconds to consider:*
-- What specific aspects went well?
-- What challenges did you encounter?
-- What would you approach differently if faced with the same situation again?
-- How did {lesson_topic} (or lack thereof) impact the outcome?
-
-**Write your reflection here:**
-________________________________
-________________________________
-________________________________
-
-### Avatar Instructor Transition
-*"Your reflection provides the foundation for our learning journey. Keep these experiences in mind as we explore evidence-based strategies that can enhance your {lesson_topic} capabilities."*
-
----
-
-## 📖 Evidence-Based Content Foundation: {lesson_topic} (3-4 minutes)
+def get_chatgpt_enhancement(topic, content_type, corpus_percentage, gap_description):
+    """Get ChatGPT content to fill gaps in corpus coverage"""
+    
+    try:
+        import openai
+        api_key = os.getenv("OPENAI_API_KEY")
+        
+        if not api_key:
+            return {
+                'enhanced_content': f"AI enhancement unavailable for {topic}. Proceeding with {corpus_percentage:.1f}% corpus content.",
+                'gap_areas': ["AI enhancement not available"],
+                'references': ["ChatGPT unavailable - using corpus content only"]
+            }
+        
+        openai.api_key = api_key
+        
+        prompt = f"""You are creating {content_type} content for "{topic}" leadership development.
+
+CONTEXT:
+- Corpus provides {corpus_percentage:.1f}% coverage
+- Need to fill {100-corpus_percentage:.1f}% gap
+- Gap areas: {gap_description}
+
+Create content that:
+1. Complements existing corpus knowledge (don't duplicate)
+2. Fills specific gaps in {topic} coverage
+3. Provides practical, actionable strategies
+4. Includes real-world examples and case studies
+
+Format your response with:
+- Clear, specific strategies for {topic}
+- Implementation frameworks
+- Common challenges and solutions
+- Success metrics
 
-### 🔍 Insights from Your Leadership Corpus
+CRITICAL: Provide specific citations and references for all claims."""
 
-**Your corpus contains {len(corpus_content)} high-relevance insights on this topic:**
+        response = openai.ChatCompletion.create(
+            model="gpt-4",
+            messages=[
+                {"role": "system", "content": "You are an expert leadership development specialist with extensive research background."},
+                {"role": "user", "content": prompt}
+            ],
+            max_tokens=1200,
+            temperature=0.3
+        )
+        
+        content = response.choices[0].message.content
+        
+        # Extract references (simplified)
+        references = re.findall(r'\([A-Za-z]+.*?\d{4}\)', content)
+        if not references:
+            references = ["AI-generated content - manual verification of sources recommended"]
+        
+        return {
+            'enhanced_content': content,
+            'gap_areas': [gap_description],
+            'references': references
+        }
+        
+    except Exception as e:
+        return {
+            'enhanced_content': f"AI enhancement error: {str(e)}",
+            'gap_areas': [f"Error: {str(e)}"],
+            'references': [f"Enhancement failed: {str(e)}"]
+        }
 
+def generate_content_analysis_interface(search_term):
+    """Main interface function for content analysis"""
+    
+    if not search_term or len(search_term.strip()) < 3:
+        return "⚠️ Please enter a search term (at least 3 characters)"
+    
+    try:
+        # Extract real topics from corpus
+        print(f"🔍 Extracting topics related to: {search_term}")
+        real_topics = extract_real_topics_from_corpus(search_term, max_topics=15)
+        
+        if not real_topics:
+            return f"❌ No relevant topics found in corpus for '{search_term}'. Try broader terms like 'leadership', 'management', or 'communication'."
+        
+        # Analyze top 5 topics for buildability
+        analysis_results = []
+        for i, topic in enumerate(real_topics[:5]):
+            analysis = analyze_content_sufficiency(topic)
+            if analysis['status'] == 'success':
+                analysis_results.append(analysis)
+        
+        # Format results
+        output = f"""# 📊 INTELLIGENT CONTENT ANALYSIS: "{search_term}"
+
+## 🎯 Available Topics from Your Corpus
+
+**Found {len(real_topics)} relevant topics:**
 """
+        
+        for i, topic in enumerate(real_topics[:10], 1):
+            output += f"{i}. {topic}\n"
+        
+        if len(real_topics) > 10:
+            output += f"... and {len(real_topics)-10} more topics\n"
+        
+        output += f"""
 
-    # Add corpus insights
-    for i, content in enumerate(corpus_content, 1):
-        lesson += f"""
-**💡 Corpus Insight {i}:** (Relevance: {content['score']:.3f})
-*Source: {content['topic']} | Document ID: {content['id']}*
-
-"{content['content'][:500]}{'...' if len(content['content']) > 500 else ''}"
-
-**Avatar Instructor Note:** *"This insight from your corpus demonstrates {lesson_topic.lower()} in action. Notice how the specific strategies and outcomes connect to real leadership challenges you face."*
+## 🔍 BUILDABILITY ANALYSIS (Top 5 Topics)
 
 """
+        
+        for analysis in analysis_results:
+            topic = analysis['topic']
+            corpus_pct = analysis['corpus_percentage']
+            max_level = analysis['max_feasible']
+            
+            # Status emoji
+            if max_level == 'course':
+                status_emoji = "🏆"
+            elif max_level == 'unit':
+                status_emoji = "🎯"
+            elif max_level == 'lesson':
+                status_emoji = "📚"
+            else:
+                status_emoji = "❌"
+                
+            output += f"""### {status_emoji} **{topic}**
 
-    # Add key themes
-    if analysis['themes']:
-        lesson += f"""
-### 🎯 Core Themes from Your Leadership Content
+**Corpus Coverage:** {corpus_pct:.1f}%
+**Maximum Buildable:** {max_level.title()}
+**AI Enhancement Needed:** {100-corpus_pct:.1f}%
+**Supporting Documents:** {analysis['total_relevant_documents']} documents
 
-**Primary themes detected in your corpus for {lesson_topic}:**
+**What Can Be Built:**
 """
-        for theme, score in analysis['themes'][:5]:
-            lesson += f"- **{theme.replace('_', ' ').title()}** (emphasis strength: {score:.2f})\n"
-
-    # Add AI enhancement content
-    lesson += f"""
-
-### 🤖 Research-Enhanced Understanding
-
-{ai_enhancement['content'][:800]}{'...' if len(ai_enhancement['content']) > 800 else ''}
-
-**Avatar Instructor Note:** *"This research-based content fills gaps in your corpus coverage, providing academic validation for the practical insights from your materials."*
+            
+            for buildable in analysis['buildable_content']:
+                output += f"- **{buildable['type'].title()}:** {buildable['title']} (Corpus: {buildable['corpus_support']}, AI: {buildable['ai_needed']})\n"
+            
+            if analysis['content_samples']:
+                output += f"""
+**Sample Content Preview:**
+"{analysis['content_samples'][0]['content'][:200]}..." 
+*(Relevance: {analysis['content_samples'][0]['score']:.3f})*
 
 ---
 
-## 🤔 Socratic Deep Dive & Avatar Interaction (2 minutes)
-
-### Avatar Instructor Facilitation
-*"Now let's deepen your understanding through Socratic questioning. I'll pose questions that challenge your thinking and help you connect these insights to your leadership practice. Take 30-45 seconds with each question - resist the urge to rush."*
-
-**🧠 Question 1: Personal Experience Integration**
-*"Based on the evidence we've explored, what specific situations in your leadership role have taught you the most about {lesson_topic}? How do these experiences align with or challenge the insights from your corpus?"*
-
-**Your Response:** ________________________________
-
-**Avatar Instructor Follow-up:** *"Excellent. Notice how your lived experience provides context for the evidence-based strategies we're learning."*
-
-**🧠 Question 2: Challenge Analysis**
-*"What makes {lesson_topic} particularly difficult or complex in your current leadership context? What systemic or cultural factors create barriers to effective {lesson_topic}?"*
-
-**Your Response:** ________________________________
-
-**🧠 Question 3: Success Pattern Recognition**
-*"Reflecting on times when you've been successful with {lesson_topic}, what specific factors contributed to that success? How can you systematically replicate those conditions?"*
-
-**Your Response:** ________________________________
-
-**🧠 Question 4: Growth Opportunity Identification**
-*"If you could significantly improve one aspect of your {lesson_topic} capabilities, which would have the greatest impact on your leadership effectiveness? Why?"*
-
-**Your Response:** ________________________________
-
-**🧠 Question 5: Integration & Systems Thinking**
-*"How does {lesson_topic} connect with and reinforce your other leadership strengths? Where might there be tensions between {lesson_topic} and other priorities you manage?"*
-
-**Your Response:** ________________________________
-
-**Avatar Instructor Transition:** *"These reflections provide the foundation for applying {lesson_topic} strategies. Now let's practice through gamified scenarios that simulate real leadership challenges."*
-
----
-
-## 🎮 Complete Gamification Suite: {lesson_topic} (2-3 minutes)
-
-### 🎯 Game 1: Strategic Leadership Challenge - Multi-Stakeholder {lesson_topic} Crisis (8 minutes)
-
-**Avatar Instructor Setup:** *"You're about to lead your team through a complex situation where {lesson_topic} is critical to success. This scenario will test your ability to adapt {lesson_topic} strategies to different stakeholder needs while maintaining effectiveness under pressure."*
-
-**🎬 Scenario Context:**
-You're the Director of Operations leading a cross-functional team through a critical project. Multiple pressures are testing your {lesson_topic} capabilities:
-
-- **Timeline Pressure:** Project deadline moved up 3 weeks due to market opportunity
-- **Resource Constraints:** Budget reduced 20%, two key team members reassigned
-- **Stakeholder Complexity:** Engineering, Marketing, and Sales have conflicting priorities
-- **Performance Scrutiny:** CEO personally monitoring project success
-- **Team Dynamics:** Mix of remote/in-office, varying experience levels, previous project failures affecting morale
-
-**Round 1: Immediate {lesson_topic} Response** (Choose your approach)
-
-**Avatar Instructor:** *"A crisis requires immediate action. Which {lesson_topic} approach will you take first?"*
-
-**A)** Address {lesson_topic} challenges directly in emergency team meeting
-- *Rationale: Transparency and collective problem-solving*
-- *Risk: May overwhelm team or expose vulnerabilities publicly*
-
-**B)** Handle {lesson_topic} through individual stakeholder conversations
-- *Rationale: Customized approach to different needs and concerns*
-- *Risk: May appear secretive or create inconsistent messaging*
-
-**C)** Implement structured {lesson_topic} framework immediately
-- *Rationale: Systematic approach ensures nothing is missed*
-- *Risk: May feel rigid or slow in crisis situation*
-
-**D)** Research best practices and consult mentors before action
-- *Rationale: Evidence-based decision making*
-- *Risk: Analysis paralysis while situation deteriorates*
-
-**Your Choice:** ___ **Reasoning:** ________________________________
-
-**Avatar Instructor Feedback:** *"Interesting choice. Now you'll see how stakeholders respond to your {lesson_topic} approach."*
-
-**Round 2: Multi-Stakeholder {lesson_topic} Management**
-
-Three critical stakeholders approach you with different {lesson_topic} perspectives:
-
-**💼 Sarah Chen - Engineering Lead:**
-*"Your {lesson_topic} approach isn't accounting for technical complexity. We need more time for quality work, not rushed solutions that will create technical debt."*
-
-**📊 Marcus Williams - Sales Director:**
-*"The market window is closing. Can you help my team understand how your {lesson_topic} strategy will accelerate delivery without compromising what customers need?"*
-
-**👩‍💼 Dr. Lisa Rodriguez - CEO:**
-*"I need you to prove that your {lesson_topic} approach will deliver results. Show me metrics and concrete outcomes, not just process improvements."*
-
-**Avatar Instructor Challenge:** *"Design your response strategy for each stakeholder. Consider how {lesson_topic} principles apply differently to technical, commercial, and executive concerns."*
-
-**Your Sarah Chen Response:** ________________________________
-
-**Your Marcus Williams Response:** ________________________________
-
-**Your Dr. Rodriguez Response:** ________________________________
-
-**Round 3: Crisis Adaptation & Innovation**
-
-**Avatar Instructor Escalation:** *"Midway through implementation, a major change forces complete strategy revision. Your {lesson_topic} approach must evolve rapidly."*
-
-**🔥 Crisis Event:** Key technology platform fails, requiring manual processes for 2 weeks minimum. Your {lesson_topic} strategy must adapt while maintaining team confidence and stakeholder trust.
-
-**Innovation Challenge:** Create a new {lesson_topic} approach that turns this obstacle into an opportunity for:
-- Enhanced team resilience and problem-solving
-- Stronger stakeholder relationships through crisis leadership
-- Long-term competitive advantage through improved processes
-
-**Your Crisis {lesson_topic} Strategy:**
-1. **Immediate Actions (Today):** ________________________________
-2. **Short-term Adaptations (This Week):** ________________________________
-3. **Opportunity Creation (This Month):** ________________________________
-
-**Avatar Instructor Debrief:** *"Excellent work navigating complexity. Notice how {lesson_topic} principles adapted to changing circumstances while maintaining core effectiveness."*
-
-### 🃏 Game 2: Strategic Priority Matrix - {lesson_topic} Implementation Challenge (5 minutes)
-
-**Avatar Instructor Introduction:** *"You're consulting with a leadership team that wants to improve their {lesson_topic} capabilities but has limited time and resources. Help them create a strategic implementation roadmap."*
-
-**🎲 Implementation Elements for {lesson_topic}:**
-
-**🔥 High-Impact Options:**
-• Clear communication systems establishing {lesson_topic} expectations and feedback loops
-• Leadership modeling of {lesson_topic} behaviors with consistent, visible demonstration
-• Measurement and analytics systems tracking {lesson_topic} effectiveness and outcomes
-• Cultural integration embedding {lesson_topic} principles into organizational values and recognition
-• Strategic alignment ensuring {lesson_topic} initiatives support broader business objectives
-
-**⚡ Medium-Impact Options:**
-• Comprehensive training programs developing {lesson_topic} skills across all leadership levels
-• Process optimization streamlining workflows to support effective {lesson_topic} implementation
-• Technology tools and platforms enabling better {lesson_topic} practice and collaboration
-• Cross-functional integration breaking down silos to enable {lesson_topic} collaboration
-• External partnerships leveraging outside expertise for {lesson_topic} development and benchmarking
-
-**🔧 Foundation Options:**
-• Individual coaching providing personalized {lesson_topic} development for key leaders
-• Team-building activities and experiences that build {lesson_topic} trust and cohesion
-• Documentation and standardization creating clear guidelines for {lesson_topic} practices
-• Recognition programs celebrating {lesson_topic} successes and learning from failures
-• Regular feedback mechanisms gathering input on {lesson_topic} effectiveness and improvement
-
-**Game Mechanics:**
-
-**Phase 1: Strategic Prioritization** (3 points each)
-Rank all 15 elements in order of implementation priority for {lesson_topic} development. Consider:
-- **Speed of Impact:** Which changes improve {lesson_topic} fastest?
-- **Resource Requirements:** What can be implemented with current capabilities?
-- **Strategic Value:** Which elements create foundation for future {lesson_topic} growth?
-- **Risk vs. Reward:** What offers best return on {lesson_topic} investment?
-
-**Your Priority Ranking (1-15):**
-High-Impact: ___, ___, ___, ___, ___
-Medium-Impact: ___, ___, ___, ___, ___
-Foundation: ___, ___, ___, ___, ___
-
-**Phase 2: Implementation Matrix** (5 points)
-Place your top 8 elements into strategic quadrants:
-
-**🟢 Quick Wins (High Impact, Low Effort):**
-1. _________________ 2. _________________
-
-**🟡 Strategic Projects (High Impact, High Effort):**
-1. _________________ 2. _________________
-
-**🟠 Fill-In Projects (Low Impact, Low Effort):**
-1. _________________ 2. _________________
-
-**🔴 Questionable (Low Impact, High Effort):**
-1. _________________ 2. _________________
-
-**Phase 3: Timeline Development** (7 points)
-Create 12-month {lesson_topic} implementation roadmap:
-
-**Months 1-3 (Foundation Building):** ________________________________
-
-**Months 4-6 (Momentum Creation):** ________________________________
-
-**Months 7-9 (Strategic Initiatives):** ________________________________
-
-**Months 10-12 (Integration & Evolution):** ________________________________
-
-**Avatar Instructor Assessment:** *"Your prioritization shows strategic thinking about {lesson_topic} implementation. Notice how quick wins create momentum for larger transformations."*
-
-### 🔗 Game 3: Dynamic Behavior-Outcome Mapping - {lesson_topic} Systems Challenge (5 minutes)
-
-**Avatar Instructor Setup:** *"This game develops your systems thinking about {lesson_topic}. You'll map complex relationships between behaviors and outcomes, then design integrated strategies."*
-
-**🎯 Level 1: Direct Behavior-Outcome Matching**
-
-**{lesson_topic} Leadership Behaviors:**
-1. **Consistent Communication:** Regular, transparent updates about {lesson_topic} expectations, progress, and challenges
-2. **Vulnerability and Learning:** Leader openly shares {lesson_topic} mistakes, learning, and growth opportunities
-3. **Inclusive Decision-Making:** Involving team members in {lesson_topic} choices that directly affect their work
-4. **Recognition and Celebration:** Acknowledging both {lesson_topic} successes and meaningful learning attempts
-5. **Development Investment:** Dedicated time and resources for growing team {lesson_topic} capabilities
-
-**Match with Likely Outcomes:**
-A. **Increased Psychological Safety:** Team feels safe to take {lesson_topic} risks and share concerns
-B. **Enhanced Trust and Credibility:** Stronger stakeholder confidence in leadership {lesson_topic} decisions
-C. **Improved Innovation and Creativity:** More breakthrough approaches and solutions using {lesson_topic}
-D. **Better Performance and Results:** Measurable improvements in {lesson_topic} effectiveness metrics
-E. **Sustained Culture Change:** {lesson_topic} behaviors continue even when leader isn't directly present
-
-**Your Matching:**
-1-___, 2-___, 3-___, 4-___, 5-___
-
-**🎯 Level 2: Systems Integration Challenge**
-
-**Avatar Instructor:** *"Now design reinforcing loops. How do these {lesson_topic} behaviors create positive cycles that strengthen each other?"*
+"""
+        
+        # Add summary recommendations
+        best_analysis = max(analysis_results, key=lambda x: x['corpus_percentage']) if analysis_results else None
+        
+        if best_analysis:
+            output += f"""
 
-**Systems Mapping Exercise:**
-Choose 3 behaviors and map how they reinforce each other:
+## 🎯 RECOMMENDATION
 
-**Behavior A:** _________________
-**→ Creates:** _________________
-**→ Which Enables:** _________________
-**→ Which Reinforces:** _________________
+**Best Topic for Generation:** {best_analysis['topic']}
+- **Corpus Coverage:** {best_analysis['corpus_percentage']:.1f}%
+- **Recommended Build:** {best_analysis['max_feasible'].title()}
+- **Quality Level:** {'Excellent' if best_analysis['corpus_percentage'] >= 70 else 'Good' if best_analysis['corpus_percentage'] >= 50 else 'Basic'}
 
-**Advanced Challenge:** *Design a {lesson_topic} intervention that simultaneously activates all 5 behaviors*
+**Next Step:** Use this topic to generate actual content with the lesson/unit/course builder.
+"""
+        
+        return output
+        
+    except Exception as e:
+        return f"❌ Analysis error: {str(e)}\n\nPlease try again with a different search term."
+
+def generate_actual_lesson(topic):
+    """Generate actual lesson content using the topic analysis"""
+    
+    # First analyze the topic
+    analysis = analyze_content_sufficiency(topic)
+    
+    if analysis['status'] == 'error':
+        return f"❌ Cannot generate lesson: {analysis['message']}"
+    
+    if analysis['max_feasible'] == 'insufficient':
+        return f"""❌ Insufficient Content for "{topic}"
 
-**Your Integrated {lesson_topic} Strategy:** ________________________________
+**Analysis Results:**
+- Corpus coverage: {analysis['corpus_percentage']:.1f}%
+- Required minimum: 50% for lesson generation
+- Supporting documents: {analysis.get('total_relevant_documents', 0)}
+
+**Recommendation:** Try a broader topic or add more content to your corpus."""
+    
+    # Get enhancement content if needed
+    gap_percentage = 100 - analysis['corpus_percentage']
+    enhancement = get_chatgpt_enhancement(
+        topic, 
+        "lesson", 
+        analysis['corpus_percentage'],
+        f"Fill {gap_percentage:.1f}% content gap for comprehensive {topic} lesson"
+    )
+    
+    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")
+    
+    lesson_content = f"""# 🎯 COMPLETE LESSON: {topic}
 
-**Avatar Instructor Systems Analysis:** *"Excellent systems thinking. Notice how {lesson_topic} behaviors create reinforcing cycles that multiply leadership effectiveness."*
+**Generated:** {timestamp}
+**Content Source Analysis:**
+- 📚 Corpus Content: {analysis['corpus_percentage']:.1f}%
+- 🤖 AI Enhancement: {100-analysis['corpus_percentage']:.1f}%
+- 📊 Supporting Documents: {analysis['total_relevant_documents']} relevant documents
+- ✅ Quality Standard: {'70/30' if analysis['corpus_percentage'] >= 70 else '60/40' if analysis['corpus_percentage'] >= 60 else '50/50'}
 
 ---
 
-## ✅ Complete Assessment Suite & Avatar Certification
-
-### Avatar Instructor Assessment Introduction
-*"You've engaged deeply with {lesson_topic} through evidence, reflection, and practice. Now demonstrate your mastery through assessment pathways designed to match your learning goals and time availability."*
-
-### 🏃‍♂️ Speed Mastery Certification (10 minutes)
-**Avatar Instructor:** *"Quick application check for busy leaders who need immediate implementation guidance."*
-
-1. **Evidence Integration:** What are 3 key evidence-based insights about {lesson_topic} from this lesson that challenge or confirm your previous assumptions?
-
-**Your Response:** ________________________________
-
-2. **Immediate Application:** Describe one current leadership challenge where you'll apply {lesson_topic} principles within the next 48 hours.
-
-**Your Response:** ________________________________
-
-3. **Behavioral Commitment:** What's one specific {lesson_topic} behavior you'll practice consistently this week?
-
-**Your Response:** ________________________________
-
-**Avatar Instructor Certification:** *"Your responses demonstrate understanding. You've earned Speed Mastery certification for {lesson_topic}."*
-
-**Passing Standard:** Clear understanding + specific application plan + behavioral commitment
-
-### 🎮 Complete Quest Mastery (20 minutes)
-**Avatar Instructor:** *"Comprehensive engagement for leaders who want deep integration and implementation planning."*
-
-**Quest Requirements:**
-✓ Complete all 3 gamified activities with quality reflection and strategic thinking
-✓ Answer all 5 Socratic questions with depth, connecting to personal experience
-✓ Create detailed {lesson_topic} implementation plan with timeline and metrics
-✓ Design approach to teach {lesson_topic} concepts to a colleague or team member
-
-**Implementation Planning Template:**
-
-**My {lesson_topic} Development Strategy:**
-- **30-Day Goal:** ________________________________
-- **Key Behaviors to Practice:** ________________________________
-- **Success Metrics:** ________________________________
-- **Support Resources Needed:** ________________________________
-
-**Teaching Design Challenge:**
-*How would you teach the most important {lesson_topic} insight to someone else?*
-
-**Your Teaching Approach:** ________________________________
-
-**Avatar Instructor Certification:** *"Outstanding comprehensive engagement. You've earned Complete Quest Mastery for {lesson_topic}."*
-
-### 🏆 Innovation & Leadership Mastery Plus (30 minutes)
-**Avatar Instructor:** *"Advanced certification for leaders who want to create value for others and drive organizational change."*
-
-**Mastery Plus Requirements:**
-- All Complete Quest elements PLUS:
-- Create novel {lesson_topic} solution for your specific organizational context
-- Design 5-minute {lesson_topic} lesson for colleagues using lesson insights
-- Research and analyze one additional {lesson_topic} resource, comparing findings
-- Plan comprehensive approach to develop {lesson_topic} capabilities in your team
-
-**Innovation Challenge:**
-*Design an original {lesson_topic} approach that addresses a specific challenge in your organization*
-
-**Your Innovation:** ________________________________
-
-**Leadership Development Plan:**
-*How will you develop {lesson_topic} capabilities in others?*
-
-**Your Plan:** ________________________________
-
-**Avatar Instructor Certification:** *"Exceptional leadership thinking and innovation. You've earned Innovation & Leadership Mastery Plus for {lesson_topic}."*
+## 🎯 Learning Objectives
 
-## 🎖️ Achievement Badge System
+By the end of this lesson on **{topic}**, you will:
 
-**🥉 {lesson_topic} Explorer:** Completed learning journey with solid understanding
-**🥈 {lesson_topic} Practitioner:** Demonstrated practical application skills and implementation planning
-**🥇 {lesson_topic} Innovator:** Showed creative problem-solving and organizational application capability
-**💎 {lesson_topic} Leader:** Created value for others through expertise and teaching capability
+1. **Understand** the core principles and frameworks of {topic} in leadership contexts
+2. **Apply** evidence-based {topic} strategies to real-world leadership challenges  
+3. **Analyze** the impact of {topic} on team performance and organizational outcomes
+4. **Evaluate** different approaches to {topic} for various leadership scenarios
 
 ---
 
-## 🔄 Integration & Next Steps (1 minute)
-
-### Avatar Instructor Final Integration
-*"Let's consolidate your learning and create momentum for continued growth."*
-
-### 🎯 Growth Tracking & Reflection
-**Return to your opening assessment:**
-- How has your understanding of {lesson_topic} evolved through this evidence-based lesson?
-- What insights most surprised you or challenged your previous assumptions?
-- On the 1-10 confidence scale, how would you rate your {lesson_topic} capabilities now? ___
-
-### 🚀 Action Commitment & Implementation
-**Avatar Instructor:** *"Learning without action is merely intellectual entertainment. Commit to specific implementation."*
-
-**My {lesson_topic} Power Move:** What specific {lesson_topic} action will you take in the next 48 hours?
-
-**Action:** ________________________________
-**Context/Situation:** ________________________________
-**Success Measure:** ________________________________
-**Accountability Partner:** ________________________________
+## 📚 Foundation Knowledge (From Your Leadership Corpus)
 
-### 🔮 Future Learning Questions
-**Avatar Instructor:** *"Curiosity drives continued growth. What questions will guide your ongoing {lesson_topic} development?"*
-
-**Future Exploration:**
-1. ________________________________
-2. ________________________________
-
-**Avatar Instructor Closing:** *"You've completed a comprehensive, evidence-based exploration of {lesson_topic}. Your corpus insights, combined with research validation and practical application, provide a solid foundation for leadership growth. Remember: mastery comes through consistent practice and reflection. You have the tools—now use them with intention and commitment."*
-
----
-
-*Lesson Generated: {timestamp} | Corpus Standard: {analysis['standards_met']} | Full Content: ✅ | Avatar Integration: ✅*
 """
+    
+    # Add corpus content
+    if analysis['content_samples']:
+        lesson_content += "**Key Insights from Your Corpus:**\n\n"
+        
+        for i, sample in enumerate(analysis['content_samples'][:3], 1):
+            lesson_content += f"""**Insight {i}** (Relevance: {sample['score']:.3f})
+*Source: {sample['source_topic']}*
 
-    # Generate comprehensive source attribution appendix
-    lesson += generate_source_attribution_appendix(analysis, ai_enhancement, "lesson")
-
-    return lesson
-
-def generate_source_attribution_appendix(analysis, ai_enhancement, content_type):
-    """Generate comprehensive source attribution appendix"""
-
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")
-
-    appendix = f"""
-
-{'='*100}
-# 📚 SOURCE ATTRIBUTION APPENDIX
-{'='*100}
-
-**Content Type:** {content_type.title()}
-**Topic:** {analysis.get('topic', 'Leadership Development')}
-**Attribution Generated:** {timestamp}
-**Content Standards:** {analysis.get('standards_met', 'Standard')} (Corpus {analysis.get('corpus_percentage', 0):.1f}% / AI {100 - analysis.get('corpus_percentage', 0):.1f}%)
+"{sample['content'][:400]}{'...' if len(sample['content']) > 400 else ''}"
 
 ---
 
-## 🎯 CORPUS SOURCE DOCUMENTATION
-
-### Primary Source Documents from Your Leadership Corpus
-
-**Total Corpus Documents Contributing:** {len(analysis.get('source_documents', {}))}
-**Content Entries Analyzed:** {analysis.get('corpus_matches', 0)}
-**Average Relevance Score:** {analysis.get('corpus_percentage', 0)/10:.3f}
-
 """
+    
+    # Add AI enhancement
+    lesson_content += f"""## 🤖 Enhanced Understanding
 
-    # Document corpus sources with detailed attribution
-    if analysis.get('source_documents'):
-        for doc_name, doc_entries in analysis['source_documents'].items():
-            if doc_entries:
-                top_score = max([entry['score'] for entry in doc_entries])
-                total_content = sum([entry['content_length'] for entry in doc_entries])
-                entry_count = len(doc_entries)
-
-                appendix += f"""
-### 📄 {doc_name}
-- **Document Type:** Corpus Source
-- **Relevance Score:** {top_score:.3f} (0-1 scale)
-- **Content Contributions:** {entry_count} individual entries
-- **Total Content Used:** {total_content:,} characters
-- **Contribution Level:** {'High' if top_score >= 0.4 else 'Moderate' if top_score >= 0.3 else 'Supporting'}
-- **Usage Context:** {'Primary content source' if top_score >= 0.4 else 'Supporting evidence' if top_score >= 0.3 else 'Background validation'}
+{enhancement['enhanced_content']}
 
-**Specific Document Entries:**
+**References:**
 """
-                for j, entry in enumerate(doc_entries[:3], 1):  # Top 3 entries per document
-                    appendix += f"- Entry {j}: ID {entry['id']} (Score: {entry['score']:.3f}, Length: {entry['content_length']} chars)\n"
-
-                if len(doc_entries) > 3:
-                    appendix += f"- Plus {len(doc_entries)-3} additional entries from this document\n"
-    else:
-        appendix += "\n❌ **No Corpus Documents Contributed:** Insufficient corpus coverage for this topic\n"
-
-    appendix += f"""
-
-### 📊 Corpus Utilization Analysis
-
-**Content Quality Assessment:**
-- **High Relevance Content (>0.4 score):** {len([c for c in analysis.get('content_analysis', []) if c['score'] > 0.4])} pieces
-- **Medium Relevance Content (0.3-0.4):** {len([c for c in analysis.get('content_analysis', []) if 0.3 <= c['score'] <= 0.4])} pieces
-- **Supporting Content (0.2-0.3):** {len([c for c in analysis.get('content_analysis', []) if 0.2 <= c['score'] < 0.3])} pieces
-
-**Theme Integration:**
-{f"Primary themes from corpus: {', '.join([theme for theme, score in analysis.get('themes', [])[:5]])}" if analysis.get('themes') else "Limited theme identification from corpus"}
-
-**Content Authenticity:**
-- **Corpus Authenticity Level:** {'High' if analysis.get('corpus_percentage', 0) >= 70 else 'Moderate' if analysis.get('corpus_percentage', 0) >= 60 else 'Limited'}
-- **Personal Relevance:** Content aligned with themes and approaches in your leadership materials
-- **Evidence Foundation:** Major concepts supported by corpus evidence or established research
+    
+    for ref in enhancement['references'][:5]:
+        lesson_content += f"- {ref}\n"
+    
+    # Add interactive elements
+    lesson_content += f"""
 
 ---
 
-## 🤖 AI ENHANCEMENT DOCUMENTATION
+## 🎮 Interactive Application
 
-### ChatGPT Content Contributions
+### 🤔 Reflection Questions
 
-**AI Usage Level:** {100 - analysis.get('corpus_percentage', 0):.1f}% of total content
-**AI Role:** {'Minimal supplementation' if analysis.get('corpus_percentage', 0) >= 70 else 'Moderate gap filling' if analysis.get('corpus_percentage', 0) >= 60 else 'Significant content generation'}
+1. **Personal Experience:** Describe a situation where {topic} was critical to your leadership success or failure. What specific elements made the difference?
 
-"""
-
-    # Document AI contributions with academic references
-    if ai_enhancement.get('content') and not ai_enhancement.get('error'):
-        appendix += f"""
-### 📖 Academic References Provided by AI Enhancement
-
-**Reference Quality:** {'High - Specific academic citations provided' if len(ai_enhancement.get('references', [])) >= 5 else 'Moderate - Some references provided' if ai_enhancement.get('references') else 'Limited - Manual verification required'}
-
-**Specific Academic Sources Cited by ChatGPT:**
-"""
-
-        if ai_enhancement.get('references'):
-            for i, ref in enumerate(ai_enhancement['references'][:10], 1):  # Top 10 references
-                appendix += f"{i}. {ref}\n"
+2. **Challenge Analysis:** What barriers in your current environment make {topic} difficult to implement effectively?
 
-            if len(ai_enhancement['references']) > 10:
-                appendix += f"Plus {len(ai_enhancement['references'])-10} additional references in AI content\n"
-        else:
-            appendix += "❌ No specific academic references extracted from AI content\n❗ Manual verification required for all AI-generated claims\n"
+3. **Implementation Planning:** Based on the evidence presented, what 2-3 specific actions will you take to improve your {topic} capabilities?
 
-        appendix += f"""
+### 🎯 Practical Exercise
 
-### 🔍 AI Content Verification Requirements
+**Scenario:** You're leading a team through a complex project where {topic} is essential for success.
 
-**Critical Verification Needed:**
-- **Academic Claims:** All research citations require manual verification for accuracy
-- **Statistical Data:** Numerical claims need source validation
-- **Methodological Assertions:** Leadership frameworks require evidence confirmation
-- **Best Practices:** Implementation recommendations need case study validation
-
-**Verification Process:**
-1. **Cross-Reference Citations:** Verify all academic sources exist and support claims
-2. **Check Publication Details:** Confirm author names, publication years, and page numbers
-3. **Validate Research Findings:** Ensure studies support the conclusions presented
-4. **Assess Source Quality:** Evaluate credibility and relevance of cited research
-5. **Update References:** Replace any invalid sources with verified alternatives
-
-"""
-    else:
-        appendix += f"""
-### ❌ Limited AI Enhancement Available
-
-**AI Status:** {ai_enhancement.get('error', 'Content generation limited')}
-**Impact:** Content relies more heavily on corpus sources and established leadership principles
-**Recommendation:** Consider adding external research validation for enhanced academic rigor
-"""
+**Your Challenge:** Design a specific action plan that incorporates the {topic} principles covered in this lesson.
 
-    appendix += f"""
+**Consider:**
+- How will you measure the effectiveness of your {topic} approach?
+- What potential obstacles might you encounter?
+- How will you adapt your {topic} strategy based on team feedback?
 
 ---
 
-## ✅ SOURCE VERIFICATION & QUALITY ASSURANCE
+## ✅ Lesson Complete
 
-### 🔍 Content Validation Checklist
+**Next Steps:**
+- Practice the {topic} techniques in your current leadership role
+- Observe how {topic} impacts your team dynamics and results
+- Consider taking the full unit or course on {topic} for deeper development
 
-**Corpus Content Validation:**
-✅ All corpus sources identified with document names and relevance scores
-✅ Content usage clearly attributed to specific corpus documents
-✅ Corpus contribution percentage accurately calculated and reported
-✅ Theme analysis based on actual corpus content rather than assumptions
-
-**AI Content Validation:**
-{'✅ Academic references provided and listed for verification' if ai_enhancement.get('references') else '❌ Limited academic references - manual research required'}
-{'✅ Reference extraction completed with systematic identification' if ai_enhancement.get('references') else '❌ Reference quality requires improvement'}
-✅ AI contribution level clearly identified and separated from corpus content
-✅ Verification requirements explicitly stated for all AI-generated claims
-
----
-
-*Source Attribution Complete: {timestamp} | Corpus Sources: {len(analysis.get('source_documents', {}))} | AI References: {len(ai_enhancement.get('references', []))} | Verification: Required*
+**Content Sources Utilized:**
+- Corpus documents: {analysis['total_relevant_documents']} 
+- AI enhancement: {'Yes' if enhancement['enhanced_content'] else 'No'}
+- Quality standard met: {'✅' if analysis['corpus_percentage'] >= 50 else '❌'}
 """
-
-    return appendix
-
-# Create interface
-def create_complete_intelligent_interface():
-    with gr.Blocks(title="🧠 Complete Intelligent Course Generator", theme=gr.themes.Soft()) as demo:
-
-        gr.Markdown("""
-        # 🧠 Complete Intelligent Course Generator
-        **Full Content + 70/30 Corpus Standards + Complete Source Attribution**
-
-        🎯 **COMPLETE CONTENT:** Every lesson fully written with Socratic method + gamification
-        📊 **70/30 STANDARD:** 70% corpus, 30% AI enhancement (60/40 minimum fallback)
-        🎮 **FULL GAMIFICATION:** All scenarios, card sorts, matching, role-plays completely written
-        📚 **SOURCE ATTRIBUTION:** Comprehensive appendix showing all corpus + AI sources
+    
+    return lesson_content
+
+# Gradio Interface
+def create_interface():
+    
+    with gr.Blocks(title="🧠 Intelligent Course Generator", theme=gr.themes.Soft()) as interface:
+        
+        gr.HTML("""
+        <div style='text-align: center; padding: 20px;'>
+            <h1>🧠 Intelligent Course Generator</h1>
+            <p><strong>Analyzes your corpus → Identifies buildable topics → Generates real content</strong></p>
+        </div>
         """)
-
-        with gr.Row():
-            with gr.Column():
-                topic_input = gr.Textbox(
-                    placeholder="Enter leadership topic (e.g., Emotional Intelligence, Communication, Delegation)",
-                    label="🎯 Leadership Topic for Analysis",
-                    lines=1
-                )
-
-                gr.Markdown("### 🔍 Step 1: Content Sufficiency Analysis")
-
-                analyze_btn = gr.Button("📊 Analyze Corpus Sufficiency", variant="primary", size="lg")
-
-                gr.Markdown("### 🎮 Step 2: Generate Complete Content")
-
+        
+        with gr.Tabs():
+            
+            # Content Analysis Tab
+            with gr.TabItem("📊 Content Analysis"):
+                gr.HTML("<h3>Step 1: Analyze What You Can Build</h3>")
+                
                 with gr.Row():
-                    lesson_btn = gr.Button("📚 Complete Lesson\n(5-8 min full content)", variant="secondary", size="lg")
-                    unit_btn = gr.Button("📋 Complete Unit\n(up to 1 hour all lessons)", variant="secondary", size="lg")
-                    course_btn = gr.Button("🎓 Complete Course\n(up to 5 hours all content)", variant="secondary", size="lg")
-
-            with gr.Column():
-                output_display = gr.Textbox(
-                    label="🧠 Analysis Results & Complete Generated Content",
-                    lines=40,
-                    show_copy_button=True,
-                    placeholder="Step 1: Click 'Analyze Corpus Sufficiency' to check 70/30 standards\nStep 2: Generate based on analysis\n\n✅ All content completely written (no summaries)\n✅ Full gamification in every lesson\n✅ Complete source attribution appendix\n✅ 70/30 corpus/AI standards enforced"
-                )
-
-        with gr.Accordion("📊 Content Standards & Attribution", open=False):
-            gr.Markdown("""
-            **70/30 Standard (Preferred):** 70% from your corpus, 30% AI enhancement
-            **60/40 Fallback:** 60% corpus minimum if insufficient content for 70/30
-            **Complete Content:** All lessons fully written with Socratic method + gamification
-            **Source Attribution:** Detailed appendix shows all corpus documents + AI references
-            **Academic Verification:** AI references require manual verification for validity
-            **Length Limits:** Lessons (5-8 min), Units (≤1 hour), Courses (≤5 hours)
-            """)
-
-        # Event handlers with sufficiency checking
-        def analyze_sufficiency_safe(topic):
-            if not topic.strip():
-                return "❌ Please enter a leadership topic to analyze."
-            try:
-                analysis = analyze_corpus_sufficiency(topic)
-                if analysis['status'] == 'error':
-                    return f"❌ Analysis Error: {analysis['message']}"
-
-                timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")
-
-                return f"""# 🔍 CORPUS SUFFICIENCY ANALYSIS REPORT
-
-**Topic Analyzed:** {analysis['topic']}
-**Analysis Timestamp:** {timestamp}
-
----
-
-## 📊 Content Standards Assessment
-
-### 🎯 Corpus Coverage Results
-- **Corpus Content Available:** {analysis['corpus_percentage']:.1f}%
-- **Standards Assessment:** {analysis['standards_met']} ({'✅ Meets preferred 70/30 standard' if analysis['standards_met'] == '70/30' else '⚠️ Meets minimum 60/40 standard' if analysis['standards_met'] == '60/40' else '❌ Insufficient for standards'})
-- **Maximum Feasible Generation:** {analysis['max_feasible'].title()}
-- **Available Lessons:** {len(analysis['buildable_lessons'])}
-
-### 🎮 Generation Feasibility
-
-**What Can Be Generated with Current Corpus:**
-
-{f"✅ **{analysis['max_feasible'].title()} Generation Supported**" if analysis['max_feasible'] != 'insufficient' else "❌ **Insufficient Content for Standards-Based Generation**"}
-
-{"**Buildable Lesson Topics:**" if analysis['buildable_lessons'] else "**No Specific Lessons Identified**"}
-"""
-                + "\n".join([f"{i}. {lesson}" for i, lesson in enumerate(analysis['buildable_lessons'][:10], 1)]) +
-                (f"\n...plus {len(analysis['buildable_lessons'])-10} additional lessons" if len(analysis['buildable_lessons']) > 10 else "") + f"""
-
----
-
-## 📚 Corpus Content Analysis
-
-**Source Documents Available:** {len(analysis['source_documents'])}
-**Content Matches Found:** {analysis['corpus_matches']}
-**Total Content Volume:** {analysis.get('total_content_length', 0):,} characters
-
-**Primary Content Themes:**
-"""
-                + "\n".join([f"• {theme.replace('_', ' ').title()} (strength: {score:.2f})" for theme, score in analysis['themes'][:8]]) + f"""
-
----
-
-## 🚀 Generation Recommendations
-
-{"**✅ Proceed with Generation**" if analysis['max_feasible'] != 'insufficient' else "**❌ Cannot Generate with Current Standards**"}
-
-{f"**Recommended Action:** Generate {analysis['max_feasible']} using available corpus content with {analysis['standards_met']} standard" if analysis['max_feasible'] != 'insufficient' else "**Required Action:** Add more corpus content for this topic to meet 60/40 minimum standard"}
-
-**Content Quality:** {'High corpus authenticity with minimal AI supplementation' if analysis['standards_met'] == '70/30' else 'Good corpus foundation with moderate AI enhancement' if analysis['standards_met'] == '60/40' else 'Insufficient corpus coverage for quality generation'}
-
----
-
-*Analysis Complete: {timestamp} | Standards: {analysis['standards_met']} | Feasible: {analysis['max_feasible']}*
-"""
-            except Exception as e:
-                return f"❌ Sufficiency analysis failed: {str(e)}\n\nPlease try again."
-
-        def generate_complete_lesson_safe(topic):
-            if not topic.strip():
-                return "❌ Please enter a leadership topic."
-            try:
-                return generate_complete_lesson(topic)
-            except Exception as e:
-                return f"❌ Complete lesson generation failed: {str(e)}\n\nPlease try again."
-
-        # Placeholder functions for unit and course (simplified for space)
-        def generate_complete_unit_safe(topic):
-            if not topic.strip():
-                return "❌ Please enter a leadership topic."
-            return "🚧 Complete unit generation - use lesson generation for now while full system is deployed."
-
-        def generate_complete_course_safe(topic):
-            if not topic.strip():
-                return "❌ Please enter a leadership topic."
-            return "🚧 Complete course generation - use lesson generation for now while full system is deployed."
-
-        analyze_btn.click(fn=analyze_sufficiency_safe, inputs=topic_input, outputs=output_display)
-        lesson_btn.click(fn=generate_complete_lesson_safe, inputs=topic_input, outputs=output_display)
-        unit_btn.click(fn=generate_complete_unit_safe, inputs=topic_input, outputs=output_display)
-        course_btn.click(fn=generate_complete_course_safe, inputs=topic_input, outputs=output_display)
-
-        gr.Markdown("""
-        ---
-        **🧠 COMPLETE INTELLIGENT GENERATION:**
-        • **70/30 Standards:** Preferred corpus/AI ratio with 60/40 fallback
-        • **Complete Content:** All lessons fully written with Socratic method + full gamification
-        • **Source Attribution:** Comprehensive appendix with corpus + AI reference documentation
-        • **Academic Rigor:** AI references provided for manual verification
-        • **No Summaries:** Every lesson, unit, and course completely developed
-
-        **🎯 Generates only what your corpus can support with complete transparency!**
+                    analysis_input = gr.Textbox(
+                        label="🔍 Search Term", 
+                        placeholder="Enter topic area (e.g., 'communication', 'leadership', 'conflict')",
+                        lines=1
+                    )
+                    analyze_btn = gr.Button("🔍 Analyze Content", variant="primary")
+                
+                analysis_output = gr.Markdown(label="Analysis Results")
+                analyze_btn.click(generate_content_analysis_interface, analysis_input, analysis_output)
+            
+            # Lesson Generator Tab  
+            with gr.TabItem("📚 Lesson Generator"):
+                gr.HTML("<h3>Step 2: Generate Actual Lesson Content</h3>")
+                
+                with gr.Row():
+                    lesson_topic = gr.Textbox(
+                        label="🎯 Specific Topic", 
+                        placeholder="Enter the exact topic from analysis (e.g., 'Strategic Communication')",
+                        lines=1
+                    )
+                    generate_lesson_btn = gr.Button("📚 Generate Complete Lesson", variant="primary")
+                
+                lesson_output = gr.Markdown(label="Complete Lesson Content")
+                generate_lesson_btn.click(generate_actual_lesson, lesson_topic, lesson_output)
+            
+            # Quick Generator Tab
+            with gr.TabItem("⚡ Quick Generate"):
+                gr.HTML("<h3>Direct Generation (Skip Analysis)</h3>")
+                
+                with gr.Row():
+                    quick_topic = gr.Textbox(
+                        label="🎯 Topic", 
+                        placeholder="Enter any leadership topic",
+                        lines=1
+                    )
+                    quick_generate_btn = gr.Button("⚡ Generate Now", variant="secondary")
+                
+                quick_output = gr.Markdown(label="Generated Content")
+                quick_generate_btn.click(generate_actual_lesson, quick_topic, quick_output)
+        
+        gr.HTML("""
+        <div style='margin-top: 30px; padding: 15px; background-color: #f0f0f0; border-radius: 10px;'>
+            <h4>🔍 How This Works:</h4>
+            <ol>
+                <li><strong>Content Analysis:</strong> Searches your corpus for relevant topics</li>
+                <li><strong>Sufficiency Check:</strong> Determines what can be built with your content</li> 
+                <li><strong>Gap Identification:</strong> Shows what needs AI enhancement</li>
+                <li><strong>Real Content Generation:</strong> Creates actual lessons using your topics</li>
+            </ol>
+            <p><strong>Result:</strong> Real, complete lessons on actual topics from your corpus - not generic "micro lessons"!</p>
+        </div>
         """)
+    
+    return interface
 
-    return demo
-
+# Launch the interface
 if __name__ == "__main__":
-    demo = create_complete_intelligent_interface()
-    demo.launch(share=False)  # Let Gradio find available port
+    interface = create_interface()
+    interface.launch(share=True, server_name="0.0.0.0", server_port=7860)

requirements.txt

@@ -1,7 +1,9 @@
 gradio==4.44.0
-datasets==2.18.0
-sentence-transformers==2.2.2
-scikit-learn==1.3.2
-numpy==1.24.3
+datasets==2.20.0
+sentence-transformers==2.7.0
+scikit-learn==1.5.1
+numpy==1.26.4
 openai==0.28.1
-huggingface_hub==0.19.4
+huggingface-hub==0.24.0
+transformers==4.44.0
+torch==2.3.1