feat: implement distribution normalization with default disabled

- Add distribution normalization to ensure consistent difficulty across topics
- Support three methods: similarity_range, composite_zscore, percentile_recentering
- Set default to disabled based on analysis showing natural semantic relationships are preferable
- Add comprehensive analysis documentation and test suite

Signed-off-by: Vimal Kumar <vimal78@gmail.com>

crossword-app/backend-py/docs/distribution_normalization_analysis.md

@@ -0,0 +1,176 @@
+# Distribution Normalization Analysis
+
+## Overview
+
+Distribution normalization is a feature implemented to ensure consistent difficulty levels across different topics in the crossword generator. This document analyzes the trade-offs between normalized and non-normalized approaches and provides recommendations.
+
+## The Problem
+
+The original question was: *"Should we normalize the distribution before display? Perhaps the distribution will be centered at the same position for a difficulty level irrespective of topic."*
+
+Different topics naturally have different semantic similarity ranges:
+- **"Animals"**: Rich vocabulary, similarities often range 0.4-0.9
+- **"Philosophy"**: Abstract concepts, similarities might range 0.1-0.6  
+- **"Technology"**: Mixed range, similarities around 0.2-0.8
+
+This led to perceived "inconsistent difficulty" where "Easy Animals" felt easier than "Easy Philosophy" crosswords.
+
+## Current Implementation
+
+### Composite Score Formula
+```
+composite = (1 - difficulty_weight) * similarity + difficulty_weight * freq_score
+```
+
+With default `difficulty_weight = 0.5`:
+```
+composite = 0.5 * similarity + 0.5 * freq_score
+```
+
+### Normalization Methods
+
+1. **`similarity_range` (default)**: Normalizes similarities to [0,1] before composite calculation
+2. **`composite_zscore`**: Z-score normalization (unbounded, typically -3 to +3)
+3. **`percentile_recentering`**: Boosts scores based on proximity to target percentile (can exceed 1.0)
+
+### Configuration
+- `ENABLE_DISTRIBUTION_NORMALIZATION=true` (default)
+- `NORMALIZATION_METHOD=similarity_range` (default)
+
+## Trade-offs Analysis
+
+### Before Normalization (Original System)
+
+#### Advantages ✅
+1. **Natural semantic relationships preserved**
+   - Topics with broader vocabulary naturally had higher similarity ranges
+   - Reflected genuine linguistic density differences
+   - Authentic representation of semantic space
+
+2. **Simpler and more predictable**
+   - Straightforward composite score calculation
+   - Always bounded to [0,1] naturally
+   - No artificial transformations
+
+3. **Semantic honesty**
+   - Some topics ARE inherently harder to generate crosswords for
+   - System reflected this reality rather than masking it
+   - Valuable information for both system and users
+
+4. **Computational efficiency**
+   - No additional normalization calculations
+   - Cleaner code path
+
+#### Disadvantages ❌
+1. **Inconsistent difficulty across topics**
+   - "Easy" for animals genuinely easier than "Easy" for philosophy
+   - Could confuse users expecting uniform difficulty
+
+2. **User expectation mismatch**
+   - Players might expect same difficulty label = same challenge level
+
+### After Normalization (Current System)
+
+#### Advantages ✅
+1. **Consistent difficulty intent**
+   - Attempts to make "Easy" equally easy across all topics
+   - Meets user expectations for uniform difficulty labels
+
+2. **Debug visualization enhancements**
+   - Shows normalization effects in debug tab
+   - Helpful for analysis and understanding
+
+#### Disadvantages ❌
+1. **Artificial stretching of similarity ranges**
+   - Forces sparse topics to use full [0,1] range
+   - Genuinely dissimilar words appear artificially similar
+   - Loss of semantic authenticity
+
+2. **Implementation complexity and bugs**
+   - Different methods produce different ranges
+   - Z-score normalization is unbounded
+   - Percentile recentering can exceed 1.0
+   - Softmax sensitivity to inconsistent ranges
+
+3. **Loss of valuable information**
+   - Masks natural vocabulary density differences
+   - Hides genuine topic difficulty characteristics
+   - Makes debugging harder (what's "real" vs "normalized"?)
+
+4. **Computational overhead**
+   - Additional calculations for normalization
+   - More complex code paths
+   - Potential for numerical issues
+
+## Composite Score Ranges
+
+### Without Normalization
+- **Theoretical range**: [0, 1]
+- **Practical range**: Depends on actual similarities in the 150-word thematic pool
+- **Example**: If similarities range 0.3-0.7, composite ≈ [0.15, 0.85]
+
+### With Normalization
+- **`similarity_range`**: ~[0, 1] (most consistent)
+- **`composite_zscore`**: Unbounded (typically [-3, +3])
+- **`percentile_recentering`**: Can exceed 1.0 due to boosting
+
+## Problems with Current Implementation
+
+1. **Range inconsistency**: Different normalization methods produce different ranges
+2. **Unbounded z-scores**: Affect softmax probability calculations unpredictably  
+3. **Values exceeding [0,1]**: Break assumptions about composite score bounds
+4. **Complexity without clear benefit**: Added complexity for questionable gains
+
+## Recommendation
+
+### **Revert to Non-Normalized Approach** 
+
+The original system was **better** for these reasons:
+
+1. **The "problem" wasn't really a problem**
+   - Different topics having different difficulty distributions is natural and informative
+   - Philosophy IS harder to make crosswords for than animals - this is linguistic reality
+
+2. **Normalization introduces distortions**
+   - Stretching narrow ranges doesn't make words more semantically similar
+   - Creates artificial relationships that don't exist
+
+3. **Alternative solutions are better**:
+   - Show users the natural difficulty of each topic
+   - Adjust word count based on topic vocabulary density  
+   - Provide topic difficulty ratings to set expectations
+   - Use adaptive difficulty within topics rather than across them
+
+### If Normalization is Kept
+
+If normalization must be retained:
+
+1. **Make it opt-in, not default**: `ENABLE_DISTRIBUTION_NORMALIZATION=false`
+2. **Fix range consistency**: Ensure all methods produce [0,1] outputs
+3. **Add proper bounds checking**: Clamp scores to [0,1] after normalization
+4. **Document trade-offs clearly**: Let users make informed choices
+
+## Proposed Implementation Fixes
+
+If keeping normalization, fix these issues:
+
+```python
+# After normalization, ensure consistent [0,1] range
+if method == "composite_zscore":
+    # Map z-scores to [0,1] using sigmoid
+    scores = 1 / (1 + np.exp(-normalized_scores))
+elif method == "percentile_recentering":
+    # Clamp boosted scores to valid range
+    scores = np.clip(boosted_scores, 0, 1)
+
+# Final safety clamp for all methods
+composite_scores = np.clip(composite_scores, 0, 1)
+```
+
+## Conclusion
+
+The **non-normalized approach respects semantic reality** and provides more honest, interpretable results. The "inconsistency" across topics is actually valuable information about linguistic structure, not a bug to be fixed.
+
+**Recommendation**: Disable normalization by default (`ENABLE_DISTRIBUTION_NORMALIZATION=false`) and let the natural semantic relationships guide difficulty distribution. This preserves the system's authenticity while maintaining simplicity and predictability.
+
+The original system's variation across topics was a **feature representing real linguistic diversity**, not a problem requiring artificial correction.

crossword-app/backend-py/src/services/thematic_word_service.py

@@ -288,6 +288,13 @@ class ThematicWordService:
         self.difficulty_weight = float(os.getenv("DIFFICULTY_WEIGHT", "0.5"))
         self.thematic_pool_size = int(os.getenv("THEMATIC_POOL_SIZE", "150"))
         
+        # Distribution normalization configuration
+        # Default: DISABLED based on analysis showing non-normalized approach is better
+        # See docs/distribution_normalization_analysis.md for detailed reasoning
+        # Preserves natural semantic relationships and avoids artificial distortions
+        self.enable_distribution_normalization = os.getenv("ENABLE_DISTRIBUTION_NORMALIZATION", "false").lower() == "true"
+        self.normalization_method = os.getenv("NORMALIZATION_METHOD", "similarity_range").lower()  # "similarity_range", "composite_zscore", "percentile_recentering"
+        
         # Debug tab configuration
         self.enable_debug_tab = os.getenv("ENABLE_DEBUG_TAB", "false").lower() == "true"
         
@@ -359,6 +366,9 @@ class ThematicWordService:
         logger.info(f"🎲 Softmax selection: {'ENABLED' if self.use_softmax_selection else 'DISABLED'}")
         if self.use_softmax_selection:
             logger.info(f"🌡️ Similarity temperature: {self.similarity_temperature}")
+        logger.info(f"🎯 Distribution normalization: {'ENABLED' if self.enable_distribution_normalization else 'DISABLED'}")
+        if self.enable_distribution_normalization:
+            logger.info(f"🔧 Normalization method: {self.normalization_method}")
     
     async def initialize_async(self):
         """Initialize the generator (async version for backend compatibility)."""
@@ -716,6 +726,85 @@ class ThematicWordService:
         composite = final_alpha * similarity + final_beta * freq_score
         return composite
     
+    def _apply_distribution_normalization(self, composite_scores: np.ndarray, candidates: List[Dict[str, Any]], difficulty: str) -> np.ndarray:
+        """
+        Apply distribution normalization to ensure consistent difficulty distributions across topics.
+        
+        This method normalizes the composite score distribution to ensure that the same difficulty level
+        produces consistent selection patterns regardless of the topic's inherent semantic similarity range.
+        
+        Args:
+            composite_scores: Raw composite scores from similarity + frequency alignment
+            candidates: List of candidate word dictionaries  
+            difficulty: Difficulty level for target percentile calculation
+            
+        Returns:
+            Normalized composite scores with consistent distribution shape
+        """
+        if len(composite_scores) <= 1:
+            return composite_scores
+            
+        method = self.normalization_method.lower()
+        
+        if method == "similarity_range":
+            # Method 1: Normalize similarity ranges to [0,1] before composite scoring
+            # This ensures all topics use the full similarity spectrum
+            similarities = np.array([c['similarity'] for c in candidates])
+            if len(similarities) > 1 and np.std(similarities) > 0:
+                min_sim, max_sim = np.min(similarities), np.max(similarities)
+                if max_sim > min_sim:  # Avoid division by zero
+                    # Recalculate composite scores with normalized similarities
+                    normalized_scores = []
+                    for i, candidate in enumerate(candidates):
+                        normalized_sim = (candidate['similarity'] - min_sim) / (max_sim - min_sim)
+                        word = candidate['word']
+                        # Recompute composite score with normalized similarity
+                        percentile = self.word_percentiles.get(word.lower(), 0.0)
+                        
+                        # Calculate difficulty alignment score (same as _compute_composite_score)
+                        if difficulty == "easy":
+                            freq_score = np.exp(-((percentile - 0.9) ** 2) / (2 * 0.1 ** 2))
+                        elif difficulty == "hard":
+                            freq_score = np.exp(-((percentile - 0.2) ** 2) / (2 * 0.15 ** 2))
+                        else:  # medium
+                            freq_score = 0.5 + 0.5 * np.exp(-((percentile - 0.5) ** 2) / (2 * 0.3 ** 2))
+                        
+                        # Apply difficulty weight with normalized similarity
+                        final_alpha = 1.0 - self.difficulty_weight
+                        final_beta = self.difficulty_weight
+                        composite = final_alpha * normalized_sim + final_beta * freq_score
+                        normalized_scores.append(composite)
+                    
+                    return np.array(normalized_scores)
+                    
+        elif method == "composite_zscore":
+            # Method 2: Z-score normalization of composite scores
+            # Centers distribution at 0 with unit variance
+            mean_score = np.mean(composite_scores)
+            std_score = np.std(composite_scores)
+            if std_score > 0:
+                return (composite_scores - mean_score) / std_score
+                
+        elif method == "percentile_recentering":
+            # Method 3: Force distribution center to match target percentile
+            target_percentiles = {"easy": 0.9, "medium": 0.5, "hard": 0.2}
+            target = target_percentiles.get(difficulty, 0.5)
+            
+            # Calculate current probability-weighted percentile center
+            percentiles = np.array([self.word_percentiles.get(c['word'].lower(), 0.0) for c in candidates])
+            
+            # Simple linear transformation to center distribution
+            current_center = np.mean(percentiles)  # Simplified: use mean percentile
+            shift = target - current_center
+            
+            # Apply proportional boost to scores based on how close they are to target
+            percentile_alignment = np.exp(-((percentiles - target) ** 2) / (2 * 0.2 ** 2))
+            boosted_scores = composite_scores * (1 + 0.5 * percentile_alignment)
+            return boosted_scores
+        
+        # If no valid method or normalization not needed, return original scores
+        return composite_scores
+    
     def _softmax_with_temperature(self, scores: np.ndarray, temperature: float = 1.0) -> np.ndarray:
         """
         Apply softmax with temperature control to similarity scores.
@@ -821,6 +910,12 @@ class ThematicWordService:
         
         composite_scores = np.array(composite_scores)
         
+        # Apply distribution normalization if enabled
+        original_composite_scores = composite_scores.copy()  # Keep for debug comparison
+        if self.enable_distribution_normalization:
+            composite_scores = self._apply_distribution_normalization(composite_scores, candidates, difficulty)
+            logger.info(f"🎯 Applied distribution normalization ({self.normalization_method})")
+        
         # Log debug information
         logger.info(f"🔍 Debug: Top 10 composite scores for difficulty={difficulty}:")
         for info in debug_info:
@@ -856,7 +951,7 @@ class ThematicWordService:
         # Create probability distribution data for debug visualization
         prob_distribution = []
         for i, candidate in enumerate(candidates):
-            prob_distribution.append({
+            prob_item = {
                 "word": candidate["word"],
                 "probability": float(probabilities[i]),
                 "composite_score": float(composite_scores[i]),
@@ -865,7 +960,17 @@ class ThematicWordService:
                 "similarity": candidate["similarity"],
                 "tier": candidate.get("tier", "unknown"),
                 "percentile": self.word_percentiles.get(candidate["word"].lower(), 0.0)
-            })
+            }
+            
+            # Add normalization debug data if normalization was applied
+            if self.enable_distribution_normalization and 'original_composite_scores' in locals():
+                prob_item["original_composite_score"] = float(original_composite_scores[i])
+                prob_item["normalization_applied"] = True
+                prob_item["normalization_method"] = self.normalization_method
+            else:
+                prob_item["normalization_applied"] = False
+            
+            prob_distribution.append(prob_item)
         
         # Sort by probability descending for display
         prob_distribution.sort(key=lambda x: x["probability"], reverse=True)
@@ -879,7 +984,9 @@ class ThematicWordService:
             "temperature": temperature,
             "difficulty": difficulty,
             "total_candidates": len(candidates),
-            "selected_count": len(selected_candidates)
+            "selected_count": len(selected_candidates),
+            "normalization_enabled": self.enable_distribution_normalization,
+            "normalization_method": self.normalization_method if self.enable_distribution_normalization else None
         }
         
         return selected_candidates, prob_data

crossword-app/backend-py/test_distribution_normalization.py

@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+"""
+Test script for distribution normalization feature.
+
+This script demonstrates how distribution normalization ensures consistent 
+difficulty levels across different topics by normalizing similarity ranges
+and standardizing distribution shapes.
+"""
+
+import os
+import sys
+import numpy as np
+from collections import defaultdict
+
+# Add src directory to path
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src'))
+
+def test_normalization_across_topics():
+    """Test normalization consistency across different topics."""
+    print("🧪 Testing distribution normalization across topics...")
+    
+    # Set up environment for testing normalization
+    os.environ['SIMILARITY_TEMPERATURE'] = '0.7'
+    os.environ['USE_SOFTMAX_SELECTION'] = 'true'
+    os.environ['DIFFICULTY_WEIGHT'] = '0.3'
+    os.environ['ENABLE_DEBUG_TAB'] = 'true'
+    
+    # Test with normalization ENABLED
+    os.environ['ENABLE_DISTRIBUTION_NORMALIZATION'] = 'true'
+    os.environ['NORMALIZATION_METHOD'] = 'similarity_range'
+    
+    from services.thematic_word_service import ThematicWordService
+    
+    # Create service instance
+    service = ThematicWordService()
+    service.initialize()
+    
+    # Test topics with expected different similarity ranges
+    test_topics = [
+        ("animals", "Expected high similarity range - many animals in vocabulary"),
+        ("technology", "Expected medium similarity range - some tech words"),
+        ("geology", "Expected low similarity range - fewer geology terms"),
+        ("food", "Expected high similarity range - many food words"),
+        ("philosophy", "Expected very low similarity range - abstract concepts")
+    ]
+    
+    difficulty = "medium"  # Use medium difficulty for consistent comparison
+    num_words = 15
+    
+    print(f"\n🎯 Testing normalization for difficulty: {difficulty.upper()}")
+    print(f"📊 Requesting {num_words} words per topic")
+    print(f"🔧 Normalization: {service.enable_distribution_normalization} ({service.normalization_method})")
+    
+    results = {}
+    
+    for topic, description in test_topics:
+        print(f"\n📚 Topic: {topic.upper()}")
+        print(f"   {description}")
+        
+        try:
+            # Generate words using crossword-specific method to get debug data
+            result = service.find_words_for_crossword([topic], difficulty, num_words)
+            words = result["words"]
+            debug_data = result.get("debug", {})
+            
+            if debug_data and "probability_distribution" in debug_data:
+                prob_data = debug_data["probability_distribution"]
+                probabilities = prob_data["probabilities"]
+                
+                # Calculate distribution statistics
+                similarities = [p["similarity"] for p in probabilities]
+                percentiles = [p["percentile"] for p in probabilities]
+                composite_scores = [p["composite_score"] for p in probabilities]
+                probs = [p["probability"] for p in probabilities]
+                
+                # Check for normalization data
+                has_normalization_data = any(p.get("normalization_applied", False) for p in probabilities)
+                original_scores = []
+                if has_normalization_data:
+                    original_scores = [p.get("original_composite_score", p["composite_score"]) for p in probabilities]
+                
+                stats = {
+                    "topic": topic,
+                    "word_count": len(words),
+                    "similarity_range": (min(similarities), max(similarities)),
+                    "similarity_mean": np.mean(similarities),
+                    "similarity_std": np.std(similarities),
+                    "percentile_mean": np.mean(percentiles), 
+                    "percentile_std": np.std(percentiles),
+                    "composite_mean": np.mean(composite_scores),
+                    "composite_std": np.std(composite_scores),
+                    "prob_entropy": -sum(p * np.log(p + 1e-10) for p in probs),  # Selection entropy
+                    "selected_words": [w["word"] for w in words[:5]],  # First 5 words
+                    "normalization_applied": has_normalization_data
+                }
+                
+                if original_scores:
+                    stats["original_composite_mean"] = np.mean(original_scores)
+                    stats["original_composite_std"] = np.std(original_scores)
+                    stats["normalization_effect"] = abs(stats["composite_mean"] - stats["original_composite_mean"])
+                
+                results[topic] = stats
+                
+                # Display key statistics
+                print(f"   ✅ Generated {len(words)} words")
+                print(f"   📊 Similarity range: {stats['similarity_range'][0]:.3f} - {stats['similarity_range'][1]:.3f}")
+                print(f"   📈 Similarity mean±std: {stats['similarity_mean']:.3f}±{stats['similarity_std']:.3f}")
+                print(f"   🎯 Percentile mean±std: {stats['percentile_mean']:.3f}±{stats['percentile_std']:.3f}")
+                print(f"   🔢 Composite mean±std: {stats['composite_mean']:.3f}±{stats['composite_std']:.3f}")
+                if has_normalization_data:
+                    print(f"   🎯 Normalization applied: Original composite mean was {stats['original_composite_mean']:.3f}")
+                    print(f"   📈 Normalization effect: {stats['normalization_effect']:.3f} change in mean")
+                print(f"   📝 Selected words: {', '.join(stats['selected_words'])}")
+                
+            else:
+                print(f"   ❌ No debug data available for {topic}")
+                
+        except Exception as e:
+            print(f"   ❌ Error testing {topic}: {e}")
+            continue
+    
+    # Analyze consistency across topics
+    if len(results) >= 3:
+        print(f"\n📊 NORMALIZATION CONSISTENCY ANALYSIS")
+        print(f"=" * 60)
+        
+        # Compare similarity ranges (should be more consistent after normalization)
+        sim_ranges = [stats['similarity_range'][1] - stats['similarity_range'][0] for stats in results.values()]
+        sim_means = [stats['similarity_mean'] for stats in results.values()]
+        composite_stds = [stats['composite_std'] for stats in results.values()]
+        percentile_means = [stats['percentile_mean'] for stats in results.values()]
+        
+        print(f"🎯 Similarity Range Consistency:")
+        print(f"   Range spread: {np.std(sim_ranges):.4f} (lower = more consistent)")
+        print(f"   Mean variation: {np.std(sim_means):.4f} (lower = more consistent)")
+        
+        print(f"\n🎲 Selection Distribution Consistency:")
+        print(f"   Composite score std variation: {np.std(composite_stds):.4f} (lower = more consistent)")
+        print(f"   Percentile targeting consistency: {np.std(percentile_means):.4f} (should be near 0.5 for medium)")
+        
+        print(f"\n🏆 Normalization Effectiveness:")
+        if any(stats.get('normalization_applied', False) for stats in results.values()):
+            normalization_effects = [stats.get('normalization_effect', 0) for stats in results.values() if stats.get('normalization_effect') is not None]
+            if normalization_effects:
+                avg_effect = np.mean(normalization_effects)
+                print(f"   Average normalization effect: {avg_effect:.4f}")
+                print(f"   Normalization was {'SIGNIFICANT' if avg_effect > 0.05 else 'MINIMAL'}")
+            print("   ✅ Normalization data found in debug output")
+        else:
+            print("   ⚠️ No normalization data found - check ENABLE_DISTRIBUTION_NORMALIZATION")
+        
+        # Ideal targets for medium difficulty
+        target_percentile = 0.5
+        percentile_deviation = np.mean([abs(pm - target_percentile) for pm in percentile_means])
+        print(f"\n🎯 Difficulty Targeting Accuracy:")
+        print(f"   Target percentile (medium): {target_percentile}")
+        print(f"   Average deviation: {percentile_deviation:.4f}")
+        print(f"   Targeting accuracy: {'EXCELLENT' if percentile_deviation < 0.05 else 'GOOD' if percentile_deviation < 0.1 else 'NEEDS IMPROVEMENT'}")
+    
+    print(f"\n✅ Distribution normalization test completed!")
+    return results
+
+def test_normalization_methods():
+    """Test different normalization methods."""
+    print(f"\n🧪 Testing different normalization methods...")
+    
+    methods = ["similarity_range", "composite_zscore", "percentile_recentering"]
+    topic = "animals"  # Use consistent topic
+    difficulty = "easy"  # Use easy difficulty to see clear effects
+    
+    for method in methods:
+        print(f"\n🔧 Testing method: {method.upper()}")
+        
+        os.environ['NORMALIZATION_METHOD'] = method
+        
+        from services.thematic_word_service import ThematicWordService
+        
+        service = ThematicWordService()
+        service.initialize()
+        
+        try:
+            result = service.find_words_for_crossword([topic], difficulty, 10)
+            words = result["words"]
+            debug_data = result.get("debug", {})
+            
+            if debug_data and "probability_distribution" in debug_data:
+                prob_data = debug_data["probability_distribution"]
+                probabilities = prob_data["probabilities"]
+                
+                similarities = [p["similarity"] for p in probabilities]
+                percentiles = [p["percentile"] for p in probabilities]
+                
+                print(f"   📊 Similarity range: {min(similarities):.3f} - {max(similarities):.3f}")
+                print(f"   🎯 Mean percentile: {np.mean(percentiles):.3f} (target for easy: 0.9)")
+                print(f"   📈 Selected words: {', '.join([w['word'] for w in words[:5]])}")
+                
+                if any(p.get("normalization_applied", False) for p in probabilities):
+                    print(f"   ✅ Normalization applied successfully")
+                else:
+                    print(f"   ⚠️ Normalization not detected in debug data")
+            else:
+                print(f"   ❌ No debug data available")
+                
+        except Exception as e:
+            print(f"   ❌ Error with method {method}: {e}")
+
+if __name__ == "__main__":
+    print("🎯 Distribution Normalization Test Suite")
+    print("=" * 50)
+    
+    test_normalization_across_topics()
+    test_normalization_methods()
+    
+    print(f"\n🎉 All tests completed!")
+    print(f"\n💡 To see normalization effects in the UI:")
+    print(f"   1. Set ENABLE_DISTRIBUTION_NORMALIZATION=true")
+    print(f"   2. Set ENABLE_DEBUG_TAB=true") 
+    print(f"   3. Generate crosswords with different topics at the same difficulty")
+    print(f"   4. Check the Debug tab for normalization indicators and tooltips")

crossword-app/frontend/src/components/DebugTab.jsx

@@ -313,13 +313,21 @@ const DebugTab = ({ debugData }) => {
             },
             label: function(context) {
               const item = sortedByPercentile[context.dataIndex];
-              return [
+              const labels = [
                 `Probability: ${(item.probability * 100).toFixed(2)}%`,
                 `Composite Score: ${item.composite_score.toFixed(3)}`,
                 `Similarity: ${item.similarity.toFixed(3)}`,
                 `Percentile: ${(item.percentile * 100).toFixed(1)}%`,
                 `Tier: ${item.tier.replace('tier_', '').replace('_', ' ')}`
               ];
+              
+              // Add normalization data if available
+              if (item.normalization_applied && item.original_composite_score !== undefined) {
+                labels.splice(2, 0, `Original Score: ${item.original_composite_score.toFixed(3)}`);
+                labels.splice(3, 0, `🎯 Normalized: ${item.normalization_method}`);
+              }
+              
+              return labels;
             }
           },
           backgroundColor: 'rgba(0, 0, 0, 0.8)',
@@ -394,13 +402,21 @@ const DebugTab = ({ debugData }) => {
             },
             label: function(context) {
               const item = sortedByPercentile[context.dataIndex];
-              return [
+              const labels = [
                 `Probability: ${(item.probability * 100).toFixed(2)}%`,
                 `Composite Score: ${item.composite_score.toFixed(3)}`,
                 `Similarity: ${item.similarity.toFixed(3)}`,
                 `Percentile: ${(item.percentile * 100).toFixed(1)}%`,
                 `Tier: ${item.tier.replace('tier_', '').replace('_', ' ')}`
               ];
+              
+              // Add normalization data if available
+              if (item.normalization_applied && item.original_composite_score !== undefined) {
+                labels.splice(2, 0, `Original Score: ${item.original_composite_score.toFixed(3)}`);
+                labels.splice(3, 0, `🎯 Normalized: ${item.normalization_method}`);
+              }
+              
+              return labels;
             }
           },
           backgroundColor: 'rgba(0, 0, 0, 0.8)',
@@ -500,6 +516,11 @@ const DebugTab = ({ debugData }) => {
           <div><strong>Top Probability:</strong> {(Math.max(...sortedByPercentile.map(p => p.probability)) * 100).toFixed(1)}%</div>
           <div><strong>Average:</strong> {((1/probData.total_candidates) * 100).toFixed(1)}%</div>
           <div><strong>Temperature Effect:</strong> {probData.temperature < 1 ? 'More deterministic' : probData.temperature > 1 ? 'More random' : 'Balanced'}</div>
+          {probData.normalization_enabled && (
+            <div style={{backgroundColor: '#e8f5e8', padding: '4px', borderRadius: '4px'}}>
+              <strong>🎯 Distribution Normalization:</strong> ENABLED ({probData.normalization_method})
+            </div>
+          )}
           <div><strong>Mean Position:</strong> Word #{meanWordIndex + 1} ({sortedByPercentile[meanWordIndex]?.word})</div>
           <div><strong>Distribution Width (σ):</strong> {sigma.toFixed(1)} words</div>
           <div><strong>σ Sampling Zone:</strong> {(sigmaRangeProbMass * 100).toFixed(1)}% of probability mass</div>
@@ -516,6 +537,9 @@ const DebugTab = ({ debugData }) => {
             frequency percentile (100% → 0%, common → rare). This reveals whether the Gaussian frequency targeting 
             is working correctly for your selected difficulty level. Look for probability peaks at the intended percentile ranges:
             <strong> Easy (90%+), Medium (50%), Hard (20%)</strong>.
+            {probData.normalization_enabled && (
+              <> <strong>🎯 Distribution normalization is ENABLED</strong> to ensure consistent difficulty across topics.</>
+            )}
           </p>
         </div>