feat: Add percentile-sorted probability visualization to debug tab

- Implement Chart.js probability distribution visualization
- Sort chart by frequency percentile (100% → 0%) to reveal Gaussian targeting
- Add comprehensive probability distribution analysis documentation
- Enable statistical markers (μ, σ) with proper sampling zone visualization

Fixes visualization issue where probability-sorted charts couldn't show
difficulty-based frequency targeting effectiveness.

Signed-off-by: Vimal Kumar <vimal78@gmail.com>

crossword-app/backend-py/docs/probability_distribution_analysis.md

@@ -0,0 +1,297 @@
+# Probability Distribution Analysis: Theory vs. Practice
+
+## Executive Summary
+
+This document analyzes the **actual behavior** of the crossword word selection system, complementing the theoretical framework described in [`composite_scoring_algorithm.md`](composite_scoring_algorithm.md). While the composite scoring theory is sound, empirical analysis reveals significant discrepancies between intended and actual behavior.
+
+### Key Findings
+- **Similarity dominates**: Difficulty-based frequency preferences are too weak to create distinct selection patterns
+- **Exponential distributions**: Actual probability distributions follow exponential decay, not normal distributions
+- **Statistical misconceptions**: Using normal distribution concepts (μ ± σ) on exponentially decaying data is misleading
+- **Mode-mean divergence**: Statistical measures don't represent where selections actually occur
+
+## Observed Probability Distributions
+
+### Data Source: Technology Topic Analysis
+Using the debug visualization with `ENABLE_DEBUG_TAB=true`, we analyzed the actual probability distributions for different difficulties:
+
+```
+Topic: Technology
+Candidates: 150 words
+Temperature: 0.2
+Selection method: Softmax with composite scoring
+```
+
+### Empirical Results
+
+#### Easy Difficulty
+```
+Mean Position: Word #42 (IMPLEMENT)
+Distribution Width (σ): 33.4 words
+σ Sampling Zone: 70.5% of probability mass
+σ Range: Words #9-#76
+Top Probability: 2.3%
+```
+
+#### Medium Difficulty  
+```
+Mean Position: Word #60 (COMPUTERIZED)
+Distribution Width (σ): 42.9 words
+σ Sampling Zone: 61.0% of probability mass  
+σ Range: Words #17-#103
+Top Probability: 1.5%
+```
+
+#### Hard Difficulty
+```
+Mean Position: Word #37 (DIGITISATION)
+Distribution Width (σ): 40.2 words
+σ Sampling Zone: 82.1% of probability mass
+σ Range: Words #1-#77  
+Top Probability: 4.1%
+```
+
+### Critical Observation
+**All three difficulty levels show similar exponential decay patterns**, with only minor variations in peak height and mean position. This indicates the frequency-based difficulty targeting is not working as intended.
+
+## Statistical Misconceptions in Current Approach
+
+### The Mode-Mean Divergence Problem
+
+The visualization shows a red line (μ) at positions 37-60, but the highest probability bars are at positions 0-5. This reveals a fundamental statistical concept:
+
+```
+Distribution Type: Exponentially Decaying (Highly Skewed)
+
+Mode (Peak):     Position 0-3     (2-4% probability)
+Median:          Position ~15     (Where 50% of probability mass is reached)  
+Mean (μ):        Position 37-60   (Weighted average position)
+```
+
+### Why μ is "Wrong" for Understanding Selection
+
+In an exponential distribution with long tail:
+
+1. **Mode (0-3)**: Where individual words have highest probability
+2. **Practical sampling zone**: First 10-20 words contain ~60-80% of probability mass
+3. **Mean (37-60)**: Pulled far right by 100+ words with tiny probabilities
+
+The mean doesn't represent where sampling actually occurs—it's mathematically correct but practically misleading.
+
+### Standard Deviation Misapplication
+
+The σ visualization assumes a normal distribution where:
+- **Normal assumption**: μ ± σ contains ~68% of probability mass
+- **Our reality**: Exponential distribution with μ ± σ often missing the high-probability words entirely
+
+For exponential distributions, percentiles or cumulative probability are more meaningful than standard deviation.
+
+## Actual vs. Expected Behavior Analysis
+
+### What Should Happen (Theory)
+According to the composite scoring algorithm:
+
+- **Easy**: Gaussian peak at 90th percentile → common words dominate
+- **Medium**: Gaussian peak at 50th percentile → balanced selection  
+- **Hard**: Gaussian peak at 20th percentile → rare words favored
+
+### What Actually Happens (Empirical)
+```
+Easy:   MULTIMEDIA, TECH, TECHNOLOGY, IMPLEMENTING... (similar to others)
+Medium: TECH, TECHNOLOGY, COMPUTERIZED, TECHNOLOGICAL... (similar pattern)
+Hard:   TECH, TECHNOLOGY, DIGITISATION, TECHNICIAN... (still similar)
+```
+
+**All difficulties select similar high-similarity technology words**, regardless of their frequency percentiles.
+
+### Root Cause Analysis
+
+The problem isn't in the Gaussian curves—they work correctly. The issue is in the composite formula:
+
+```python
+# Current approach
+composite = 0.5 * similarity + 0.5 * frequency_score
+
+# What happens with real data:
+# High-similarity word: similarity=0.9, wrong_freq_score=0.1
+# → composite = 0.5*0.9 + 0.5*0.1 = 0.50
+
+# Medium-similarity word: similarity=0.7, perfect_freq_score=1.0  
+# → composite = 0.5*0.7 + 0.5*1.0 = 0.85
+```
+
+Even with perfect frequency alignment, a word needs **very high similarity** to compete with high-similarity words that have wrong frequency profiles.
+
+## Sampling Mechanics Deep Dive
+
+### np.random.choice Behavior
+The selection uses `np.random.choice` with:
+- **Without replacement**: Each word can only be selected once
+- **Probability weighting**: Based on computed probabilities  
+- **Sample size**: 10 words from 150 candidates
+
+### Where Selections Actually Occur
+Despite μ being at position 37-60, most actual selections come from positions 0-30 because:
+
+1. **High probabilities concentrate early**: First 20 words often have 60%+ of total probability
+2. **Without replacement effect**: Once high-probability words are chosen, selection moves to next-highest
+3. **Exponential decay**: Probability drops rapidly, making later positions unlikely
+
+This explains why the green bars (selected words) appear mostly in the left portion of all distributions, regardless of where μ is located.
+
+## Better Visualization Approaches
+
+### Current Problems
+- **μ ± σ assumes normality**: Not applicable to exponential distributions
+- **Mean position misleading**: Doesn't show where selection actually occurs
+- **Standard deviation meaningless**: For highly skewed distributions
+
+### Recommended Alternatives
+
+#### 1. Cumulative Probability Visualization
+```
+First 10 words: 45% of total probability mass
+First 20 words: 65% of total probability mass  
+First 30 words: 78% of total probability mass
+First 50 words: 90% of total probability mass
+```
+
+#### 2. Percentile Markers Instead of μ ± σ
+```
+P50 (Median):  Position where 50% of probability mass is reached
+P75:           Position where 75% of probability mass is reached  
+P90:           Position where 90% of probability mass is reached
+```
+
+#### 3. Mode Annotation
+- Show the actual peak (mode) position
+- Mark the top-5 highest probability words
+- Distinguish between statistical mean and practical selection zone
+
+#### 4. Selection Concentration Metric
+```
+Effective Selection Range: Positions covering 80% of selection probability
+Selection Concentration: Gini coefficient of probability distribution
+```
+
+## Difficulty Differentiation Failure
+
+### Expected Pattern
+Different difficulty levels should show visually distinct probability distribution patterns:
+- **Easy**: Steep peak at common words, rapid falloff  
+- **Medium**: Moderate peak, balanced distribution
+- **Hard**: Peak shifted toward rare words
+
+### Observed Pattern  
+All difficulties show similar exponential decay curves with:
+- Similar-shaped distributions
+- Similar high-probability words (TECH, TECHNOLOGY, etc.)
+- Only minor differences in peak height and position
+
+### Quantitative Evidence
+```
+Similarity scores of top words (all difficulties):
+TECHNOLOGY:     0.95+ similarity to "technology" 
+TECH:           0.90+ similarity to "technology"
+MULTIMEDIA:     0.85+ similarity to "technology"
+
+These high semantic matches dominate regardless of their frequency percentiles.
+```
+
+## Recommended Fixes
+
+### 1. Multiplicative Scoring (Immediate Fix)
+Replace additive formula with multiplicative gates:
+
+```python
+# Current (additive)
+composite = 0.5 * similarity + 0.5 * frequency_score
+
+# Proposed (multiplicative)  
+frequency_modifier = get_frequency_modifier(percentile, difficulty)
+composite = similarity * frequency_modifier
+
+# Where frequency_modifier ranges 0.1-1.2 instead of 0.0-1.0
+```
+
+**Effect**: Frequency acts as a gate rather than just another score component.
+
+### 2. Two-Stage Filtering (Structural Fix)
+```python
+# Stage 1: Filter by frequency percentile ranges
+easy_candidates = [w for w in candidates if w.percentile > 0.7]      # Common words
+medium_candidates = [w for w in candidates if 0.3 < w.percentile < 0.7]  # Medium words  
+hard_candidates = [w for w in candidates if w.percentile < 0.3]      # Rare words
+
+# Stage 2: Rank filtered candidates by similarity
+selected = softmax_selection(filtered_candidates, similarity_only=True)
+```
+
+**Effect**: Guarantees different frequency pools for each difficulty, then optimizes within each pool.
+
+### 3. Exponential Temperature Scaling (Parameter Fix)
+Use different temperature values by difficulty to create more distinct distributions:
+
+```python
+easy_temperature = 0.1    # Very deterministic (sharp peak)
+medium_temperature = 0.3  # Moderate randomness
+hard_temperature = 0.2    # Deterministic but different peak
+```
+
+### 4. Adaptive Frequency Weights (Dynamic Fix)
+```python
+# Calculate frequency dominance needed to overcome similarity differences
+max_similarity_diff = max_similarity - min_similarity  # e.g., 0.95 - 0.6 = 0.35
+required_freq_weight = max_similarity_diff / (1 - max_similarity_diff)  # e.g., 0.35/0.65 ≈ 0.54
+
+# Use higher frequency weight when similarity ranges are wide
+adaptive_weight = min(0.8, required_freq_weight)
+```
+
+## Empirical Data Summary
+
+### Word Selection Patterns (Technology Topic)
+```
+Easy Mode Top Selections:
+- MULTIMEDIA (percentile: ?, similarity: high)
+- IMPLEMENT (percentile: ?, similarity: high) 
+- TECHNOLOGICAL (percentile: ?, similarity: high)
+
+Hard Mode Top Selections:  
+- TECH (percentile: ?, similarity: very high)
+- DIGITISATION (percentile: likely low, similarity: high)
+- TECHNICIAN (percentile: ?, similarity: high)
+```
+
+### Statistical Summary
+- **σ Width Variation**: Easy (33.4) vs Medium (42.9) vs Hard (40.2) - only 28% difference
+- **Peak Variation**: 1.5% to 4.1% - moderate difference
+- **Mean Position Variation**: Position 37 to 60 - 62% range but all in middle zone
+- **Selection Concentration**: Most selections from first 30 words in all difficulties
+
+## Conclusions
+
+### The Core Problem
+The difficulty-aware word selection system is theoretically sound but practically ineffective because:
+
+1. **Semantic similarity signals are too strong** compared to frequency signals
+2. **Additive scoring allows high-similarity words to dominate** regardless of frequency appropriateness  
+3. **Statistical visualization assumes normal distributions** but data is exponentially skewed
+
+### Success Metrics for Fixes
+A working system should show:
+
+1. **Visually distinct probability distributions** for each difficulty
+2. **Different word frequency profiles** in actual selections
+3. **Mode and mean alignment** with intended difficulty targets
+4. **Meaningful σ ranges** that represent actual selection zones
+
+### Next Steps
+1. Implement multiplicative scoring or two-stage filtering
+2. Update visualization to use percentiles instead of μ ± σ
+3. Collect empirical data on word frequency percentiles in actual selections
+4. Validate fixes show distinct patterns across difficulties
+
+---
+
+*This analysis represents empirical findings from the debug visualization system, revealing gaps between the theoretical composite scoring model and its practical implementation.*

crossword-app/backend-py/src/services/thematic_word_service.py

@@ -744,7 +744,7 @@ class ThematicWordService:
         return probabilities
     
     def _softmax_weighted_selection(self, candidates: List[Dict[str, Any]], 
-                                  num_words: int, temperature: float = None, difficulty: str = "medium") -> List[Dict[str, Any]]:
+                                  num_words: int, temperature: float = None, difficulty: str = "medium") -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
         """
         Select words using softmax-based probabilistic sampling weighted by composite scores.
         
@@ -784,10 +784,17 @@ class ThematicWordService:
             difficulty: Difficulty level ("easy", "medium", "hard") for frequency weighting
         
         Returns:
-            Selected word dictionaries, sampled without replacement according to composite probabilities
+            Tuple of (selected_word_dictionaries, probability_distribution_data)
+            - selected_word_dictionaries: Words chosen for crossword
+            - probability_distribution_data: Dict with candidate probabilities for debug visualization
         """
         if len(candidates) <= num_words:
-            return candidates
+            # Return all candidates with trivial probability distribution
+            prob_data = {
+                "probabilities": [{"word": c["word"], "probability": 1.0/len(candidates), "composite_score": 0.0, "selected": True, "rank": i+1} 
+                                for i, c in enumerate(candidates)]
+            }
+            return candidates, prob_data
             
         if temperature is None:
             temperature = self.similarity_temperature
@@ -832,6 +839,7 @@ class ThematicWordService:
         
         # Return selected candidates
         selected_candidates = [candidates[i] for i in selected_indices]
+        selected_word_set = {candidates[i]["word"] for i in selected_indices}
         
         logger.info(f"🎲 Composite softmax selection (T={temperature:.2f}, difficulty={difficulty}): {len(selected_candidates)} from {len(candidates)} candidates")
         
@@ -845,7 +853,36 @@ class ThematicWordService:
             tier = word_data.get('tier', 'unknown')
             logger.info(f"   {word:<15} sim:{similarity:.3f} perc:{percentile:.3f} comp:{composite:.3f} ({tier})")
         
-        return selected_candidates
+        # Create probability distribution data for debug visualization
+        prob_distribution = []
+        for i, candidate in enumerate(candidates):
+            prob_distribution.append({
+                "word": candidate["word"],
+                "probability": float(probabilities[i]),
+                "composite_score": float(composite_scores[i]),
+                "selected": candidate["word"] in selected_word_set,
+                "rank": i + 1,
+                "similarity": candidate["similarity"],
+                "tier": candidate.get("tier", "unknown"),
+                "percentile": self.word_percentiles.get(candidate["word"].lower(), 0.0)
+            })
+        
+        # Sort by probability descending for display
+        prob_distribution.sort(key=lambda x: x["probability"], reverse=True)
+        
+        # Update ranks based on probability order
+        for i, item in enumerate(prob_distribution):
+            item["probability_rank"] = i + 1
+        
+        prob_data = {
+            "probabilities": prob_distribution,
+            "temperature": temperature,
+            "difficulty": difficulty,
+            "total_candidates": len(candidates),
+            "selected_count": len(selected_candidates)
+        }
+        
+        return selected_candidates, prob_data
     
     def _detect_multiple_themes(self, inputs: List[str], max_themes: int = 3) -> List[np.ndarray]:
         """Detect multiple themes using clustering."""
@@ -1167,12 +1204,13 @@ class ThematicWordService:
         final_words = []
         
         # Select words using either softmax weighted selection or traditional random selection
+        probability_data = None
         if self.use_softmax_selection:
             logger.info(f"🎲 Using softmax weighted selection on all {len(candidate_words)} candidates (temperature: {self.similarity_temperature})")
             
             # Apply softmax selection to ALL candidate words regardless of clue quality
             if len(candidate_words) > requested_words:
-                selected_words = self._softmax_weighted_selection(candidate_words, requested_words, difficulty=difficulty)
+                selected_words, probability_data = self._softmax_weighted_selection(candidate_words, requested_words, difficulty=difficulty)
                 final_words.extend(selected_words)
             else:
                 final_words.extend(candidate_words)  # Take all words if not enough
@@ -1243,6 +1281,11 @@ class ThematicWordService:
                     for word_data in final_words
                 ]
             }
+            
+            # Add probability distribution data if available
+            if probability_data:
+                debug_data["probability_distribution"] = probability_data
+            
             result["debug"] = debug_data
             logger.info(f"🐛 Debug data collected: {len(debug_data['thematic_pool'])} thematic words, {len(debug_data['candidate_words'])} candidates, {len(debug_data['selected_words'])} selected")
         

crossword-app/frontend/package-lock.json

@@ -8,7 +8,10 @@
       "name": "crossword-frontend",
       "version": "1.0.0",
       "dependencies": {
+        "chart.js": "^4.5.0",
+        "chartjs-plugin-annotation": "^3.1.0",
         "react": "^18.2.0",
+        "react-chartjs-2": "^5.3.0",
         "react-dom": "^18.2.0"
       },
       "devDependencies": {
@@ -850,6 +853,12 @@
         "@jridgewell/sourcemap-codec": "^1.4.14"
       }
     },
+    "node_modules/@kurkle/color": {
+      "version": "0.3.4",
+      "resolved": "https://registry.npmjs.org/@kurkle/color/-/color-0.3.4.tgz",
+      "integrity": "sha512-M5UknZPHRu3DEDWoipU6sE8PdkZ6Z/S+v4dD+Ke8IaNlpdSQah50lz1KtcFBa2vsdOnwbbnxJwVM4wty6udA5w==",
+      "license": "MIT"
+    },
     "node_modules/@nodelib/fs.scandir": {
       "version": "2.1.5",
       "resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz",
@@ -1669,6 +1678,27 @@
         "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
+    "node_modules/chart.js": {
+      "version": "4.5.0",
+      "resolved": "https://registry.npmjs.org/chart.js/-/chart.js-4.5.0.tgz",
+      "integrity": "sha512-aYeC/jDgSEx8SHWZvANYMioYMZ2KX02W6f6uVfyteuCGcadDLcYVHdfdygsTQkQ4TKn5lghoojAsPj5pu0SnvQ==",
+      "license": "MIT",
+      "dependencies": {
+        "@kurkle/color": "^0.3.0"
+      },
+      "engines": {
+        "pnpm": ">=8"
+      }
+    },
+    "node_modules/chartjs-plugin-annotation": {
+      "version": "3.1.0",
+      "resolved": "https://registry.npmjs.org/chartjs-plugin-annotation/-/chartjs-plugin-annotation-3.1.0.tgz",
+      "integrity": "sha512-EkAed6/ycXD/7n0ShrlT1T2Hm3acnbFhgkIEJLa0X+M6S16x0zwj1Fv4suv/2bwayCT3jGPdAtI9uLcAMToaQQ==",
+      "license": "MIT",
+      "peerDependencies": {
+        "chart.js": ">=4.0.0"
+      }
+    },
     "node_modules/color-convert": {
       "version": "2.0.1",
       "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
@@ -3816,6 +3846,16 @@
         "node": ">=0.10.0"
       }
     },
+    "node_modules/react-chartjs-2": {
+      "version": "5.3.0",
+      "resolved": "https://registry.npmjs.org/react-chartjs-2/-/react-chartjs-2-5.3.0.tgz",
+      "integrity": "sha512-UfZZFnDsERI3c3CZGxzvNJd02SHjaSJ8kgW1djn65H1KK8rehwTjyrRKOG3VTMG8wtHZ5rgAO5oTHtHi9GCCmw==",
+      "license": "MIT",
+      "peerDependencies": {
+        "chart.js": "^4.1.1",
+        "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
+      }
+    },
     "node_modules/react-dom": {
       "version": "18.3.1",
       "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-18.3.1.tgz",

crossword-app/frontend/package.json

@@ -13,7 +13,10 @@
     "format": "prettier --write \"src/**/*.{js,jsx,css,md}\""
   },
   "dependencies": {
+    "chart.js": "^4.5.0",
+    "chartjs-plugin-annotation": "^3.1.0",
     "react": "^18.2.0",
+    "react-chartjs-2": "^5.3.0",
     "react-dom": "^18.2.0"
   },
   "devDependencies": {
@@ -39,4 +42,4 @@
       "last 1 safari version"
     ]
   }
-}
+}

crossword-app/frontend/src/components/DebugTab.jsx

@@ -1,4 +1,26 @@
 import React, { useState } from 'react';
+import {
+  Chart as ChartJS,
+  CategoryScale,
+  LinearScale,
+  BarElement,
+  Title,
+  Tooltip,
+  Legend,
+} from 'chart.js';
+import annotationPlugin from 'chartjs-plugin-annotation';
+import { Bar } from 'react-chartjs-2';
+
+// Register Chart.js components
+ChartJS.register(
+  CategoryScale,
+  LinearScale,
+  BarElement,
+  Title,
+  Tooltip,
+  Legend,
+  annotationPlugin
+);
 
 const DebugTab = ({ debugData }) => {
   const [activeSection, setActiveSection] = useState('overview');
@@ -18,6 +40,7 @@ const DebugTab = ({ debugData }) => {
     { id: 'thematic-pool', label: 'Thematic Pool' },
     { id: 'candidates', label: 'Candidates' },
     { id: 'selection', label: 'Selection' },
+    { id: 'probabilities', label: 'Probabilities' },
     { id: 'selected', label: 'Selected Words' }
   ];
 
@@ -218,6 +241,339 @@ const DebugTab = ({ debugData }) => {
     </div>
   );
 
+  const renderProbabilities = () => {
+    const probData = debugData.probability_distribution;
+    
+    if (!probData || !probData.probabilities) {
+      return (
+        <div className="debug-section">
+          <h3>Probability Distribution</h3>
+          <p>Probability data not available (only shown with softmax selection).</p>
+        </div>
+      );
+    }
+
+    try {
+      const probabilities = probData.probabilities;
+      
+      // Sort by percentile (descending) to show 100% -> 0% left to right
+      const sortedByPercentile = [...probabilities].sort((a, b) => b.percentile - a.percentile);
+      
+      // Calculate distribution statistics based on position in sorted array
+      const mean = sortedByPercentile.reduce((sum, p, i) => sum + (p.probability || 0) * i, 0);
+      const variance = sortedByPercentile.reduce((sum, p, i) => sum + (p.probability || 0) * Math.pow(i - mean, 2), 0);
+      const sigma = Math.sqrt(Math.max(0, variance)); // Ensure no negative variance
+      const meanWordIndex = Math.max(0, Math.min(sortedByPercentile.length - 1, Math.round(mean)));
+      const sigmaRangeStart = Math.max(0, Math.round(mean - sigma));
+      const sigmaRangeEnd = Math.min(sortedByPercentile.length - 1, Math.round(mean + sigma));
+      
+      // Calculate sampling statistics with bounds checking
+      const sigmaRangeProbMass = sortedByPercentile
+        .slice(sigmaRangeStart, sigmaRangeEnd + 1)
+        .reduce((sum, p) => sum + (p.probability || 0), 0);
+    
+    // Prepare chart data - sorted by percentile to reveal Gaussian targeting
+    const chartData = {
+      labels: sortedByPercentile.map(p => `${p.word}\n(${(p.percentile * 100).toFixed(0)}%)`),
+      datasets: [
+        {
+          label: 'Selection Probability (%)',
+          data: sortedByPercentile.map(p => p.probability * 100),
+          backgroundColor: sortedByPercentile.map(p => 
+            p.selected ? 'rgba(76, 175, 80, 0.8)' : 'rgba(158, 158, 158, 0.6)'
+          ),
+          borderColor: sortedByPercentile.map(p => 
+            p.selected ? 'rgba(76, 175, 80, 1)' : 'rgba(158, 158, 158, 0.8)'
+          ),
+          borderWidth: 2
+        }
+      ]
+    };
+
+    const chartOptions = {
+      responsive: true,
+      maintainAspectRatio: false,
+      plugins: {
+        legend: {
+          display: false
+        },
+        title: {
+          display: true,
+          text: `Probability Distribution by Frequency Percentile (Temperature: ${probData.temperature})`,
+          font: {
+            size: 16,
+            weight: 'bold'
+          }
+        },
+        tooltip: {
+          callbacks: {
+            title: function(context) {
+              const item = sortedByPercentile[context[0].dataIndex];
+              return `${item.word} ${item.selected ? '✓ SELECTED' : ''}`;
+            },
+            label: function(context) {
+              const item = sortedByPercentile[context.dataIndex];
+              return [
+                `Probability: ${(item.probability * 100).toFixed(2)}%`,
+                `Composite Score: ${item.composite_score.toFixed(3)}`,
+                `Similarity: ${item.similarity.toFixed(3)}`,
+                `Percentile: ${(item.percentile * 100).toFixed(1)}%`,
+                `Tier: ${item.tier.replace('tier_', '').replace('_', ' ')}`
+              ];
+            }
+          },
+          backgroundColor: 'rgba(0, 0, 0, 0.8)',
+          titleColor: 'white',
+          bodyColor: 'white',
+          borderColor: 'rgba(255, 255, 255, 0.3)',
+          borderWidth: 1
+        }
+      },
+      scales: {
+        x: {
+          title: {
+            display: true,
+            text: 'Words (sorted by frequency percentile: 100% → 0%)',
+            font: {
+              size: 14,
+              weight: 'bold'
+            }
+          },
+          ticks: {
+            maxRotation: 45,
+            minRotation: 45,
+            font: {
+              size: 11,
+              weight: 'bold'
+            }
+          }
+        },
+        y: {
+          title: {
+            display: true,
+            text: 'Selection Probability (%)',
+            font: {
+              size: 14,
+              weight: 'bold'
+            }
+          },
+          beginAtZero: true,
+          ticks: {
+            callback: function(value) {
+              return value.toFixed(1) + '%';
+            }
+          }
+        }
+      },
+      interaction: {
+        intersect: false,
+        mode: 'index'
+      }
+    };
+    
+    // Configure all plugins including annotation
+    const chartOptionsWithAnnotations = {
+      ...chartOptions,
+      plugins: {
+        legend: {
+          display: false
+        },
+        title: {
+          display: true,
+          text: `Probability Distribution by Frequency Percentile (Temperature: ${probData.temperature})`,
+          font: {
+            size: 16,
+            weight: 'bold'
+          }
+        },
+        tooltip: {
+          callbacks: {
+            title: function(context) {
+              const item = sortedByPercentile[context[0].dataIndex];
+              return `${item.word} ${item.selected ? '✓ SELECTED' : ''}`;
+            },
+            label: function(context) {
+              const item = sortedByPercentile[context.dataIndex];
+              return [
+                `Probability: ${(item.probability * 100).toFixed(2)}%`,
+                `Composite Score: ${item.composite_score.toFixed(3)}`,
+                `Similarity: ${item.similarity.toFixed(3)}`,
+                `Percentile: ${(item.percentile * 100).toFixed(1)}%`,
+                `Tier: ${item.tier.replace('tier_', '').replace('_', ' ')}`
+              ];
+            }
+          },
+          backgroundColor: 'rgba(0, 0, 0, 0.8)',
+          titleColor: 'white',
+          bodyColor: 'white',
+          borderColor: 'rgba(255, 255, 255, 0.3)',
+          borderWidth: 1
+        },
+        annotation: {
+          annotations: {
+            meanLine: {
+              type: 'line',
+              xMin: meanWordIndex,
+              xMax: meanWordIndex,
+              borderColor: 'rgba(255, 99, 132, 0.8)',
+              borderWidth: 3,
+              borderDash: [5, 5],
+              label: {
+                display: true,
+                content: 'μ',
+                position: 'start',
+                backgroundColor: 'rgba(255, 99, 132, 0.8)',
+                color: 'white',
+                font: {
+                  weight: 'bold',
+                  size: 12
+                }
+              }
+            },
+            sigmaBox: {
+              type: 'box',
+              xMin: sigmaRangeStart,
+              xMax: sigmaRangeEnd,
+              backgroundColor: 'rgba(54, 162, 235, 0.15)',
+              borderColor: 'rgba(54, 162, 235, 0.5)',
+              borderWidth: 2,
+              label: {
+                display: true,
+                content: `σ (${(sigmaRangeProbMass * 100).toFixed(1)}%)`,
+                position: 'center',
+                backgroundColor: 'rgba(54, 162, 235, 0.8)',
+                color: 'white',
+                font: {
+                  weight: 'bold',
+                  size: 11
+                }
+              }
+            },
+            sigmaStartLine: {
+              type: 'line',
+              xMin: sigmaRangeStart,
+              xMax: sigmaRangeStart,
+              borderColor: 'rgba(54, 162, 235, 0.8)',
+              borderWidth: 2,
+              borderDash: [3, 3],
+              label: {
+                display: true,
+                content: 'μ-σ',
+                position: 'start',
+                backgroundColor: 'rgba(54, 162, 235, 0.6)',
+                color: 'white',
+                font: {
+                  size: 10
+                }
+              }
+            },
+            sigmaEndLine: {
+              type: 'line',
+              xMin: sigmaRangeEnd,
+              xMax: sigmaRangeEnd,
+              borderColor: 'rgba(54, 162, 235, 0.8)',
+              borderWidth: 2,
+              borderDash: [3, 3],
+              label: {
+                display: true,
+                content: 'μ+σ',
+                position: 'start',
+                backgroundColor: 'rgba(54, 162, 235, 0.6)',
+                color: 'white',
+                font: {
+                  size: 10
+                }
+              }
+            }
+          }
+        }
+      }
+    };
+    
+    return (
+      <div className="debug-section">
+        <h3>Probability Distribution ({probData.total_candidates} candidates)</h3>
+        <p>Selection probabilities from softmax algorithm (temperature: {probData.temperature}, difficulty: {probData.difficulty})</p>
+        
+        <div className="prob-summary">
+          <div><strong>Selected:</strong> {probData.selected_count} words</div>
+          <div><strong>Top Probability:</strong> {(Math.max(...sortedByPercentile.map(p => p.probability)) * 100).toFixed(1)}%</div>
+          <div><strong>Average:</strong> {((1/probData.total_candidates) * 100).toFixed(1)}%</div>
+          <div><strong>Temperature Effect:</strong> {probData.temperature < 1 ? 'More deterministic' : probData.temperature > 1 ? 'More random' : 'Balanced'}</div>
+          <div><strong>Mean Position:</strong> Word #{meanWordIndex + 1} ({sortedByPercentile[meanWordIndex]?.word})</div>
+          <div><strong>Distribution Width (σ):</strong> {sigma.toFixed(1)} words</div>
+          <div><strong>σ Sampling Zone:</strong> {(sigmaRangeProbMass * 100).toFixed(1)}% of probability mass</div>
+          <div><strong>σ Range:</strong> Words #{sigmaRangeStart + 1}-#{sigmaRangeEnd + 1}</div>
+        </div>
+
+        {/* Interactive Bar Chart */}
+        <div className="chart-container">
+          <div style={{ height: '500px', marginBottom: '20px' }}>
+            <Bar data={chartData} options={chartOptionsWithAnnotations} />
+          </div>
+          <p className="chart-description">
+            <strong>📊 Frequency-Based Analysis:</strong> This chart shows ALL {probData.total_candidates} candidate words sorted by 
+            frequency percentile (100% → 0%, common → rare). This reveals whether the Gaussian frequency targeting 
+            is working correctly for your selected difficulty level. Look for probability peaks at the intended percentile ranges:
+            <strong> Easy (90%+), Medium (50%), Hard (20%)</strong>.
+          </p>
+        </div>
+
+        {/* Detailed Table */}
+        <h4>Detailed Probability Data</h4>
+        <div className="probability-table-container">
+          <table className="probability-table">
+            <thead>
+              <tr>
+                <th>Rank</th>
+                <th>Word</th>
+                <th>Probability</th>
+                <th>Composite</th>
+                <th>Similarity</th>
+                <th>Percentile</th>
+                <th>Selected</th>
+              </tr>
+            </thead>
+            <tbody>
+              {sortedByPercentile.map((item, idx) => (
+                <tr key={idx} className={item.selected ? 'selected-word' : ''}>
+                  <td>{item.probability_rank}</td>
+                  <td><strong>{item.word}</strong></td>
+                  <td>
+                    <div className="probability-cell">
+                      <span className="prob-text">{(item.probability * 100).toFixed(2)}%</span>
+                      <div 
+                        className="prob-bar" 
+                        style={{ 
+                          width: `${Math.max(2, item.probability * 100 * 2)}px`,
+                          backgroundColor: item.selected ? '#4CAF50' : '#e0e0e0'
+                        }}
+                      />
+                    </div>
+                  </td>
+                  <td>{item.composite_score.toFixed(3)}</td>
+                  <td>{item.similarity.toFixed(3)}</td>
+                  <td>{item.percentile.toFixed(3)}</td>
+                  <td>{item.selected ? '✓' : '✗'}</td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        </div>
+      </div>
+    );
+    } catch (error) {
+      console.error('Error rendering probabilities:', error);
+      return (
+        <div className="debug-section">
+          <h3>Probability Distribution</h3>
+          <p style={{color: 'red'}}>Error rendering chart: {error.message}</p>
+          <p>Debug data available: {JSON.stringify(Object.keys(probData || {}))}</p>
+        </div>
+      );
+    }
+  };
+
   const renderSelected = () => {
     const selected = debugData.selected_words || [];
     
@@ -236,6 +592,7 @@ const DebugTab = ({ debugData }) => {
       case 'thematic-pool': return renderThematicPool();
       case 'candidates': return renderCandidates();
       case 'selection': return renderSelection();
+      case 'probabilities': return renderProbabilities();
       case 'selected': return renderSelected();
       default: return renderOverview();
     }

crossword-app/frontend/src/styles/puzzle.css

@@ -720,6 +720,117 @@
   line-height: 1.4;
 }
 
+/* Probability Distribution Styling */
+.prob-summary {
+  display: grid;
+  grid-template-columns: repeat(4, 1fr);
+  gap: 10px 15px;
+  margin: 15px 0 20px 0;
+  padding: 15px;
+  background: #f8f9fa;
+  border-radius: 8px;
+  font-size: 0.9rem;
+}
+
+.chart-container {
+  margin: 20px 0;
+  padding: 20px;
+  background: #ffffff;
+  border: 1px solid #dee2e6;
+  border-radius: 8px;
+  box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+}
+
+.chart-description {
+  background: #e3f2fd;
+  padding: 12px 15px;
+  border-radius: 6px;
+  border-left: 4px solid #1976d2;
+  margin-top: 15px;
+  font-size: 0.9rem;
+  line-height: 1.4;
+  color: #1565c0;
+}
+
+.probability-table-container {
+  max-height: 600px;
+  overflow-y: auto;
+  border: 1px solid #dee2e6;
+  border-radius: 8px;
+}
+
+.probability-table {
+  width: 100%;
+  border-collapse: collapse;
+  font-size: 0.9rem;
+}
+
+.probability-table th {
+  background: #495057;
+  color: white;
+  padding: 12px 8px;
+  text-align: left;
+  font-weight: 600;
+  position: sticky;
+  top: 0;
+  z-index: 10;
+  border-bottom: 2px solid #343a40;
+}
+
+.probability-table td {
+  padding: 8px;
+  border-bottom: 1px solid #e9ecef;
+  vertical-align: middle;
+}
+
+.probability-table tr:hover {
+  background: #f8f9fa;
+}
+
+.probability-table tr.selected-word {
+  background: #e8f5e8;
+  border-left: 4px solid #4CAF50;
+}
+
+.probability-table tr.selected-word:hover {
+  background: #d4edda;
+}
+
+.probability-cell {
+  display: flex;
+  align-items: center;
+  gap: 10px;
+}
+
+.prob-text {
+  min-width: 60px;
+  font-weight: 600;
+}
+
+.prob-bar {
+  height: 16px;
+  border-radius: 8px;
+  transition: all 0.3s ease;
+  min-width: 2px;
+}
+
+.probability-table td:first-child {
+  text-align: center;
+  color: #6c757d;
+  font-weight: 600;
+}
+
+.probability-table td:last-child {
+  text-align: center;
+  font-size: 1.1rem;
+  font-weight: bold;
+  color: #4CAF50;
+}
+
+.probability-table tr:not(.selected-word) td:last-child {
+  color: #f44336;
+}
+
 /* Responsive */
 @media (max-width: 768px) {
   .debug-nav {
@@ -743,4 +854,32 @@
   .word-table td {
     padding: 4px 8px;
   }
+  
+  .prob-summary {
+    grid-template-columns: repeat(2, 1fr);
+    text-align: center;
+  }
+  
+  .chart-container {
+    padding: 10px;
+    margin: 10px 0;
+  }
+  
+  .probability-table {
+    font-size: 0.75rem;
+  }
+  
+  .probability-table th,
+  .probability-table td {
+    padding: 6px 4px;
+  }
+  
+  .prob-text {
+    min-width: 50px;
+    font-size: 0.8rem;
+  }
+  
+  .prob-bar {
+    height: 12px;
+  }
 }