crossword-app/backend-py/docs/distribution_normalization_analysis.md

# Distribution Normalization Analysis

## Overview

Distribution normalization is a feature implemented to ensure consistent difficulty levels across different topics in the crossword generator. This document analyzes the trade-offs between normalized and non-normalized approaches and provides recommendations.

## The Problem

The original question was: *"Should we normalize the distribution before display? Perhaps the distribution will be centered at the same position for a difficulty level irrespective of topic."*

Different topics naturally have different semantic similarity ranges:
- **"Animals"**: Rich vocabulary, similarities often range 0.4-0.9
- **"Philosophy"**: Abstract concepts, similarities might range 0.1-0.6  
- **"Technology"**: Mixed range, similarities around 0.2-0.8

This led to perceived "inconsistent difficulty" where "Easy Animals" felt easier than "Easy Philosophy" crosswords.

## Current Implementation

### Composite Score Formula
```
composite = (1 - difficulty_weight) * similarity + difficulty_weight * freq_score
```

With default `difficulty_weight = 0.5`:
```
composite = 0.5 * similarity + 0.5 * freq_score
```

### Normalization Methods

1. **`similarity_range` (default)**: Normalizes similarities to [0,1] before composite calculation
2. **`composite_zscore`**: Z-score normalization (unbounded, typically -3 to +3)
3. **`percentile_recentering`**: Boosts scores based on proximity to target percentile (can exceed 1.0)

### Configuration
- `ENABLE_DISTRIBUTION_NORMALIZATION=true` (default)
- `NORMALIZATION_METHOD=similarity_range` (default)

## Trade-offs Analysis

### Before Normalization (Original System)

#### Advantages ✅
1. **Natural semantic relationships preserved**
   - Topics with broader vocabulary naturally had higher similarity ranges
   - Reflected genuine linguistic density differences
   - Authentic representation of semantic space

2. **Simpler and more predictable**
   - Straightforward composite score calculation
   - Always bounded to [0,1] naturally
   - No artificial transformations

3. **Semantic honesty**
   - Some topics ARE inherently harder to generate crosswords for
   - System reflected this reality rather than masking it
   - Valuable information for both system and users

4. **Computational efficiency**
   - No additional normalization calculations
   - Cleaner code path

#### Disadvantages ❌
1. **Inconsistent difficulty across topics**
   - "Easy" for animals genuinely easier than "Easy" for philosophy
   - Could confuse users expecting uniform difficulty

2. **User expectation mismatch**
   - Players might expect same difficulty label = same challenge level

### After Normalization (Current System)

#### Advantages ✅
1. **Consistent difficulty intent**
   - Attempts to make "Easy" equally easy across all topics
   - Meets user expectations for uniform difficulty labels

2. **Debug visualization enhancements**
   - Shows normalization effects in debug tab
   - Helpful for analysis and understanding

#### Disadvantages ❌
1. **Artificial stretching of similarity ranges**
   - Forces sparse topics to use full [0,1] range
   - Genuinely dissimilar words appear artificially similar
   - Loss of semantic authenticity

2. **Implementation complexity and bugs**
   - Different methods produce different ranges
   - Z-score normalization is unbounded
   - Percentile recentering can exceed 1.0
   - Softmax sensitivity to inconsistent ranges

3. **Loss of valuable information**
   - Masks natural vocabulary density differences
   - Hides genuine topic difficulty characteristics
   - Makes debugging harder (what's "real" vs "normalized"?)

4. **Computational overhead**
   - Additional calculations for normalization
   - More complex code paths
   - Potential for numerical issues

## Composite Score Ranges

### Without Normalization
- **Theoretical range**: [0, 1]
- **Practical range**: Depends on actual similarities in the 150-word thematic pool
- **Example**: If similarities range 0.3-0.7, composite ≈ [0.15, 0.85]

### With Normalization
- **`similarity_range`**: ~[0, 1] (most consistent)
- **`composite_zscore`**: Unbounded (typically [-3, +3])
- **`percentile_recentering`**: Can exceed 1.0 due to boosting

## Problems with Current Implementation

1. **Range inconsistency**: Different normalization methods produce different ranges
2. **Unbounded z-scores**: Affect softmax probability calculations unpredictably  
3. **Values exceeding [0,1]**: Break assumptions about composite score bounds
4. **Complexity without clear benefit**: Added complexity for questionable gains

## Recommendation

### **Revert to Non-Normalized Approach** 

The original system was **better** for these reasons:

1. **The "problem" wasn't really a problem**
   - Different topics having different difficulty distributions is natural and informative
   - Philosophy IS harder to make crosswords for than animals - this is linguistic reality

2. **Normalization introduces distortions**
   - Stretching narrow ranges doesn't make words more semantically similar
   - Creates artificial relationships that don't exist

3. **Alternative solutions are better**:
   - Show users the natural difficulty of each topic
   - Adjust word count based on topic vocabulary density  
   - Provide topic difficulty ratings to set expectations
   - Use adaptive difficulty within topics rather than across them

### If Normalization is Kept

If normalization must be retained:

1. **Make it opt-in, not default**: `ENABLE_DISTRIBUTION_NORMALIZATION=false`
2. **Fix range consistency**: Ensure all methods produce [0,1] outputs
3. **Add proper bounds checking**: Clamp scores to [0,1] after normalization
4. **Document trade-offs clearly**: Let users make informed choices

## Proposed Implementation Fixes

If keeping normalization, fix these issues:

```python
# After normalization, ensure consistent [0,1] range
if method == "composite_zscore":
    # Map z-scores to [0,1] using sigmoid
    scores = 1 / (1 + np.exp(-normalized_scores))
elif method == "percentile_recentering":
    # Clamp boosted scores to valid range
    scores = np.clip(boosted_scores, 0, 1)

# Final safety clamp for all methods
composite_scores = np.clip(composite_scores, 0, 1)
```

## Conclusion

The **non-normalized approach respects semantic reality** and provides more honest, interpretable results. The "inconsistency" across topics is actually valuable information about linguistic structure, not a bug to be fixed.

**Recommendation**: Disable normalization by default (`ENABLE_DISTRIBUTION_NORMALIZATION=false`) and let the natural semantic relationships guide difficulty distribution. This preserves the system's authenticity while maintaining simplicity and predictability.

The original system's variation across topics was a **feature representing real linguistic diversity**, not a problem requiring artificial correction.