Spaces:

happy-place
/

transformers_recsys

Sleeping

App Files Files Community

transformers_recsys / TESTING_FRAMEWORK_DOCUMENTATION.md

minhajHP

Initial commit: Transformer recommendation system with inference weights

e762dab 11 months ago

preview code

Raw

History Blame Contribute Delete

43.3 kB

Testing Framework Documentation

Two-Tower Transformer Recommendation System

Generated: 2025-09-29 (Updated) Project: Two-Tower Transformer-Based Recommendation System Version: 2.0

Overview
Framework Architecture
Test Organization
Advanced Test Runner
Core Test Scripts
Analysis Scripts
Debug Scripts
Testing Methodologies
Configuration System
Expected Results & Metrics
Usage Guidelines
Dependencies

Overview

The Two-Tower Transformer Recommendation System features an advanced testing framework designed to validate architecture, training dynamics, embedding quality, and recommendation performance. Version 2.0 introduces significant improvements including parallel execution, centralized configuration, and enhanced reporting capabilities.

Key Features

🚀 Advanced Test Runner: Parallel execution with intelligent scheduling
📊 Comprehensive Coverage: Tests architecture, training, inference, and recommendation quality
🔧 Modular Design: Object-oriented test structure with inheritance
📈 Statistical Rigor: Proper statistical analysis and significance testing
🔍 Issue Detection: Proactive detection of common deep learning issues
💡 Actionable Outputs: Specific recommendations for fixing detected issues
⚡ Parallel Execution: Concurrent test execution for faster results
📋 Centralized Config: Unified configuration management system
📊 Enhanced Reporting: Rich HTML/JSON reports with visualizations

Framework Architecture

The testing framework follows a layered architecture:

tests/
├── config/
│   └── test_config.py          # Centralized configuration
├── unit/                       # Unit tests
├── integration/                # Integration tests
├── performance/                # Performance tests
├── similarity/                 # Similarity tests
├── analysis/                   # Analysis utilities
├── debug/                      # Debug and diagnostic tools
├── test_results/               # Test outputs and reports
├── base_test.py               # Base test class
├── test_runner.py             # Advanced test runner
└── run_all_tests.py           # Legacy comprehensive runner

Core Components

BaseTest Class: Abstract base class providing common functionality
TestConfig: Centralized configuration management
TestRegistry: Dynamic test discovery and registration
AdvancedTestRunner: Parallel execution engine with comprehensive reporting

Test Organization

Test Categories

Tests are organized into five main categories:

🔧 Unit Tests (`tests/unit/`)

Purpose: Test individual components in isolation
Execution: Parallel (fast execution)
Timeout: 30 seconds per test
Examples:
- architectural_validation_tests.py - Architecture compliance
- transformer_layer_analysis.py - Layer-by-layer analysis
- gradient_flow_validation.py - Gradient health monitoring
- test_embedding_quality.py - ⭐ NEW Item embedding quality validation
- test_user_embeddings.py - ⭐ NEW User embedding quality validation

🔗 Integration Tests (`tests/integration/`)

Purpose: Test component interactions and workflows
Execution: Sequential (dependencies between tests)
Timeout: 2 minutes per test
Examples:
- test_main_transformer_integration.py - End-to-end integration
- joint_training_analysis.py - Joint training workflows
- test_improved_joint_training.py - ⭐ Category-aware InfoNCE validation
- user_embedding_generation_test.py - User embedding pipeline
- test_transformer_inference_loading.py - ⭐ NEW Inference engine validation

🎯 Similarity Tests (`tests/similarity/`)

Purpose: Validate embedding similarity and alignment
Execution: Parallel (independent similarity computations)
Timeout: 3 minutes per test
Examples:
- embedding_similarity_analysis.py - User-user similarity analysis
- similarity_alignment_validation.py - Alignment mechanism validation
- test_semantic_item_similarity.py - Item semantic similarity

⚡ Performance Tests (`tests/performance/`)

Purpose: Benchmark system performance and resource usage
Execution: Sequential (resource intensive)
Timeout: 5 minutes per test
Examples:
- comprehensive_recommendation_analysis.py - End-to-end performance
- category_alignment_analysis.py - Category processing performance

📊 Analysis Tests (`tests/analysis/`)

Purpose: Deep analysis and diagnostic insights
Execution: Parallel (independent analysis tasks)
Timeout: 4 minutes per test
Examples:
- analyze_item_embeddings.py - Item embedding deep analysis
- analyze_training_issues.py - Training diagnostics

Advanced Test Runner

Features

The new test_runner.py provides advanced capabilities:

# Run all tests with parallel execution
python tests/test_runner.py

# Run specific categories
python tests/test_runner.py --categories similarity unit

# Force sequential execution
python tests/test_runner.py --sequential

# List available tests
python tests/test_runner.py --list-tests

# Custom output directory
python tests/test_runner.py --output-dir custom_results/

Execution Modes

Parallel Execution (Default)

Tests within categories run concurrently
Configurable worker pool (default: 4 workers)
Intelligent scheduling based on test dependencies
Real-time progress monitoring

Sequential Execution

Traditional one-after-another execution
Better for debugging and resource-constrained environments
Detailed per-test logging
Immediate failure reporting

Reporting Capabilities

Console Output

Real-time test status updates
Color-coded results (✅ success, ❌ failed, ⏭️ skipped)
Progress indicators and timing information
Summary statistics and health scores

JSON Reports

{
  "execution_summary": {
    "total_tests": 25,
    "successful_tests": 23,
    "failed_tests": 1,
    "success_rate": 0.92,
    "health_score": 0.85,
    "total_duration": 45.7
  },
  "results_by_category": {...},
  "issues_summary": {...},
  "detailed_results": [...]
}

Health Scoring

🟢 Healthy (0.8-1.0): System performing optimally
🟡 Needs Attention (0.6-0.8): Minor issues detected
🔴 Critical Issues (0.0-0.6): Major problems requiring immediate attention

Configuration System

Centralized Configuration (`tests/config/test_config.py`)

All test parameters are managed through the centralized configuration system:

class TestConfig:
    # Test execution settings
    EXECUTION = {
        'max_parallel_tests': 4,
        'retry_failed_tests': True,
        'retry_count': 2,
        'continue_on_failure': True
    }

    # Quality thresholds
    THRESHOLDS = {
        'embedding_variance': 0.01,
        'similarity_collapse': 0.95,
        'gradient_vanishing': 1e-8,
        'health_score_critical': 0.5
    }

    # Model parameters
    MODEL_PARAMS = {
        'embedding_dim': 128,
        'transformer_layers': 4,
        'transformer_heads': 8
    }

Benefits

Consistency: Same parameters across all tests
Maintainability: Single location for configuration changes
Flexibility: Easy parameter tuning for different environments
Validation: Built-in artifact path validation

Core Test Scripts

1. run_all_tests.py - Master Test Orchestrator

Purpose: Unified test runner that executes and coordinates all test suites

What it tests:

Entire system integration
Cross-component compatibility
Overall system health

How it works:

Sequential execution of test suites
Health score calculation (0-1 scale)
Cross-test analysis and correlation
Unified JSON report generation

Expected Results:

{
  "overall_health_score": 0.85,
  "test_results": {...},
  "critical_issues": [...],
  "recommendations": [...]
}

Usage:

python tests/run_all_tests.py

2. architectural_validation_tests.py - Architecture Validator

Purpose: Validates transformer architecture design and implementation

What it tests:

Model architecture compliance
Layer connections and dimensions
Parameter flow integrity
Component compatibility

How it works:

White-box testing of architectural components
Dimension flow validation
Parameter count verification
Layer connectivity analysis

Expected Results:

Architecture compliance report
Structural validation metrics
Dimension mismatch detection
Component health scores

Key Metrics:

Total parameters: ~1M for user tower
Layer dimensions: 128D embeddings
Attention heads: 8 per layer
Transformer layers: 4

3. transformer_layer_analysis.py - Layer-by-Layer Inspector

Purpose: Deep analysis of each transformer layer's behavior

What it tests:

Individual layer performance
Weight distribution health
Activation patterns
Gradient flow integrity

How it works:

Statistical analysis of layer outputs
Weight histogram analysis
Dead neuron detection
Gradient magnitude monitoring

Expected Results:

{
  "layer_health": {
    "attention_layer_1": {
      "gradient_norm": 0.001,
      "dead_neurons": 0.05,
      "weight_variance": 0.02
    }
  }
}

Healthy Ranges:

Gradient norms: 1e-6 to 10
Dead neuron ratio: < 20%
Weight variance: > 0.001

4. embedding_alignment_tests.py - Embedding Quality Validator

Purpose: Tests alignment and quality of user/item embeddings

What it tests:

Embedding space quality
User-item alignment
Similarity computation accuracy
Embedding collapse detection

How it works:

Statistical analysis of embedding distributions
Correlation analysis between embeddings
Cosine similarity validation
Variance and diversity metrics

Expected Results:

Embedding quality scores (0-1)
Alignment metrics
Collapse severity assessment
Diversity measurements

Quality Thresholds:

Embedding variance: > 0.01
Similarity separation: > 0.1
Dead dimensions: < 50%

5. similarity_alignment_validation.py - Similarity Mechanism Tester

Purpose: Validates similarity computation and alignment mechanisms

What it tests:

Temperature parameter effectiveness
Anti-correlation fixes
Similarity transformations
Bias correction mechanisms

How it works:

Parameter sensitivity analysis
Correlation testing across user types
Temperature scaling validation
Bias impact assessment

Expected Results:

Optimal temperature values
Correlation improvement metrics
Bias effectiveness scores
Alignment quality indicators

6. gradient_flow_validation.py - Gradient Health Monitor

Purpose: Validates gradient flow through transformer architecture

What it tests:

Gradient magnitude health
Vanishing/exploding gradient detection
Training stability
Layer-wise gradient analysis

How it works:

Gradient tape analysis
Statistical gradient monitoring
Layer-wise norm computation
Training step analysis

Expected Results:

{
  "gradient_health": {
    "mean_gradient_norm": 0.005,
    "vanishing_layers": 0,
    "exploding_layers": 0,
    "healthy_ratio": 0.95
  }
}

Healthy Ranges:

Gradient norms: 1e-6 to 10
Vanishing threshold: < 1e-8
Exploding threshold: > 100

7. user_embedding_generation_test.py - User Embedding Validator

Purpose: Tests user embedding generation across diverse profiles

What it tests:

User tower functionality
Demographic vs. interaction influence
Cold-start capability
Embedding consistency

How it works:

Systematic user profile generation
Embedding analysis across archetypes
Clustering analysis
Temporal stability testing

User Archetypes Tested:

Young tech professionals
Middle-aged executives
College students
Senior retirees
Working parents

Expected Results:

Embedding diversity metrics
Cold-start vs. warm user analysis
Clustering purity scores
Temporal stability indicators

8. interaction_embedding_analysis.py - Interaction Pattern Analyzer

Purpose: Analyzes how interaction patterns affect user embeddings

What it tests:

Sequence length impact
Behavioral consistency
Attention pattern quality
Interaction vs. demographic influence

How it works:

Controlled interaction pattern variation
Fixed demographic analysis
Sequence length testing (0-50 items)
Behavioral pattern classification

Expected Results:

Sequence length impact curves
Behavioral consistency scores
Attention weight distributions
Influence ratio metrics

9. test_embedding_collapse_detection.py - Collapse Detection System

Purpose: Detects embedding collapse in user and item towers

What it tests:

Embedding variance health
Dimensional diversity
Dead dimension detection
Collapse severity assessment

How it works:

Statistical variance analysis
Similarity threshold testing (0.95)
Variance threshold validation (0.01)
Dimensional utilization analysis

Expected Results:

Collapse severity: NONE/MILD/MODERATE/CRITICAL
Dead dimension ratios
Variance statistics
Remediation recommendations

Collapse Indicators:

Variance < 0.01 (critical)
Similarity > 0.95 (high collapse)
Dead dimensions > 50% (critical)

10. joint_training_analysis.py - Joint Training Inspector

Purpose: Comprehensive analysis of joint training dynamics

What it tests:

Joint model convergence
Gradient conflict detection
Loss balance optimization
Embedding alignment during training

How it works:

Multi-objective loss analysis
Gradient conflict measurement
Training dynamics monitoring
Anti-correlation pattern detection

Expected Results:

Loss balance metrics
Gradient conflict scores
Training stability indicators
Embedding evolution analysis

11. test_improved_joint_training.py - Category-Aware InfoNCE Validator ⭐ NEW

Purpose: Validates improved category-aware InfoNCE loss implementation with hard negative mining

What it tests:

Category-aware positive/negative definition
Hard negative mining effectiveness
Temperature parameter sensitivity
Gradient flow through enhanced InfoNCE loss
Training step integration with real artifacts

How it works:

Tests enhanced InfoNCE loss with 3 sample types:
- Positives: High rating (≥4.0) + category match
- Hard negatives: High rating + category mismatch (3x weight)
- Easy negatives: Low rating (<4.0) any category (1x weight)
Temperature scaling validation (0.01-0.5 range)
Synthetic data generation for controlled testing
Real artifact integration testing

Test Scenarios:

# Scenario testing
1. All positive samples → Expected: Lower loss
2. All negative samples → Expected: Minimal loss (0.001)
3. Structured by category → Expected: Medium loss
4. Temperature sensitivity → Expected: Responsive to changes
5. Multiple training steps → Expected: Stable execution

Expected Results:

{
  "infonce_loss": 1.3965,
  "gradient_health": {
    "mean": 1.59, "max": 8.70, "min": 0.00
  },
  "temperature_sensitivity": [
    {"temp": 0.01, "loss": 0.673},
    {"temp": 0.05, "loss": 0.609},
    {"temp": 0.1, "loss": 0.777},
    {"temp": 0.5, "loss": 1.020}
  ],
  "training_integration": "✅ SUCCESS"
}

Key Validations:

InfoNCE loss computation accuracy
Proper tensor shape handling (tf.function compatibility)
Category-aware sample classification
Hard negative mining with weighted importance
Integration with real pre-trained artifacts
Multi-step training stability

Health Indicators:

Test pass rate: 100% (2/2 tests)
Gradient norms: 0.0-8.7 (healthy range)
Temperature responsiveness: ✅ Confirmed
Loss values: Within expected ranges
Training stability: ✅ Confirmed across multiple steps

Usage:

# Run improved joint training validation
python test_improved_joint_training.py

# Expected output: 🎉 ALL TESTS PASSED!

Added: 2025-09-29 (Part of joint training improvement initiative)

12. test_embedding_quality.py - Item Embedding Quality Validator ⭐ NEW

Purpose: Comprehensive validation of transformer item embedding quality and meaningfulness

What it tests:

Basic embedding properties (shape, dtype, NaN/infinity detection)
Embedding collapse detection (collapsed dimensions analysis)
Semantic diversity measurement (pairwise similarity analysis)
Value distribution health (range, variance, extreme values)
Meaningful differentiation between items

How it works:

Loads transformer item embeddings from artifacts
Statistical analysis of embedding distributions
Cosine similarity computation on sample items (100 items)
Variance and standard deviation analysis across dimensions
Embedding norm validation and consistency checks

Test Components:

Property Testing:
- Embedding shape: Expected (19095, 128)
- Data type validation: float32
- NaN/Infinity detection
- Norm consistency (should be ~1.0 for normalized embeddings)
Collapse Detection:
- Collapsed dimensions: < 50% of total dimensions
- Mean standard deviation: > 0.01 (healthy variance)
- Diversity score: > 0.001 (meaningful differentiation)
Similarity Analysis:
- Pairwise cosine similarity on random sample
- Mean similarity: Should not be > 0.99 (indicates collapse)
- Standard deviation: > 0.001 (indicates diversity)
- Range analysis: Healthy spread of similarities
Distribution Testing:
- Value range validation (reasonable bounds)
- Extreme value detection (< 10% outliers)
- Mean and variance analysis

Expected Results:

{
  "properties": {
    "collapsed_dims": 0,
    "mean_std": 0.0052,
    "norm_stats": {"min": 1.0, "max": 1.0, "mean": 1.0},
    "has_nan": false,
    "has_inf": false,
    "embedding_shape": [19095, 128]
  },
  "similarity": {
    "mean_similarity": 0.9996,
    "diversity_score": 0.0002,
    "samples_tested": 100
  },
  "distribution": {
    "min": -0.2881,
    "max": 0.2908,
    "extreme_values": 0
  }
}

Health Indicators:

✅ No Collapse: 0 collapsed dimensions
⚠️ Low Variance: Mean std = 0.0052 (below 0.01 threshold)
⚠️ Low Diversity: Diversity score = 0.0002 (below 0.001 threshold)
✅ No Technical Issues: No NaN/infinity values
✅ Proper Normalization: Consistent L2 norms

Critical Issues Detected:

Very Low Variance Across Dimensions (Warning)
- Indicates embeddings may not be meaningfully differentiated
- Suggests potential training issues or insufficient learning
Very Low Embedding Diversity (Critical)
- Mean similarity: 0.9996 (extremely high)
- Items are nearly identical in embedding space
- Severely impacts recommendation quality

Recommendations Generated:

Medium Priority: Consider increasing learning rate or training epochs
High Priority: Check for embedding collapse - may need regularization
Technical: Review item tower architecture for sufficient capacity
Training: Implement techniques to encourage embedding diversity

Integration:

Part of unit test suite (fast execution)
Integrated with framework's BaseTest class
Generates structured JSON reports
Provides actionable issue detection and recommendations

Usage:

# Run standalone
python tests/test_embedding_quality.py

# Run through framework
python tests/test_runner.py --categories unit

# Expected output: ⚠️ Issues detected but test completes successfully

Quality Thresholds:

Collapsed dimensions: < 50% of total dimensions
Mean variance: > 0.01 (healthy)
Diversity score: > 0.001 (meaningful differentiation)
Similarity mean: < 0.99 (not collapsed)
Extreme values: < 10% of total values

Added: 2025-09-30 (Part of embedding quality validation initiative)

13. test_transformer_inference_loading.py - Inference Engine Validator ⭐ NEW

Purpose: Comprehensive validation of transformer inference engine loading and performance

What it tests:

Transformer engine initialization and loading times
Component loading verification (user tower, item tower, FAISS index)
Vocabulary loading and size validation
User embedding generation functionality
Item embedding retrieval from FAISS index
Recommendation generation workflow
Rating prediction performance optimization

How it works:

Loads TransformerRecommendationEngine with timing measurements
Validates all essential components are loaded correctly
Tests basic functionality with sample user profiles
Measures performance of rating prediction (should be instant)
Generates structured reports with component status

Test Components:

Engine Initialization:
- Loading time: < 3.0 seconds (good performance)
- Component verification: 5/5 essential components
- Memory usage and initialization efficiency
Component Loading Validation:
- Vocabularies: 19,095 items, 238 categories, 1,151 brands
- User tower: Transformer architecture with attention
- Item tower: Improved embeddings with quality validation
- FAISS index: Fast similarity search ready
- Items dataframe: Metadata available
Functionality Testing:
- User embedding shape: (128,) - correct dimensionality
- Item embedding retrieval: Valid embeddings from FAISS
- Recommendation generation: 5 recommendations with scores
- Rating prediction: Instant response (< 1ms)
Performance Validation:
- Loading time monitoring and optimization detection
- Rating prediction speed (optimized to be instant)
- Memory efficiency and component initialization

Expected Results:

{
  "loading_time": 1.65,
  "components": {
    "vocabularies": true,
    "user_tower": true,
    "item_tower": true,
    "faiss_index": true,
    "items_dataframe": true
  },
  "user_embedding": {"shape": [128], "dtype": "float32"},
  "item_embedding": {"shape": [128], "dtype": "float32"},
  "recommendations": {
    "count": 5,
    "sample_item": 1004788,
    "sample_score": 0.5141
  },
  "rating_performance": {
    "rating": 0.5,
    "prediction_time": 0.000003,
    "is_instant": true
  }
}

Health Indicators:

✅ Fast Loading: 1.65s loading time (well under 3s threshold)
✅ All Components Loaded: 5/5 essential components available
✅ Optimized Rating: 0.000003s prediction time (instant)
✅ Quality Embeddings: Using improved transformer embeddings
✅ Full Functionality: All core features working correctly

Integration Benefits:

Continuous Monitoring: Validates inference engine health
Performance Tracking: Monitors loading and prediction times
Component Verification: Ensures all required artifacts are available
Regression Detection: Catches breaking changes in inference pipeline
Deployment Readiness: Confirms system ready for production use

Usage:

# Run standalone
python tests/test_transformer_inference_loading.py

# Run through framework
python tests/test_runner.py --categories integration

# Expected output: ✅ All components loaded, fast performance confirmed

Quality Thresholds:

Loading time: < 3.0 seconds (good), < 5.0 seconds (acceptable)
Component loading: 5/5 essential components required
Rating prediction: < 0.001 seconds (instant)
User embedding: Shape (128,) with valid values
Item embedding: Available from FAISS with correct dimensions

Added: 2025-09-30 (Part of inference optimization and validation initiative)

14. test_user_embeddings.py - User Embedding Quality Validator ⭐ NEW

Purpose: Comprehensive validation of transformer user embedding quality, diversity, and semantic differentiation

What it tests:

User embedding generation for diverse demographic profiles
Embedding diversity and similarity patterns between different user types
Basic embedding properties (shape, dtype, normalization)
Variance analysis across embedding dimensions
Value distribution health and range validation
Recommendation functionality with different user profiles

How it works:

Tests 5 diverse user profiles: tech professional, healthcare worker, senior retiree, student, executive
Generates user embeddings through transformer user tower
Analyzes pairwise similarities between different user demographics
Validates embedding properties and identifies potential issues
Tests recommendation generation for functional validation

Test Components:

User Profile Diversity:
- Young Tech Professional (25, male, $80K, Technology, Urban)
- Middle-aged Healthcare Worker (45, female, $60K, Healthcare, Suburban)
- Senior Retiree (68, male, $40K, Other, Rural)
- Young Student (20, female, $15K, Education, Urban)
- High Earner Executive (40, male, $150K, Finance, Urban)
Embedding Properties Testing:
- Shape validation: (5, 128) for 5 users, 128 dimensions
- Data type: float32 consistency
- NaN/Infinity detection: Should be clean
- Normalization: L2 norms should be ~1.0
Diversity Analysis:
- Pairwise similarity computation between all user pairs
- Mean similarity: Should be < 0.95 (not too similar)
- Standard deviation: Should be > 0.01 (meaningful variation)
- Range analysis: Healthy spread of similarity values
Variance Analysis:
- Mean standard deviation across dimensions
- Collapsed dimension detection (< 1e-6 threshold)
- Dimensional utilization efficiency
Functional Testing:
- Recommendation generation for different user types
- Validation that different users get different recommendations
- Score variation analysis

Expected Results:

{
  "user_count": 5,
  "basic_properties": {
    "shape": [5, 128],
    "dtype": "float32",
    "has_nan": false,
    "has_inf": false,
    "norm_stats": {"min": 1.0, "max": 1.0, "mean": 1.0}
  },
  "diversity": {
    "mean_similarity": 0.8937,
    "std_similarity": 0.0763,
    "min_similarity": 0.8053,
    "max_similarity": 1.0000
  },
  "variance": {
    "mean_std": 0.0226,
    "collapsed_dims": 0,
    "total_dims": 128
  },
  "recommendations": [
    {"user": "Young Tech Professional", "top_item": 28720387, "top_score": 0.5135},
    {"user": "Middle-aged Healthcare Worker", "top_item": 28720387, "top_score": 0.4309}
  ]
}

Health Indicators:

✅ Technical Health: No NaN/infinity, proper normalization (1.0 norms)
⚠️ Limited Differentiation: Some user pairs show identical embeddings (1.0000 similarity)
✅ Functional: All users generate recommendations successfully
⚠️ Moderate Diversity: std=0.0763 (acceptable but could be higher)
✅ No Collapse: 0 collapsed dimensions

Issues Detected:

Identical Embeddings: Healthcare Worker vs Student vs Executive (1.0000 similarity)
High Mean Similarity: 0.8937 (users quite similar in embedding space)
Limited Variance: 0.0226 mean std per dimension (lower than item embeddings)

Comparison with Item Embeddings:

Metric	Item Embeddings	User Embeddings	Assessment
Mean std per dimension	0.0877 (excellent)	0.0226 (moderate)	Item tower superior
Diversity score	0.1396 (excellent)	0.0763 (moderate)	Item tower superior
Mean similarity	0.0071 (very diverse)	0.8937 (similar)	Item tower superior
Technical health	✅ Perfect	✅ Perfect	Both healthy
Functionality	✅ Working	✅ Working	Both functional

Integration Benefits:

User Tower Monitoring: Validates transformer user tower functionality
Demographic Coverage: Tests diverse user demographics and use cases
Quality Benchmarking: Provides baseline metrics for user embedding quality
Regression Detection: Catches degradation in user representation quality
Comparative Analysis: Enables comparison with item embedding performance

Usage:

# Run standalone
python tests/test_user_embeddings.py

# Run through framework
python tests/test_runner.py --categories unit

# Expected output: ⚠️ Some identical embeddings detected but test passes

Quality Thresholds:

Mean similarity: < 0.95 (acceptable), < 0.85 (good)
Diversity (std): > 0.01 (acceptable), > 0.05 (good)
Variance per dimension: > 0.01 (acceptable), > 0.05 (good)
Collapsed dimensions: 0 (required)
Identical pairs: < 50% of total pairs (acceptable)

Improvement Opportunities:

User tower could benefit from similar training improvements as item tower
Consider demographic feature engineering for better differentiation
Explore contrastive learning approaches for user embeddings
Investigate attention mechanism optimization for user profiles

Added: 2025-09-30 (Part of comprehensive embedding validation initiative)

Analysis Scripts

tests/analysis/analyze_item_embeddings.py

Purpose: Comprehensive item embedding analysis

Features:

Item category alignment analysis
Price-embedding correlation studies
Brand clustering validation
Semantic similarity assessment

Usage:

python tests/analysis/analyze_item_embeddings.py

tests/analysis/analyze_training_issues.py

Purpose: Training process diagnostic tool

Features:

Loss curve analysis
Convergence issue detection
Learning rate optimization suggestions
Training instability diagnosis

Debug Scripts

tests/debug/check_transformer_weights.py

Purpose: Weight inspection and validation

Features:

Weight distribution analysis
Initialization check
NaN/Infinity detection
Weight evolution tracking

tests/debug/debug_transformer_user_tower.py

Purpose: User tower specific debugging

Features:

Layer-by-layer output inspection
Demographic embedding analysis
Sequence processing validation
Attention mechanism debugging

tests/debug/debug_transformer_recommendation.py

Purpose: End-to-end recommendation debugging

Features:

Recommendation pipeline analysis
Similarity score debugging
Ranking quality assessment
Performance bottleneck identification

Testing Methodologies

1. Statistical Analysis

Mean, variance, distribution analysis
Hypothesis testing for significance
Confidence interval computation
Outlier detection and handling

2. Correlation Testing

Pearson correlation analysis
Cosine similarity validation
Cross-correlation studies
Alignment measurement

3. Threshold-Based Detection

Configurable anomaly thresholds
Multi-level severity classification
Adaptive threshold adjustment
Context-aware validation

4. Cross-Validation

Multiple trial consistency testing
Bootstrap sampling for robustness
Temporal stability validation
Reproducibility verification

Expected Results & Metrics

Health Score Ranges

Excellent (0.9-1.0): System performing optimally
Good (0.7-0.9): Minor issues, generally healthy
Needs Attention (0.5-0.7): Moderate issues requiring fixes
Critical (0.0-0.5): Major issues, system needs repair

Key Performance Indicators

{
  "architecture_health": 0.95,
  "embedding_quality": 0.87,
  "gradient_flow": 0.92,
  "training_stability": 0.78,
  "recommendation_quality": 0.85
}

Critical Issue Categories

Vanishing Gradients: Gradient norms < 1e-8
Embedding Collapse: Variance < 0.01 or similarity > 0.95
Architecture Issues: Dimension mismatches, broken connections
Training Instability: High loss variance, poor convergence

Usage Guidelines

Running Individual Tests

# Architecture validation
python tests/architectural_validation_tests.py

# Embedding analysis
python tests/embedding_alignment_tests.py

# Gradient flow check
python tests/gradient_flow_validation.py

Running Full Test Suite

# Complete system validation
python tests/run_all_tests.py

# Generate comprehensive report
python tests/run_all_tests.py --output-report results/full_analysis.json

Debug Mode Execution

# Debug specific components
python tests/debug/debug_transformer_user_tower.py --verbose

# Check specific weights
python tests/debug/check_transformer_weights.py --layer attention_1

Analysis Mode

# Analyze embeddings
python tests/analysis/analyze_item_embeddings.py --categories electronics,books

# Training diagnostics
python tests/analysis/analyze_training_issues.py --epochs 50

Dependencies

Core Requirements

tensorflow>=2.8.0
numpy>=1.21.0
scipy>=1.7.0
scikit-learn>=1.0.0
pandas>=1.3.0
matplotlib>=3.5.0
seaborn>=0.11.0

Model Dependencies

Custom transformer architecture modules
Recommendation engine components
Vocabulary and preprocessing utilities
Pre-trained model weights

Configuration Requirements

Consistent model parameters across tests
Artifact paths for pre-trained weights
Configurable batch sizes and test parameters
Environment variable setup for paths

Best Practices

Before Running Tests

Ensure all model artifacts are available
Verify environment configuration
Check system resources (GPU/CPU)
Review test parameters in configuration files

Interpreting Results

Focus on health scores first
Investigate critical issues immediately
Use debug scripts for detailed analysis
Compare results across multiple runs

Troubleshooting

Check dependency versions
Verify model weight compatibility
Review artifact paths
Monitor system resources during execution

Contributing to Tests

Adding New Tests

Follow existing naming conventions
Include comprehensive documentation
Implement proper error handling
Add expected result validation

Test Quality Standards

Include statistical validation
Provide clear success/failure criteria
Generate actionable recommendations
Maintain backward compatibility

Latest Test Results (2025-09-29 15:56:06)

Test Run Summary

Test Execution: Complete test suite run performed on 2025-09-29 at 15:56:06 Total Tests: 14 test scripts executed Status: ✅ All tests completed successfully Previous Run: 2025-09-29 15:46:12 (comparison available)

Key Findings

Architecture Health ✅

Total Parameters: 1,050,917 (User Tower)
Layer Distribution:
- Embedding layers: 7
- Transformer encoder: 1 (4 layers, 8 heads)
- Attention pooling: 1
- Dense layers: 3
- Layer normalization: 2

Embedding Quality Assessment ✅

Age Embedding: Shape [6,8], Norm: 0.225, Variance: 0.032 (↑ improved)
Income Embedding: Shape [5,8], Norm: 0.183, Variance: 0.029 (↑ improved)
Gender Embedding: Shape [2,8], Norm: 0.090, Variance: 0.022 (stable)
Profession Embedding: Shape [8,8], Norm: 0.249, Variance: 0.031 (↑ improved)
Location Embedding: Shape [3,8], Norm: 0.133, Variance: 0.026 (stable)
Education Embedding: Shape [5,8], Norm: 0.173, Variance: 0.027 (stable)
Marital Embedding: Shape [4,8], Norm: 0.143, Variance: 0.025 (stable)

Transformer Layer Analysis ✅

Configuration: 4 layers, 8 attention heads, 128D model dimension, 512D FFN
Weight Norms: All attention layers show healthy weight distributions (11.3-11.4)
FFN Layers: Proper initialization with norms around 14.3 (consistent)
Layer Normalization: Correctly initialized (gamma=1.0, beta=0.0)
Positional Embeddings: Norm: 4.61 (slightly improved from 4.60)

Gradient Flow Health ⚠️

Embedding Gradients: Healthy norms (0.022-0.046) (↑ slightly improved)
Attention Layers: Most layers show proper gradient flow
Warning Detected: Persistent vanishing gradients in encoder layer 0 key weights
Coldstart Path: Critical gradient flow issues detected (zero gradients)

Weight Distribution Analysis ✅

Embedding Layers: Proper variance and distribution
Attention Weights: Normal Xavier-like initialization
FFN Layers: Appropriate weight scaling
Bias Terms: Several bias layers correctly initialized to zero

Critical Issues Detected ⚠️

Vanishing Gradients in Coldstart Path
- Location: coldstart_dense_1 and coldstart_output layers
- Severity: Moderate
- Impact: May affect cold-start user recommendations
- Recommendation: Review coldstart learning rate and loss weighting
High Sparsity in Some Layers
- Location: FFN bias terms and some layer norm beta parameters
- Severity: Low (expected for bias initialization)
- Impact: Normal behavior for zero-initialized biases

Performance Metrics

Layer-by-Layer Health Scores

Embedding Layers: 95% healthy (excellent variance and distribution)
Transformer Encoder: 92% healthy (minor gradient issues in one layer)
Attention Mechanisms: 90% healthy (good weight norms and patterns)
Dense Layers: 88% healthy (some gradient flow concerns)
Output Layers: 85% healthy (normal operational parameters)

System Components Status

✅ Architecture Validation: All components properly connected
✅ Parameter Flow: Dimensions correctly aligned
✅ Model Compilation: No structural issues detected
⚠️ Gradient Flow: Minor issues in coldstart pathway
✅ Weight Initialization: Proper initialization patterns

Recommendations

Immediate Actions:
- Monitor coldstart user performance in production
- Consider adjusting coldstart learning rate (increase by 2-3x)
- Add gradient clipping if training instability occurs
Medium-term Improvements:
- Implement gradient monitoring in training loop
- Add learning rate schedules for different model components
- Consider warmup strategies for coldstart pathway
Long-term Monitoring:
- Regular gradient flow analysis during training
- Embedding quality monitoring over time
- Performance tracking for different user types

Test Files Generated (Latest Run)

transformer_analysis_20250929_155600.json - Layer analysis results
architectural_validation_20250929_155606.json - Architecture validation
comprehensive_test_report_20250929_155606.json - Complete test results
weight_init_test_20250929_155604.json - Weight initialization analysis

Test Files Generated (Previous Run)

transformer_analysis_20250929_154605.json - Layer analysis results
architectural_validation_20250929_154612.json - Architecture validation
comprehensive_test_report_20250929_154612.json - Complete test results
weight_init_test_20250929_154608.json - Weight initialization analysis

Test Comparison Summary

Embedding Variances: Generally improved across most embeddings
Gradient Flow: Similar issues persist in coldstart pathway
Architecture: Consistent structural validation
Overall Stability: High consistency between test runs

Key Test Files for Recommendation & Embedding Analysis

Recommendation Quality Tests 🎯

comprehensive_recommendation_analysis.py - Full recommendation pipeline analysis for 100 users
test_similarity_scores_100_users.py - Similarity score distributions and patterns
test_main_transformer_integration.py - End-to-end recommendation integration testing

Embedding Variance Tests 📊

test_embedding_collapse_detection.py - Variance thresholds (0.01) and collapse detection
user_embedding_generation_test.py - User embedding quality and similarity patterns
embedding_similarity_analysis.py - Advanced similarity and temporal stability analysis
embedding_alignment_tests.py - User-item embedding alignment validation
interaction_embedding_analysis.py - Interaction patterns effect on embeddings

Current Variance Status ✅

All embeddings exceed variance threshold (>0.025 vs 0.01 minimum)
No embedding collapse detected
Healthy diversity in user representations
Consistent similarity patterns across runs

Framework Migration Guide

Migrating from v1.0 to v2.0

Old Approach

# v1.0 - Individual test execution
python tests/embedding_similarity_analysis.py
python tests/architectural_validation_tests.py
python tests/run_all_tests.py

New Approach

# v2.0 - Unified advanced runner
python tests/test_runner.py --categories similarity unit
python tests/test_runner.py --parallel

Key Differences

Configuration: Centralized in tests/config/test_config.py
Base Classes: All tests should inherit from BaseTest
Execution: Parallel by default with intelligent scheduling
Reporting: Enhanced JSON reports with health scoring
Organization: Tests categorized by purpose (unit, integration, etc.)

Backward Compatibility

All existing test scripts continue to work
Legacy run_all_tests.py remains functional
Existing test result files are compatible
No breaking changes to test interfaces

Troubleshooting

Common Issues

Import Errors

ModuleNotFoundError: No module named 'src'

Solution: Use proper module execution or update PYTHONPATH

python -m tests.test_runner
# or
export PYTHONPATH="/path/to/project:$PYTHONPATH"

Parallel Execution Issues

# Reduce parallel workers
python tests/test_runner.py --sequential

# Or configure in environment
export MAX_PARALLEL_TESTS=2

Memory Issues

# Run resource-intensive tests sequentially
python tests/test_runner.py --categories performance --sequential

# Or run subset of tests
python tests/test_runner.py --categories unit similarity

Performance Optimization

Fast Testing (Development)

# Run only quick unit tests
python tests/test_runner.py --categories unit

# Skip analysis tests
python tests/test_runner.py --categories unit integration similarity

Full Validation (CI/CD)

# Complete test suite
python tests/test_runner.py

# With custom timeout and retries
python tests/test_runner.py --sequential

Last Updated: 2025-09-29 17:30:00 (Framework v2.0) Test Status: ✅ ENHANCED with parallel execution and advanced reporting Contact: Development Team Repository: Two-Tower Transformer RecSys

Testing Framework Documentation

Two-Tower Transformer Recommendation System

Table of Contents

Overview

Key Features

Framework Architecture

Core Components

Test Organization

Test Categories

🔧 Unit Tests (tests/unit/)

🔗 Integration Tests (tests/integration/)

🎯 Similarity Tests (tests/similarity/)

⚡ Performance Tests (tests/performance/)

📊 Analysis Tests (tests/analysis/)

Advanced Test Runner

Features

Execution Modes

Parallel Execution (Default)

Sequential Execution

Reporting Capabilities

Console Output

JSON Reports

Health Scoring

Configuration System

Centralized Configuration (tests/config/test_config.py)

Benefits

Core Test Scripts

1. run_all_tests.py - Master Test Orchestrator

2. architectural_validation_tests.py - Architecture Validator

3. transformer_layer_analysis.py - Layer-by-Layer Inspector

4. embedding_alignment_tests.py - Embedding Quality Validator

5. similarity_alignment_validation.py - Similarity Mechanism Tester

6. gradient_flow_validation.py - Gradient Health Monitor

7. user_embedding_generation_test.py - User Embedding Validator

8. interaction_embedding_analysis.py - Interaction Pattern Analyzer

9. test_embedding_collapse_detection.py - Collapse Detection System

10. joint_training_analysis.py - Joint Training Inspector

11. test_improved_joint_training.py - Category-Aware InfoNCE Validator ⭐ NEW

12. test_embedding_quality.py - Item Embedding Quality Validator ⭐ NEW

13. test_transformer_inference_loading.py - Inference Engine Validator ⭐ NEW

14. test_user_embeddings.py - User Embedding Quality Validator ⭐ NEW

Analysis Scripts

tests/analysis/analyze_item_embeddings.py

tests/analysis/analyze_training_issues.py

Debug Scripts

tests/debug/check_transformer_weights.py

tests/debug/debug_transformer_user_tower.py

tests/debug/debug_transformer_recommendation.py

Testing Methodologies

1. Statistical Analysis

2. Correlation Testing

3. Threshold-Based Detection

4. Cross-Validation

Expected Results & Metrics

Health Score Ranges

Key Performance Indicators

Critical Issue Categories

Usage Guidelines

Running Individual Tests

Running Full Test Suite

Debug Mode Execution

Analysis Mode

Dependencies

Core Requirements

Model Dependencies

Configuration Requirements

Best Practices

Before Running Tests

Interpreting Results

Troubleshooting

Contributing to Tests

Adding New Tests

Test Quality Standards

Latest Test Results (2025-09-29 15:56:06)

Test Run Summary

Key Findings

Architecture Health ✅

Embedding Quality Assessment ✅

Transformer Layer Analysis ✅

Gradient Flow Health ⚠️

🔧 Unit Tests (`tests/unit/`)

🔗 Integration Tests (`tests/integration/`)

🎯 Similarity Tests (`tests/similarity/`)

⚡ Performance Tests (`tests/performance/`)

📊 Analysis Tests (`tests/analysis/`)

Centralized Configuration (`tests/config/test_config.py`)