minhajHP Claude commited on
Commit
2593e90
·
1 Parent(s): 8e29046

Prepare for Hugging Face Spaces deployment

Browse files

- Update README with HF Spaces frontmatter configuration
- Configure Docker deployment with app_port: 8000
- Integrate React build serving with FastAPI backend
- Add comprehensive Docker build documentation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Files changed (3) hide show
  1. README.md +85 -584
  2. api/main.py +43 -3
  3. docker-build.md +149 -0
README.md CHANGED
@@ -1,85 +1,31 @@
1
- # Advanced Two-Tower Recommendation System
2
-
3
- A production-ready recommendation system implementation using TensorFlow with an enhanced two-tower architecture. This system provides personalized item recommendations through collaborative filtering, content-based filtering, category-boosted recommendations, and hybrid approaches, featuring advanced training strategies.
4
-
5
- ## 🎯 Project Overview
6
-
7
- This recommendation system addresses the challenge of providing personalized item recommendations at scale using modern deep learning techniques. The enhanced two-tower architecture enables efficient similarity search, real-time recommendations, and superior personalization through behavioral signal prioritization.
8
-
9
- ### Key Features
10
- - **🧠 Enhanced Two-Tower Architecture**: 128D embeddings with advanced neural networks
11
- - **📚 Advanced Training Pipeline**: Multi-phase training with curriculum learning
12
- - **⚡ Real-time Inference**: Sub-100ms recommendation serving with FAISS indexing
13
- - **🔄 Multi-strategy Recommendations**: Raw two-tower, category-boosted, content-based, and hybrid approaches
14
- - **🎪 Category-Aware Boosting**: 60/40 split between user categories and exploration
15
- - **🔍 Interactive Similar Items**: Click-to-explore with category-balanced discovery
16
- - **📊 Comprehensive Testing**: Quality metrics and performance evaluation tools
17
- - **🌐 Production Ready**: Complete FastAPI backend with React frontend
18
-
19
- ## 🏗️ Enhanced Architecture Overview
20
-
21
- ### Advanced Two-Tower Deep Learning Architecture
22
-
23
- The system implements a sophisticated two-tower neural network architecture optimized for recommendation tasks with significant improvements for better personalization and training stability.
24
-
25
- #### 1. Enhanced Item Tower 🏢
26
- - **Purpose**: Learns dense representations of items with improved discrimination
27
- - **Input Features**:
28
- - `item_id`: Unique product identifier (embedding layer)
29
- - `category_id`: Product category (embedding layer)
30
- - `brand_id`: Brand identifier (embedding layer)
31
- - `price`: Normalized price feature with projection
32
- - **Architecture Improvements**:
33
- - **128D embeddings** (upgraded from 64D) for better representation capacity
34
- - **Multi-head attention** (4 heads) for feature fusion
35
- - **Batch normalization** for training stability
36
- - **Enhanced dense layers**: [256, 128] with dropout (0.3)
37
- - **Item bias terms** for improved modeling capacity
38
- - **L2 normalization** for similarity computations
39
- - **Output**: 128-dimensional item embeddings with bias terms
40
-
41
- #### 2. Enhanced User Tower 👤
42
- - **Purpose**: Learns user preferences with behavioral focus
43
- - **Input Features**:
44
- - **Interaction History**: Up to 50 recent item embeddings with positional encoding
45
- - **Architecture Improvements**:
46
- - **128D embeddings** for enhanced representation
47
- - **Transformer attention** (8 heads) for history processing
48
- - **Positional encoding** for sequence understanding
49
- - **Learned weighted aggregation** instead of simple mean pooling
50
- - **Enhanced dense layers**: [256, 128] with batch normalization
51
- - **User bias terms** for personalization
52
- - **Output**: 128-dimensional user embeddings with bias terms
53
 
54
- #### 3. Temperature-Scaled Similarity & Contrastive Learning 🌡️
55
- - **Temperature Scaling**: Learnable parameter for improved score discrimination
56
- - **Hard Negative Mining**: Better training signal through difficult negative examples
57
- - **Contrastive Loss**: Prevents embedding collapse and improves representation quality
58
- - **Focal Loss**: Handles imbalanced data more effectively
59
 
60
- #### 4. Curriculum Learning Strategy 🎓
61
- - **Progressive Training**: 3-stage curriculum based on interaction complexity
62
- - **Stage 1**: Simple cases (short/no history) - 33rd percentile
63
- - **Stage 2**: Medium complexity (moderate history) - 33rd-67th percentile
64
- - **Stage 3**: Complex cases (long history) - 67th+ percentile
65
- - **Adaptive Learning Rates**: Decrease as stages progress for stability
66
 
67
- ### 5. Aggregated History Content-Based Filtering 🔄
68
- - **Revolutionary Approach**: Uses aggregated user interaction history instead of single-item similarity
69
- - **Multiple Aggregation Methods**:
70
- - **Weighted Mean**: Recent interactions weighted higher (exponential decay)
71
- - **Simple Mean**: Equal weighting of all interactions
72
- - **Max Pooling**: Element-wise maximum of embeddings
73
- - **ANN Search**: Direct similarity search using FAISS with aggregated user profile
74
- - **Enhanced Personalization**: Captures complete user preference profile, not just recent item
75
- - **Category-Aware**: Analyzes user's full category distribution for balanced recommendations
76
 
77
- ### 6. Category-Aware Recommendation Engine 🎪
78
- - **Enhanced Hybrid Recommendations**: Category boosting based on user preferences
79
- - **Category Alignment Analysis**: Measures personalization effectiveness
80
- - **Diversity Controls**: Balanced category representation in recommendations
81
- - **Subcategory Precision**: 2-level category matching (e.g., "computers.components")
82
- - **Comprehensive Analysis Tools**: Multi-algorithm comparison and alignment scoring
 
 
 
 
 
83
 
84
  ## 📁 Project Structure
85
 
@@ -94,151 +40,38 @@ RecSys-HP/
94
  │ ├── models/ # Neural network architectures
95
  │ │ ├── item_tower.py # Original item embedding tower
96
  │ │ ├── user_tower.py # User embedding tower
97
- │ │ ├── enhanced_two_tower.py # Enhanced two-tower architecture
98
  │ │ └── improved_two_tower.py # Advanced two-tower with improvements
99
  │ │
100
  │ ├── preprocessing/ # Data preparation pipeline
101
  │ │ ├── data_loader.py # Dataset loading and validation
102
- │ │ ── user_data_preparation.py # User feature engineering with categorization
103
- │ │ └── optimized_dataset_creator.py # Optimized data processing
104
- │ │
105
- │ ├── training/ # Training pipelines
106
- │ │ ├── item_pretraining.py # Phase 1: Item tower pre-training
107
- │ │ ├── joint_training.py # Original joint training
108
- │ │ ├── optimized_joint_training.py # Performance-optimized training
109
- │ │ ├── fast_joint_training.py # Fast joint training implementation
110
- │ │ ├── improved_joint_training.py # Advanced joint training with curriculum learning
111
- │ │ └── curriculum_trainer.py # Advanced curriculum learning
112
  │ │
113
- │ ├── inference/ # Production serving components
114
- │ │ ├── faiss_index.py # Vector similarity search
115
- │ │ ── recommendation_engine.py # Core inference pipeline
116
- │ │ └── enhanced_recommendation_engine_128d.py # 128D enhanced recommendations
117
  │ │
118
- │ ├── utils/ # Utility functions
119
- │ │ ── real_user_selector.py # Real user data selection for testing
 
120
  │ │
121
- │ └── artifacts/ # Model checkpoints and metadata
122
- │ ├── *.data-* / *.index # TensorFlow model weights
123
- │ ├── *.npy # Numpy arrays (embeddings)
124
- ── *.pkl # Pickled features/vocabularies
125
- │ └── *.bin # FAISS indices
126
 
127
- ├── 🌐 API Implementations # Multiple API options
128
- │ ├── api/main.py # Primary FastAPI server
129
- │ ├── api_2phase.py # 2-phase training API
130
- │ └── api_joint.py # Joint training API
 
131
 
132
- ���💻 frontend/ # Interactive web interface
133
- ── src/ # React components
134
- │ │ ├── App.js # Enhanced application
135
- │ │ └── *.css # Updated styling
136
- │ ├── public/ # Static assets
137
- │ └── package.json # Node.js dependencies
138
 
139
- ── 🚀 Training Scripts # Multiple training approaches
140
- ├── run_training_pipeline.py # Main training orchestration
141
- ├── run_2phase_training.py # 2-phase training approach
142
- └── run_joint_training.py # Joint training approach
143
-
144
- └── 📋 requirements.txt # Python dependencies
145
- ```
146
-
147
- ## 🚀 Quick Start Guide
148
-
149
- ### Prerequisites
150
- - **Python 3.8+** (3.9+ recommended)
151
- - **Node.js 16+** & npm
152
- - **TensorFlow 2.13+** (GPU version recommended)
153
- - **8GB+ RAM** (for training phase)
154
- - **5GB+ disk space** (for models and indices)
155
- - **CUDA 11.8+** (optional, for GPU acceleration)
156
-
157
- ### 1. Environment Setup
158
-
159
- #### Prerequisites
160
- - **Python 3.8+** (3.9+ recommended)
161
- - **Node.js 16+** and npm
162
- - **Git** for version control
163
- - **8GB+ RAM** (for model training)
164
- - **GPU recommended** (optional, for faster training)
165
-
166
- #### Installation Steps
167
- ```bash
168
- # Clone the repository
169
- git clone [repository-url]
170
- cd RecSys-HP
171
-
172
- # Create and activate virtual environment
173
- python -m venv env
174
- source env/bin/activate # Windows: env\Scripts\activate
175
-
176
- # Upgrade pip and install Python dependencies
177
- pip install --upgrade pip
178
- pip install -r requirements.txt
179
-
180
- # For GPU support (optional but recommended):
181
- # pip install tensorflow-gpu==2.13.0
182
-
183
- # Install React frontend dependencies
184
- cd frontend && npm install && cd ..
185
- ```
186
-
187
- #### Dataset Setup
188
- ```bash
189
- # Ensure your datasets are properly placed:
190
- # datasets/users.csv - User profiles
191
- # datasets/interactions.csv - User-item interaction data
192
- # datasets/items.csv - Item features and metadata
193
-
194
- # Verify dataset structure
195
- python -c "from src.preprocessing.data_loader import DataProcessor; dp = DataProcessor(); print('✅ Datasets loaded successfully')"
196
- ```
197
-
198
- ### 2. Training Options
199
-
200
- Choose from multiple training approaches based on your needs:
201
-
202
- #### Option A: Main Training Pipeline (Recommended) 🌟
203
- ```bash
204
- # Complete end-to-end training pipeline (20-30 minutes)
205
- python scripts/run_training_pipeline.py
206
- ```
207
-
208
- #### Option B: 2-Phase Training Approach
209
- ```bash
210
- # 2-phase training: item pretraining + joint optimization
211
- python scripts/run_2phase_training.py
212
- ```
213
-
214
- #### Option C: Joint Training Approach
215
- ```bash
216
- # Direct joint training of both towers
217
- python scripts/run_joint_training.py
218
- ```
219
-
220
- **Enhanced Training Features:**
221
- ### 3. Start Interactive Demo 🎮
222
-
223
- #### Launch the Application
224
- ```bash
225
- # Launch API server (in one terminal)
226
- cd api && python main.py
227
-
228
- # Start React frontend (in another terminal)
229
- cd frontend && npm start
230
- ```
231
-
232
- **Access Points:**
233
- - 🌐 **Frontend Demo**: http://localhost:3000
234
- - 📚 **API Documentation**: http://localhost:8000/docs
235
- - 🔧 **API Health Check**: http://localhost:8000/health
236
- - ⚡ **Real-time Recommendations**: Interactive similarity search with 60/40 category balancing
237
-
238
- ### 4. Quality Analysis 📊
239
- ```bash
240
- # Run comprehensive recommendation analysis
241
- python analyze_recommendations.py
242
  ```
243
 
244
  ## 🎯 Recommendation Strategies
@@ -248,24 +81,22 @@ python analyze_recommendations.py
248
  - **Strengths**: Superior personalization with behavioral signal focus
249
  - **Algorithm**: Two-tower neural collaborative filtering with category awareness
250
 
251
- ### 2. Category-Boosted Recommendations 🎪
252
- - **Method**: Strict 60/40 split between user categories and exploration
253
- - **Features**: Proportional distribution within user categories, parent category fallback
254
- - **Benefits**: Balanced personalization with controlled exploration
255
- - **Algorithm**: Category-aware recommendation with boost factor (1.3x)
256
-
257
- ### 3. Content-Based Filtering 📊
258
- - **Method**: Item feature similarity and aggregated user history
259
- - **Features**: FAISS-based embedding similarity, category constraints
260
- - **Strengths**: Better cold-start performance, explainable recommendations
261
- - **Algorithm**: Enhanced embedding similarity with category balancing
262
 
263
- ### 4. Hybrid Approach 🔗
264
  - **Method**: Weighted combination of collaborative and content-based
265
  - **Features**: Configurable weight mixing (default 70% collaborative, 30% content)
266
  - **Benefits**: Best of both approaches with balanced coverage
267
  - **Algorithm**: Score-based weighted combination
268
 
 
 
 
 
 
269
  ## 🔬 Technical Deep Dive
270
 
271
  ### Enhanced Training Process
@@ -274,291 +105,47 @@ python analyze_recommendations.py
274
  - **Stage 1**: Simple cases (short interaction history)
275
  - **Stage 2**: Medium complexity (moderate history)
276
  - **Stage 3**: Complex cases (long interaction history)
277
- - **Progressive Difficulty**: Gradually increase learning complexity
278
- - **Adaptive Learning**: Decay learning rate between stages
279
-
280
- #### Performance Improvements
281
- - **Score Discrimination**: 15x improvement in variance (0.0007 → 0.01+)
282
- - **Category Alignment**: 5x improvement (12% → 60%+)
283
- - **Embedding Quality**: Reduced collapse, better user diversity
284
- - **Training Stability**: Curriculum learning + batch normalization
285
-
286
- ### Advanced Features
287
-
288
- #### Temperature Scaling
289
- - **Purpose**: Improve score discrimination and ranking quality
290
- - **Implementation**: Learnable parameter in similarity computation
291
- - **Benefits**: Better separation between relevant/irrelevant items
292
 
293
- #### Hard Negative Mining
294
- - **Purpose**: Improve contrastive learning signal
295
- - **Method**: Select hardest negatives (highest similarity among negatives)
296
- - **Benefits**: Better embedding separation, reduced collapse
297
 
298
- #### Category-Aware Boosting
299
- - **Analysis**: User category preference extraction from history
300
- - **Boosting**: Amplify scores for items matching user preferences
301
- - **Diversity**: Balance personalization with exploration
302
 
303
- ## 📈 Performance Metrics & Analysis
304
 
305
- ### Quality Metrics
306
- - **Score Variance**: Measures recommendation discrimination ability
307
- - **Category Alignment**: Percentage of recommendations matching user preferences
308
- - **Embedding Collapse**: User-user similarity analysis
309
- - **Recommendation Speed**: Inference time per user
310
- - **Training Convergence**: Loss curves and validation metrics
311
 
312
- ### Expected Performance
313
- - **Score Discrimination**: 15x improvement with enhanced model
314
- - **Category Alignment**: 5x improvement (12% 60%+)
315
- - **Inference Speed**: <50ms per recommendation request
316
- - **Training Time**: 45-60 minutes with curriculum learning
317
- - **Memory Usage**: ~6GB during training, ~2GB serving
318
 
319
- ## 🔗 Enhanced API Endpoints
320
-
321
- ### Core Recommendation Endpoints
322
 
323
  | Method | Endpoint | Description | Features |
324
  |--------|----------|-------------|----------|
325
- | `GET` | `/` | Root endpoint | API information and status |
326
- | `GET` | `/health` | Health check | Service availability status |
327
- | `POST` | `/recommendations` | Personalized recommendations | Multi-strategy (collaborative/content/hybrid/enhanced) |
328
  | `POST` | `/item-similarity` | Category-balanced similar items | 60% same category + ANN search |
329
- | `POST` | `/predict-rating` | User-item rating prediction | Two-tower model predictions |
330
-
331
- ### Data & User Endpoints
332
-
333
- | Method | Endpoint | Description | Features |
334
- |--------|----------|-------------|----------|
335
- | `GET` | `/real-users/{user_id}` | Detailed user timeline | Complete interaction breakdown |
336
- | `GET` | `/behavioral-patterns` | Enriched behavioral patterns | Pre-populated item details |
337
- | `GET` | `/dataset-summary` | Dataset statistics | User/item/interaction counts |
338
- | `GET` | `/items/{item_id}` | Individual item info | Brand, category, price details |
339
- | `GET` | `/items` | Sample items | Testing and exploration |
340
-
341
- ### Example Enhanced API Usage
342
-
343
- ```python
344
- import requests
345
-
346
- # Get category-aware enhanced recommendations
347
- response = requests.post("http://localhost:8000/enhanced-recommendations", json={
348
- "user_profile": {
349
- "interaction_history": [1001, 1515, 2023, 4042]
350
- },
351
- "num_recommendations": 10,
352
- "recommendation_type": "enhanced_hybrid", # New enhanced strategy
353
- "category_boost": 1.5, # Category preference amplification
354
- "enable_diversity": True, # Balanced category representation
355
- "max_per_category": 3 # Diversity control
356
- })
357
-
358
- recommendations = response.json()
359
- # Returns enhanced recommendations with category analysis and explanations
360
- ```
361
-
362
- ### 🎯 Interactive Similar Items Feature
363
-
364
- The system now features an advanced similar items discovery interface with intelligent category balancing:
365
-
366
- #### **Click-to-Explore Functionality**
367
- - **Interactive Cards**: Click any recommendation to discover similar items
368
- - **Smart Category Balance**: 60% same category (high relevance) + 40% different categories (discovery)
369
- - **ANN-Powered**: Uses FAISS similarity search with cosine similarity scores
370
- - **Visual Indicators**: Similarity percentage badges and progress bars
371
- - **Rich Details**: Complete item information (brand, category, price)
372
-
373
- #### **Category-Balanced Algorithm**
374
- ```python
375
- # Example: Clicking on iPhone (electronics.smartphone)
376
- POST /item-similarity
377
- {
378
- "item_id": 1004565,
379
- "num_recommendations": 10
380
- }
381
-
382
- # Returns:
383
- # 60% smartphones: Samsung, Huawei, Xiaomi... (high relevance)
384
- # 40% related items: iPad, MacBook, AirPods... (discovery)
385
- # All items ranked by actual FAISS similarity scores
386
- ```
387
-
388
- #### **Similar Items API Response**
389
- ```json
390
- [
391
- {
392
- "item_id": 1003907,
393
- "score": 0.9289, // 92.9% similarity
394
- "item_info": {
395
- "brand": "huawei",
396
- "category_code": "electronics.smartphone",
397
- "price": 151.87
398
- }
399
- }
400
- ]
401
- ```
402
-
403
- **Frontend Demo**: Visit http://localhost:3000 → Get recommendations → Click any item → Explore similar products with category insights!
404
-
405
- ## 🛠️ Development & Testing
406
-
407
- ### 🚀 Training Pipeline Options
408
 
409
- #### **Complete Pipeline (Recommended)**
410
- ```bash
411
- # Run full training pipeline (item pretraining + joint training + FAISS indexing)
412
- python run_training_pipeline.py
413
-
414
- # Alternative: Run individual steps
415
- python run_2phase_training.py # 2-phase approach
416
- python run_joint_training.py # End-to-end joint training
417
- python train_improved_model.py # Enhanced model training
418
- ```
419
-
420
- #### **Multiple API Servers Available**
421
- ```bash
422
- # Main API (production-ready with all features)
423
- cd api && python main.py
424
-
425
- # 2-Phase Model API (comparison/testing)
426
- python api_2phase.py
427
-
428
- # Joint Training Model API (alternative approach)
429
- python api_joint.py
430
- ```
431
-
432
- ### 🔧 System Testing
433
- ```bash
434
- # Test core system components
435
- python -m src.utils.real_user_selector # Demo real user extraction
436
- python -m src.preprocessing.data_loader # Verify data loading
437
- ```
438
-
439
- ### 🧪 Frontend Development
440
- ```bash
441
- cd frontend
442
- npm install # Install dependencies
443
- npm start # Development server (localhost:3000)
444
- npm run build # Production build
445
- npm test # Run tests
446
- ```
447
-
448
- ### Model Training Options
449
- ```bash
450
- # Original training pipeline
451
- python run_training_pipeline.py
452
-
453
- python train_improved_model.py --embedding-dim 128
454
-
455
- # Curriculum learning with custom stages
456
- python train_improved_model.py --curriculum-stages 4 --epochs-per-stage 12
457
- ```
458
-
459
- ## 📁 Complete Project Structure
460
-
461
- ```
462
- RecSys-HP/
463
- ├── 🚀 API Services
464
- │ ├── api/
465
- │ │ └── main.py # Main production API (all features)
466
- │ ├── api_2phase.py # 2-phase model API (testing)
467
- │ └── api_joint.py # Joint training model API
468
-
469
- ├── 🧠 Machine Learning Core
470
- │ └── src/
471
- │ ├── models/ # Neural Network Architectures
472
- │ │ ├── enhanced_two_tower.py # 128D enhanced architecture
473
- │ │ ├── improved_two_tower.py # Standard enhanced model
474
- │ │ ├── item_tower.py # Item embedding tower
475
- │ │ └── user_tower.py # User embedding tower
476
- │ │
477
- │ ├── inference/ # Trained Model Serving
478
- │ │ ├── enhanced_recommendation_engine_128d.py # 128D inference engine
479
- │ │ ├── enhanced_recommendation_engine.py # Enhanced inference
480
- │ │ ├── recommendation_engine.py # Basic inference
481
- │ │ └── faiss_index.py # ANN similarity search
482
- │ │
483
- │ ├── training/ # Model Training Pipeline
484
- │ │ ├── curriculum_trainer.py # Progressive learning
485
- │ │ ├── improved_joint_training.py # Enhanced joint training
486
- │ │ ├── optimized_joint_training.py # Performance optimized
487
- │ │ ├── fast_joint_training.py # Speed optimized
488
- │ │ ├── joint_training.py # Standard joint training
489
- │ │ └── item_pretraining.py # Item tower pretraining
490
- │ │
491
- │ ├── preprocessing/ # Data Processing
492
- │ │ ├── data_loader.py # Main data processor
493
- │ │ ├── optimized_dataset_creator.py # Efficient dataset creation
494
- │ │ └── user_data_preparation.py # User feature processing
495
- │ │
496
- │ ├── utils/ # Utility Functions
497
- │ │ └── real_user_selector.py # Real user data extraction
498
- │ │
499
- │ └── artifacts/ # Trained Models & Data
500
- │ ├── *.pkl # Vocabularies & features
501
- │ ├── *_weights.* # TensorFlow model weights
502
- │ ├── faiss_* # FAISS indices & embeddings
503
- │ └── *.txt # Configuration files
504
-
505
- ├── 🌐 Frontend Interface
506
- │ └── frontend/
507
- │ ├── src/
508
- │ │ ├── App.js # Main React component
509
- │ │ ├── App.css # Styling & animations
510
- │ │ ├── index.js # React entry point
511
- │ │ └── index.css # Global styles
512
- │ ├── public/ # Static assets
513
- │ ├── package.json # Dependencies & scripts
514
- │ └── build/ # Production build
515
-
516
- ├── 🎯 Training Scripts
517
- │ ├── run_training_pipeline.py # Complete training pipeline
518
- │ ├── run_2phase_training.py # 2-phase approach
519
- │ ├── run_joint_training.py # End-to-end training
520
- │ └── train_improved_model.py # Enhanced model training
521
-
522
- ├── 📊 Analysis & Testing
523
- │ ├── analyze_recommendations.py # Quality analysis tool
524
- │ ├── recommendation_analysis_report.md # Generated analysis report
525
- │ └── recommendation_analysis_plots.png # Analysis visualizations
526
-
527
- ├── 📚 Data & Configuration
528
- │ ├── datasets/ # Training data
529
- │ │ ├── items.csv # Product catalog
530
- │ │ └── interactions.csv # User-item interactions
531
- │ ├── requirements.txt # Python dependencies
532
- │ ├── README.md # Project documentation
533
- │ └── ARCHITECTURE.md # Technical architecture
534
- ```
535
-
536
- ### 🔧 Key Components Explained
537
-
538
- #### **🚀 Multiple API Options**
539
- - **`api/main.py`**: Production API with all features (similar items, real users, behavioral patterns)
540
- - **`api_2phase.py`**: Serves 2-phase trained models for comparison
541
- - **`api_joint.py`**: Serves joint-trained models for testing
542
-
543
- #### **🧠 Three Inference Engines**
544
- - **Enhanced 128D**: Best performance, advanced features, 128D embeddings
545
- - **Enhanced Standard**: Good performance, 64D embeddings, category boosting
546
- - **Basic Engine**: Simple collaborative/content/hybrid recommendations
547
-
548
- #### **⚡ Training Pipeline Flexibility**
549
- - **Complete Pipeline**: Full training workflow (pretraining → joint → FAISS)
550
- - **2-Phase Training**: Item pretraining + joint fine-tuning
551
- - **Joint Training**: End-to-end optimization
552
 
553
- #### **🎨 Frontend Features**
554
- - **Real User Interface**: Browse genuine user profiles & interaction histories
555
- - **Interactive Recommendations**: Click any item → see similar products (60/40 category split)
556
- - **Category Analysis**: Visual breakdown of user interests vs recommendations
557
- - **Performance Monitoring**: Real-time API performance metrics
 
558
 
559
  ## 🔧 Advanced Configuration
560
 
561
- ### Enhanced Model Hyperparameters
562
  - **Embedding Dimension**: 128 (upgraded from 64)
563
  - **Hidden Layers**: [256, 128] for both towers
564
  - **Dropout Rate**: 0.3 (increased for regularization)
@@ -576,96 +163,10 @@ RecSys-HP/
576
 
577
  ## 🚀 Production Deployment
578
 
579
- ### Enhanced Infrastructure Requirements
580
- - **CPU**: 6+ cores for training, 4+ cores for serving
581
- - **Memory**: 12GB training, 4GB serving (increased for 128D embeddings)
582
- - **Storage**: 8GB for enhanced models and indices
583
- - **GPU**: Optional, provides 2-3x training speedup
584
-
585
- ### Scaling Features
586
- - **Categorical Processing**: Efficient embedding lookups
587
  - **FAISS Integration**: Sub-linear similarity search
588
  - **Batch Inference**: Vectorized computation for multiple users
589
  - **Model Versioning**: Support for A/B testing different model variants
590
 
591
- ---
592
-
593
- ## 📊 Project Achievements
594
-
595
- ✅ **Enhanced Architecture**: 128D embeddings, temperature scaling, contrastive learning
596
- ✅ **Curriculum Learning**: Progressive training for better convergence
597
- ✅ **Category-Aware Recommendations**: Intelligent personalization with diversity
598
- ✅ **Aggregated Content-Based Filtering**: Revolutionary user history aggregation approach
599
- ✅ **Enhanced Cold-Start Support**: Improved new user handling
600
- ✅ **Production Ready**: Scalable API with enhanced frontend features
601
-
602
- **🎉 Ready to deliver next-generation personalized recommendations!**
603
-
604
- ## 🗂️ Available Training Approaches
605
-
606
- This project provides multiple training strategies:
607
-
608
- 1. **Main Pipeline** (`run_training_pipeline.py`) - Complete orchestrated training
609
- 2. **2-Phase Training** (`run_2phase_training.py`) - Item pretraining + joint optimization
610
- 3. **Joint Training** (`run_joint_training.py`) - Direct joint training approach
611
- 4. **Enhanced Training** (`train_improved_model.py`) - Advanced features with curriculum learning
612
-
613
- ## 🔌 API Options
614
-
615
- - **Primary API** (`api/main.py`) - Full-featured FastAPI server
616
- - **2-Phase API** (`api_2phase.py`) - Specialized for 2-phase training
617
- - **Joint API** (`api_joint.py`) - Optimized for joint training approach
618
-
619
- ## 🔧 Development Tools
620
-
621
- - **Real User Selection** (`src.utils.real_user_selector`) - Extract real user profiles for testing
622
- - **Data Loading Utilities** (`src.preprocessing.data_loader`) - Dataset loading and validation
623
-
624
- ## 🧪 Development & Testing
625
-
626
- ### Frontend Development
627
- ```bash
628
- # Start development server with hot reload
629
- cd frontend && npm start
630
-
631
- # Build production bundle
632
- npm run build
633
-
634
- # Run frontend tests
635
- npm test
636
- ```
637
-
638
- ### Backend Testing
639
- ```bash
640
- # Test API endpoints
641
- python -m pytest tests/
642
-
643
- # Manual API testing
644
- curl http://localhost:8000/health
645
- curl http://localhost:8000/model-info
646
- ```
647
-
648
- ### Troubleshooting
649
-
650
- #### Common Issues
651
- 1. **TensorFlow GPU Issues**: Ensure CUDA 11.8+ and cuDNN are installed
652
- 2. **Memory Errors**: Reduce batch size in training scripts
653
- 3. **Port Conflicts**: Change API port in main.py if 8000 is occupied
654
- 4. **Dataset Loading**: Verify CSV files are in correct format and location
655
-
656
- #### Performance Optimization
657
- - Use GPU training for 3-5x speedup
658
- - Increase batch size for better GPU utilization
659
- - Enable mixed precision training for memory efficiency
660
-
661
- ## 📞 Support & Contributing
662
-
663
- For questions, issues, or contributions:
664
- - 🐛 **Report bugs**: Create an issue with detailed reproduction steps
665
- - 💡 **Feature requests**: Describe the enhancement and use case
666
- - 🔧 **Pull requests**: Follow the existing code style and add tests
667
- - 📚 **Documentation**: Help improve setup guides and API docs
668
-
669
- ---
670
-
671
- **Built with ❤️ using TensorFlow, React, and FastAPI**
 
1
+ ---
2
+ title: RecSys-HP
3
+ emoji: 🎯
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: docker
7
+ pinned: false
8
+ license: mit
9
+ app_port: 8000
10
+ ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
12
+ # RecSys-HP: Two-Tower Recommendation System
 
 
 
 
13
 
14
+ A production-ready recommendation system implementation using TensorFlow with an enhanced two-tower architecture. This system provides personalized item recommendations through collaborative filtering, content-based filtering, category-boosted recommendations, and hybrid approaches, featuring advanced training strategies.
 
 
 
 
 
15
 
16
+ ## 🚀 Features
 
 
 
 
 
 
 
 
17
 
18
+ - **🏗️ Enhanced Two-Tower Architecture**: 128D embeddings with temperature scaling and attention mechanisms
19
+ - **🎯 Multiple Recommendation Engines**:
20
+ - Raw Two-Tower (Collaborative Filtering)
21
+ - Content-Based Filtering
22
+ - Hybrid Recommendations
23
+ - Category-Boosted Recommendations
24
+ - **⚡ Fast Inference**: FAISS-powered similarity search with sub-100ms response times
25
+ - **🎨 Interactive Frontend**: React-based web interface with real-time recommendations
26
+ - **📊 Category Analysis**: Intelligent category preference analysis and visualization
27
+ - **🔄 Real User Profiles**: Browse genuine user interaction histories
28
+ - **🎪 Category-Aware Similarity**: 60/40 category split for balanced discovery
29
 
30
  ## 📁 Project Structure
31
 
 
40
  │ ├── models/ # Neural network architectures
41
  │ │ ├── item_tower.py # Original item embedding tower
42
  │ │ ├── user_tower.py # User embedding tower
 
43
  │ │ └── improved_two_tower.py # Advanced two-tower with improvements
44
  │ │
45
  │ ├── preprocessing/ # Data preparation pipeline
46
  │ │ ├── data_loader.py # Dataset loading and validation
47
+ │ │ ── user_data_preparation.py # User feature engineering
 
 
 
 
 
 
 
 
 
48
  │ │
49
+ │ ├── training/ # Model training pipeline
50
+ │ │ ├── item_pretraining.py # Item tower pretraining
51
+ │ │ ── joint_training.py # Joint user-item training
 
52
  │ │
53
+ │ ├── inference/ # Recommendation engines
54
+ │ │ ── recommendation_engine.py # Main recommendation engine
55
+ │ │ └── faiss_index.py # FAISS similarity search
56
  │ │
57
+ │ └── artifacts/ # Trained models & indices
58
+ │ ├── vocabularies.pkl # Feature vocabularies
59
+ │ ├── *_weights.* # Model weights
60
+ ── faiss_* # FAISS index files
 
61
 
62
+ ├── 🎨 frontend/ # React web interface
63
+ │ ├── src/
64
+ ├── App.js # Main React component
65
+ └── App.css # Styling
66
+ │ └── build/ # Production build
67
 
68
+ ├─ 🔗 api/ # FastAPI backend
69
+ ── main.py # API server with static file serving
 
 
 
 
70
 
71
+ ── 📚 Configuration
72
+ ├── requirements.txt # Python dependencies
73
+ ├── Dockerfile # Container configuration
74
+ └── docker-build.md # Deployment guide
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  ```
76
 
77
  ## 🎯 Recommendation Strategies
 
81
  - **Strengths**: Superior personalization with behavioral signal focus
82
  - **Algorithm**: Two-tower neural collaborative filtering with category awareness
83
 
84
+ ### 2. Content-Based Recommendations 📋
85
+ - **Method**: Aggregated user history embedding with weighted mean pooling
86
+ - **Features**: FAISS similarity search on aggregated user preferences
87
+ - **Benefits**: Works for users with interaction history, fast inference
 
 
 
 
 
 
 
88
 
89
+ ### 3. Hybrid Approach 🔗
90
  - **Method**: Weighted combination of collaborative and content-based
91
  - **Features**: Configurable weight mixing (default 70% collaborative, 30% content)
92
  - **Benefits**: Best of both approaches with balanced coverage
93
  - **Algorithm**: Score-based weighted combination
94
 
95
+ ### 4. Category-Boosted Recommendations 🎪
96
+ - **Method**: Intelligent category preference learning and boosting
97
+ - **Features**: Dynamic category analysis from user interaction patterns
98
+ - **Benefits**: Maintains user preferences while enabling discovery
99
+
100
  ## 🔬 Technical Deep Dive
101
 
102
  ### Enhanced Training Process
 
105
  - **Stage 1**: Simple cases (short interaction history)
106
  - **Stage 2**: Medium complexity (moderate history)
107
  - **Stage 3**: Complex cases (long interaction history)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
 
109
+ #### Two-Phase Training Strategy
110
+ 1. **Item Pretraining**: Self-supervised learning on item features
111
+ 2. **Joint Training**: User-item interaction learning with contrastive loss
 
112
 
113
+ ### Architecture Improvements
114
+ - **User Tower**: Demographics + 50-slot interaction history with attention
115
+ - **Item Tower**: Optimized embeddings with smart dimensionality
116
+ - **Training**: Contrastive learning with positive/negative pairs
117
 
118
+ ## 🚀 Getting Started
119
 
120
+ The application runs automatically in this Hugging Face Space! The system includes:
 
 
 
 
 
121
 
122
+ - **Interactive Web Interface**: Browse users, generate recommendations, analyze categories
123
+ - **Multiple Recommendation Types**: Try different algorithms
124
+ - **Real User Data**: Explore genuine user interaction patterns
125
+ - **Performance Monitoring**: Real-time API response tracking
 
 
126
 
127
+ ### API Endpoints
 
 
128
 
129
  | Method | Endpoint | Description | Features |
130
  |--------|----------|-------------|----------|
131
+ | `GET` | `/` | Web Interface | Interactive React app |
132
+ | `POST` | `/recommendations` | Personalized recommendations | Multi-strategy (collaborative/content/hybrid) |
 
133
  | `POST` | `/item-similarity` | Category-balanced similar items | 60% same category + ANN search |
134
+ | `GET` | `/real-users` | Browse real user profiles | Genuine interaction histories |
135
+ | `GET` | `/health` | System health check | API status monitoring |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
 
137
+ ## 📊 Project Achievements
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
138
 
139
+ **Enhanced Architecture**: 128D embeddings, temperature scaling, contrastive learning
140
+ **Curriculum Learning**: Progressive training for better convergence
141
+ **Category-Aware Recommendations**: Intelligent personalization with diversity
142
+ **Content-Based Filtering**: Revolutionary user history aggregation approach
143
+ **Enhanced Cold-Start Support**: Improved new user handling
144
+ ✅ **Production Ready**: Scalable API with enhanced frontend features
145
 
146
  ## 🔧 Advanced Configuration
147
 
148
+ ### Model Parameters
149
  - **Embedding Dimension**: 128 (upgraded from 64)
150
  - **Hidden Layers**: [256, 128] for both towers
151
  - **Dropout Rate**: 0.3 (increased for regularization)
 
163
 
164
  ## 🚀 Production Deployment
165
 
166
+ ### Performance Optimizations
167
+ - **Two-Tower Architecture**: Separates user and item processing for scalability
 
 
 
 
 
 
168
  - **FAISS Integration**: Sub-linear similarity search
169
  - **Batch Inference**: Vectorized computation for multiple users
170
  - **Model Versioning**: Support for A/B testing different model variants
171
 
172
+ **🎉 Ready to deliver next-generation personalized recommendations!**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
api/main.py CHANGED
@@ -1,5 +1,7 @@
1
  from fastapi import FastAPI, HTTPException
2
  from fastapi.middleware.cors import CORSMiddleware
 
 
3
  from pydantic import BaseModel
4
  from typing import List, Optional, Dict, Any
5
  import uvicorn
@@ -205,9 +207,9 @@ async def startup_event():
205
  real_user_selector = None
206
 
207
 
208
- @app.get("/")
209
- async def root():
210
- """Root endpoint with API information."""
211
  return {
212
  "message": "Two-Tower Recommendation API",
213
  "version": "1.0.0",
@@ -611,6 +613,44 @@ async def get_sample_items(limit: int = 20):
611
  raise HTTPException(status_code=500, detail=f"Error retrieving sample items: {str(e)}")
612
 
613
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
614
  if __name__ == "__main__":
615
  uvicorn.run(
616
  "main:app",
 
1
  from fastapi import FastAPI, HTTPException
2
  from fastapi.middleware.cors import CORSMiddleware
3
+ from fastapi.staticfiles import StaticFiles
4
+ from fastapi.responses import FileResponse
5
  from pydantic import BaseModel
6
  from typing import List, Optional, Dict, Any
7
  import uvicorn
 
207
  real_user_selector = None
208
 
209
 
210
+ @app.get("/api")
211
+ async def api_info():
212
+ """API information endpoint."""
213
  return {
214
  "message": "Two-Tower Recommendation API",
215
  "version": "1.0.0",
 
613
  raise HTTPException(status_code=500, detail=f"Error retrieving sample items: {str(e)}")
614
 
615
 
616
+ # Mount static files for React build - MUST be at the end
617
+ frontend_build_path = os.path.join(parent_dir, "frontend", "build")
618
+ if os.path.exists(frontend_build_path):
619
+ # Serve static files (JS, CSS, images, etc.)
620
+ app.mount("/static", StaticFiles(directory=os.path.join(frontend_build_path, "static")), name="static")
621
+
622
+ # Add a specific root route for React app
623
+ @app.get("/", include_in_schema=False)
624
+ async def serve_react_root():
625
+ """Serve React app at root route."""
626
+ frontend_build_path = os.path.join(parent_dir, "frontend", "build")
627
+ index_file = os.path.join(frontend_build_path, "index.html")
628
+ if os.path.exists(index_file):
629
+ return FileResponse(index_file)
630
+ else:
631
+ return {"message": "React build not found. Run 'npm run build' in frontend directory."}
632
+
633
+ # Catch-all route for React Router - MUST be at the very end
634
+ @app.get("/{full_path:path}", include_in_schema=False)
635
+ async def serve_react_app(full_path: str):
636
+ """Serve React app for all non-API routes."""
637
+ # If it's a known API route, let FastAPI handle the 404
638
+ if (full_path.startswith("api/") or
639
+ full_path.startswith("docs") or
640
+ full_path.startswith("redoc") or
641
+ full_path.startswith("openapi.json") or
642
+ full_path in ["health", "real-users", "dataset-summary", "behavioral-patterns", "recommendations", "item-similarity", "predict-rating", "items"]):
643
+ raise HTTPException(status_code=404, detail="API endpoint not found")
644
+
645
+ # For all other routes, serve the React app
646
+ frontend_build_path = os.path.join(parent_dir, "frontend", "build")
647
+ index_file = os.path.join(frontend_build_path, "index.html")
648
+ if os.path.exists(index_file):
649
+ return FileResponse(index_file)
650
+ else:
651
+ raise HTTPException(status_code=404, detail="React build not found")
652
+
653
+
654
  if __name__ == "__main__":
655
  uvicorn.run(
656
  "main:app",
docker-build.md ADDED
@@ -0,0 +1,149 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Docker Build & Run Instructions
2
+
3
+ ## 🐳 Docker Setup for RecSys-HP
4
+
5
+ This guide explains how to build and run the RecSys-HP recommendation system in a Docker container.
6
+
7
+ ### Prerequisites
8
+ - Docker installed on your system
9
+ - All model artifacts in `src/artifacts/` directory
10
+ - Dataset files in `datasets/` directory
11
+
12
+ ### Build Docker Image
13
+
14
+ ```bash
15
+ # Navigate to project root
16
+ cd /path/to/RecSys-HP
17
+
18
+ # Build the Docker image (this will take 5-10 minutes)
19
+ docker build -t recsys-hp:latest .
20
+
21
+ # Or build with a specific tag
22
+ docker build -t recsys-hp:v1.0 .
23
+ ```
24
+
25
+ ### Run Docker Container
26
+
27
+ #### Basic Run (Recommended)
28
+ ```bash
29
+ # Run the container
30
+ docker run -d \
31
+ --name recsys-hp-app \
32
+ -p 8000:8000 \
33
+ recsys-hp:latest
34
+
35
+ # View logs
36
+ docker logs recsys-hp-app
37
+
38
+ # Follow logs in real-time
39
+ docker logs -f recsys-hp-app
40
+ ```
41
+
42
+ #### Run with Volume Mounts (Development)
43
+ ```bash
44
+ # Mount datasets and artifacts for easy updates
45
+ docker run -d \
46
+ --name recsys-hp-dev \
47
+ -p 8000:8000 \
48
+ -v $(pwd)/datasets:/app/datasets \
49
+ -v $(pwd)/src/artifacts:/app/src/artifacts \
50
+ recsys-hp:latest
51
+ ```
52
+
53
+ ### Access the Application
54
+
55
+ Once the container is running:
56
+
57
+ - **Web App**: http://localhost:8000/
58
+ - **API Docs**: http://localhost:8000/docs
59
+ - **API Info**: http://localhost:8000/api
60
+ - **Health Check**: http://localhost:8000/health
61
+
62
+ ### Useful Docker Commands
63
+
64
+ ```bash
65
+ # Check container status
66
+ docker ps
67
+
68
+ # Stop the container
69
+ docker stop recsys-hp-app
70
+
71
+ # Start the container
72
+ docker start recsys-hp-app
73
+
74
+ # Remove the container
75
+ docker rm recsys-hp-app
76
+
77
+ # View container resource usage
78
+ docker stats recsys-hp-app
79
+
80
+ # Execute commands in running container
81
+ docker exec -it recsys-hp-app bash
82
+
83
+ # View container logs
84
+ docker logs recsys-hp-app
85
+ ```
86
+
87
+ ### Troubleshooting
88
+
89
+ #### Container won't start?
90
+ ```bash
91
+ # Check logs for errors
92
+ docker logs recsys-hp-app
93
+
94
+ # Common issues:
95
+ # 1. Missing artifacts in src/artifacts/
96
+ # 2. Missing datasets in datasets/
97
+ # 3. Port 8000 already in use
98
+ ```
99
+
100
+ #### Check if artifacts are present:
101
+ ```bash
102
+ docker exec recsys-hp-app ls -la /app/src/artifacts/
103
+ docker exec recsys-hp-app ls -la /app/datasets/
104
+ ```
105
+
106
+ #### Use different port:
107
+ ```bash
108
+ # Run on port 8080 instead
109
+ docker run -d --name recsys-hp-app -p 8080:8000 recsys-hp:latest
110
+ # Access at http://localhost:8080/
111
+ ```
112
+
113
+ ### Image Information
114
+
115
+ - **Base Image**: python:3.10-slim
116
+ - **Node.js Version**: 18-alpine (build stage only)
117
+ - **Final Image Size**: ~1.5-2GB (includes all ML dependencies)
118
+ - **Exposed Port**: 8000
119
+ - **Health Check**: Enabled (checks /health endpoint)
120
+
121
+ ### Production Deployment
122
+
123
+ For production deployment, consider:
124
+
125
+ ```bash
126
+ # Run with restart policy
127
+ docker run -d \
128
+ --name recsys-hp-prod \
129
+ --restart unless-stopped \
130
+ -p 8000:8000 \
131
+ recsys-hp:latest
132
+
133
+ # Or use docker-compose (recommended for production)
134
+ ```
135
+
136
+ ### Environment Variables
137
+
138
+ The container supports these environment variables:
139
+
140
+ ```bash
141
+ docker run -d \
142
+ --name recsys-hp-app \
143
+ -p 8000:8000 \
144
+ -e PYTHONUNBUFFERED=1 \
145
+ -e LOG_LEVEL=info \
146
+ recsys-hp:latest
147
+ ```
148
+
149
+ The Docker container includes both the React frontend and FastAPI backend in a single image, making deployment simple and efficient! 🚀