Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+weights/*.pth filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1 @@
+*.pth

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Adel Elsayed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,329 @@
+# Masked Autoencoder (MAE) for Medical Imaging
+
+A PyTorch implementation of Masked Autoencoder (MAE) for self-supervised learning on chest X-ray images, specifically designed for the CheXpert dataset.
+
+## 📋 Overview
+
+This project implements a Vision Transformer-based Masked Autoencoder that learns representations from chest X-ray images through self-supervised reconstruction. The model randomly masks 75% of image patches and learns to reconstruct the original image, enabling it to learn powerful visual representations without requiring labeled data.
+
+### Key Features
+
+- **Vision Transformer Architecture**: Encoder-decoder transformer architecture with positional encodings
+- **Self-Supervised Learning**: Pre-training through masked image reconstruction
+- **Optimized for Medical Imaging**: Designed specifically for chest X-ray analysis
+- **Production-Ready Training Pipeline**:
+  - Mixed precision training (FP16) with gradient scaling
+  - Gradient accumulation support
+  - Learning rate warmup and cosine annealing
+  - Automatic checkpointing and resumption
+- **Efficient Data Loading**:
+  - Optimized ZIP file reader with LRU caching
+  - Class-balanced sampling with weighted random sampler
+  - Multi-worker data loading with persistent workers
+- **Comprehensive Logging**: Training/validation metrics tracking and visualization
+
+## 🏗️ Architecture
+
+### Masked Autoencoder Structure
+
+```
+Input Image (384×384) 
+    ↓
+Patchify (16×16 patches → 576 patches)
+    ↓
+Random Masking (75% masked, 25% visible)
+    ↓
+┌─────────────────────────────────────┐
+│           MAE ENCODER               │
+│  - Linear patch embedding           │
+│  - Positional encoding (visible)    │
+│  - 12 Transformer blocks            │
+│  - 8 attention heads, 768 hidden    │
+└─────────────────────────────────────┘
+    ↓
+┌─────────────────────────────────────┐
+│           MAE DECODER               │
+│  - Learnable mask tokens            │
+│  - Positional encoding (all)        │
+│  - 8 Transformer blocks             │
+│  - 8 attention heads, 512 hidden    │
+│  - Pixel reconstruction head        │
+└─────────────────────────────────────┘
+    ↓
+Reconstructed Image
+    ↓
+MSE Loss (on masked patches only)
+```
+
+### Model Configuration
+
+| Parameter | Default Value | Description |
+|-----------|---------------|-------------|
+| Image Size | 384×384 | Input image resolution |
+| Patch Size | 16×16 | Size of each patch |
+| Mask Ratio | 0.75 | Fraction of patches to mask |
+| Encoder Depth | 12 layers | Number of transformer blocks |
+| Encoder Dim | 768 | Hidden dimension |
+| Encoder Heads | 8 | Number of attention heads |
+| Decoder Depth | 8 layers | Number of transformer blocks |
+| Decoder Dim | 512 | Hidden dimension |
+| Decoder Heads | 8 | Number of attention heads |
+| MLP Ratio | 4× | MLP expansion ratio (3072) |
+| Dropout | 0.25 | Dropout rate |
+
+## 🚀 Getting Started
+
+### Prerequisites
+
+- Python >= 3.8
+- CUDA-capable GPU (recommended)
+- 16GB+ RAM
+
+### Installation
+
+1. Clone the repository:
+```bash
+git clone https://github.com/adelelsayed/mae.git
+cd mae
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+### Dataset Preparation
+
+This project is configured for the **CheXpert dataset**. To use it:
+
+1. Download CheXpert-v1.0-small from [Stanford ML Group](https://stanfordmlgroup.github.io/competitions/chexpert/)
+2. Update paths in `configs/configs.py`:
+   - `root`: Base directory for your data
+   - `zip_path`: Path to zipped dataset (optional, for faster loading)
+   - `csv`: Path to training CSV
+   - `train_csv`, `val_csv`, `test_csv`: Split CSV files
+
+## 📊 Usage
+
+### Training
+
+Start training from scratch:
+```bash
+python trainer/trainer.py
+```
+
+The trainer will:
+- Automatically create checkpoint and log directories
+- Resume from the last checkpoint if available
+- Log training/validation metrics to text files
+- Save plots every 10 epochs
+- Save best model based on validation loss
+
+### Training Configuration
+
+Edit `configs/configs.py` to customize training:
+
+```python
+mae_config = {
+    # Training hyperparameters
+    "lr": 1e-4,                    # Learning rate
+    "warmup": 5,                   # Warmup epochs
+    "weight_decay": 5e-4,          # AdamW weight decay
+    "num_epochs": 200,             # Total training epochs
+    "batch_size": 96,              # Batch size
+    "accumulation": 1,             # Gradient accumulation steps
+    
+    # Model architecture
+    "mask_ratio": 0.75,            # Masking ratio
+    "encoder_depth": 12,           # Encoder layers
+    "decoder_depth": 8,            # Decoder layers
+    
+    # Paths
+    "checkpoints": "/path/to/checkpoints",
+    "logdir": "/path/to/logs",
+    ...
+}
+```
+
+### Monitoring Training
+
+Training logs are saved in three files:
+- `training_log.txt`: Training metrics per epoch
+- `val_log.txt`: Validation metrics per epoch  
+- `test_log.txt`: Test set evaluation results
+
+Metrics plots are saved every 10 epochs in `{logdir}/{epoch}/metrics.png`
+
+### Evaluation
+
+The project includes a test method in the trainer. To evaluate:
+```python
+from trainer.utils import MAETrainer
+from configs.configs import mae_config
+
+trainer = MAETrainer(mae_config)
+trainer.test()
+```
+
+## 📁 Project Structure
+
+```
+mae/
+├── configs/
+│   ├── __init__.py
+│   └── configs.py              # Training configuration
+├── data/
+│   ├── __init__.py
+│   ├── dataset.py              # CheXpert dataset loader
+│   └── splitter.py             # Dataset splitting utilities
+├── loss/
+│   ├── __init__.py
+│   └── mae_loss.py             # MAE reconstruction loss
+├── models/
+│   ├── __init__.py
+│   └── mae.py                  # MAE architecture
+├── trainer/
+│   ├── __init__.py
+│   ├── trainer.py              # Main training script
+│   └── utils.py                # Training utilities
+├── notebooks/
+│   └── chexpert_mae.ipynb      # Jupyter notebook for experiments
+├── training logs/              # Logged metrics and plots
+├── weights/                    # Model checkpoints
+├── results/                    # Evaluation results
+├── requirements.txt            # Python dependencies
+├── LICENSE                     # Project license
+└── README.md                   # This file
+```
+
+## 🔧 Components
+
+### Dataset (`data/dataset.py`)
+
+- **OptimizedZipReader**: Fast ZIP file reading with LRU caching
+- **CheXpertDataset**: PyTorch dataset for CheXpert chest X-rays
+  - 14 pathology labels: No Finding, Cardiomegaly, Edema, Consolidation, etc.
+  - Albumentations-based augmentation pipeline
+  - Class-balanced sampling support
+  - Frontal/lateral view filtering
+
+### Model (`models/mae.py`)
+
+- **Patchify/Unpatchify**: Image-to-patch conversion utilities
+- **Random Masking**: Stochastic patch masking with restore indices
+- **PositionalEncoding**: Learnable position embeddings
+- **TransformerBlock**: Multi-head self-attention + MLP
+- **MAEEncoder**: Processes visible patches only
+- **MAEDecoder**: Reconstructs full image with mask tokens
+- **MaskedAutoEncoder**: Complete MAE model
+
+### Loss (`loss/mae_loss.py`)
+
+Mean Squared Error (MSE) computed only on masked patches:
+```python
+loss = ((pred - target) ** 2 * mask).sum() / mask.sum()
+```
+
+### Trainer (`trainer/utils.py`)
+
+- **MAETrainer**: Complete training pipeline
+  - Mixed precision training (AMP)
+  - Gradient clipping and accumulation
+  - Learning rate scheduling (warmup → cosine)
+  - Automatic checkpointing
+  - Multi-file logging (train/val/test)
+  - Live metric monitoring with tqdm
+  - Periodic metric visualization
+
+## 🎯 CheXpert Pathologies
+
+The dataset includes 14 chest X-ray findings:
+
+1. No Finding
+2. Enlarged Cardiomediastinum
+3. Cardiomegaly
+4. Lung Opacity
+5. Lung Lesion
+6. Edema
+7. Consolidation
+8. Pneumonia
+9. Atelectasis
+10. Pneumothorax
+11. Pleural Effusion
+12. Pleural Other
+13. Fracture
+14. Support Devices
+
+## 📈 Training Tips
+
+1. **Learning Rate**: Start with 1e-4, use warmup for stability
+2. **Batch Size**: Maximize based on GPU memory (96 works well on 40GB GPUs)
+3. **Gradient Accumulation**: Use if batch size is limited by memory
+4. **Mixed Precision**: Enabled by default for faster training
+5. **Masking Ratio**: 75% is standard, higher ratios increase difficulty
+6. **Resume Training**: Model automatically resumes from last checkpoint
+
+## 🔬 Use Cases
+
+### Pre-training for Downstream Tasks
+Use the trained encoder as a feature extractor:
+```python
+from models.mae import MaskedAutoEncoder
+
+# Load pre-trained model
+mae = MaskedAutoEncoder()
+mae.load_state_dict(torch.load("best_mae.pth")["model"])
+
+# Use encoder for feature extraction
+encoder = mae.encoder
+features, _, _, _ = encoder(images)
+```
+
+### Fine-tuning on Classification
+Add a classification head to the encoder for supervised tasks.
+
+### Anomaly Detection
+Reconstruction error can indicate abnormalities in medical images.
+
+## 📊 Performance Optimization
+
+This implementation includes several optimizations:
+
+- **Efficient ZIP Reading**: Avoids extracting files to disk
+- **LRU Cache**: Keeps frequently accessed images in memory
+- **Persistent Workers**: Reduces data loading overhead
+- **Mixed Precision**: 2× faster training with minimal quality loss
+- **Gradient Checkpointing**: Reduces memory usage (if enabled)
+- **CUDA Memory Management**: Proper cache clearing and synchronization
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## 📄 License
+
+This project is licensed under the terms specified in the LICENSE file.
+
+## 📚 References
+
+1. **Masked Autoencoders Are Scalable Vision Learners**  
+   He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022)  
+   [arXiv:2111.06377](https://arxiv.org/abs/2111.06377)
+
+2. **CheXpert: A Large Chest Radiograph Dataset**  
+   Irvin, J., et al. (2019)  
+   [Stanford ML Group](https://stanfordmlgroup.github.io/competitions/chexpert/)
+
+## 🙏 Acknowledgments
+
+- Original MAE paper by Meta AI Research
+- CheXpert dataset by Stanford ML Group
+- PyTorch and Albumentations communities
+
+## 📧 Contact
+
+For questions or issues, please open an issue on GitHub or contact the maintainer.
+
+---
+
+**Note**: This is a research/educational implementation. For clinical applications, please ensure proper validation and regulatory compliance.

configs/configs.py

@@ -0,0 +1,27 @@
+import os
+import torch
+root = "/content/drive/MyDrive"
+mae_config={
+    "lr":1e-4,
+    "warmup":5,
+    "weight_decay":5e-4,
+    "num_epochs":200,
+    "num_classes":14,
+    "zip_path":os.path.join(root,"CheXpert-v1.0-small","chexpert.zip"),
+    "resume":os.path.join(root,"CheXpert-v1.0-small","maecheckpoints","best_mae.pth"),
+    "logdir":os.path.join(root,"CheXpert-v1.0-small","maelogs"),
+    "checkpoints":os.path.join(root,"CheXpert-v1.0-small","maecheckpoints"),
+    "datadir":root,
+    "lmdb":os.path.join(root,"CheXpert-v1.0-small","lmdb"),
+    "csv":os.path.join(root,"CheXpert-v1.0-small","train.csv"),
+    "batch_size":96,
+    "device":torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu"),
+    "accumulation":1,
+    "dirsToMake":[os.path.join(root,"CheXpert-v1.0-small","maecheckpoints"),os.path.join(root,"CheXpert-v1.0-small","maelogs")],
+    "train_csv":os.path.join(root,"CheXpert-v1.0-small","train_ready.csv"),
+    "val_csv":os.path.join(root,"CheXpert-v1.0-small","val_ready.csv"),
+    "test_csv":os.path.join(root,"CheXpert-v1.0-small","test_ready.csv")
+    ,"channels":1,"mask_ratio":0.75,"dropout":0.25,"img_size":384,"encoder_dim":768,
+    "mlp_dim":3072,"decoder_dim":512,"encoder_depth":12,"encoder_head":8,"decoder_depth":8,
+    "decoder_head":8,"patch_size":16
+  }

data/dataset.py

@@ -0,0 +1,508 @@
+# Standard library
+import os
+import io
+import zipfile
+import pickle
+from pathlib import Path
+
+# Data handling
+import pandas as pd
+import numpy as np
+
+# PyTorch
+import torch
+from torch.utils.data import Dataset
+
+# Image processing
+from PIL import Image
+import cv2
+
+# Augmentations
+import albumentations as A
+from albumentations.pytorch import ToTensorV2
+
+# Progress bar (for precompute_all_masks)
+from tqdm import tqdm
+
+class OptimizedZipReader:
+    """
+    Fast ZIP file reader with LRU caching
+    """
+    def __init__(self, zip_path, cache_size=1000):
+        """
+        Args:
+            zip_path: Path to ZIP file
+            cache_size: Number of images to cache in RAM
+        """
+        self.zip_path = zip_path
+        self.cache_size = cache_size
+        self._zip_file = None  # Will be lazily initialized
+        self._name_to_info = None
+
+        # Cache
+        self._cache = {}
+        self._cache_order = []
+        self._hits = 0
+        self._misses = 0
+
+    @property
+    def zip_file(self):
+        """Lazy initialization of ZIP file handle"""
+        if self._zip_file is None:
+            print(f"Opening ZIP file: {self.zip_path}")
+            self._zip_file = zipfile.ZipFile(self.zip_path, 'r', allowZip64=True)
+
+            # Build index on first access
+            print("Building ZIP index...")
+            self._name_to_info = {
+                info.filename: info
+                for info in self._zip_file.infolist()
+            }
+            print(f"✓ Indexed {len(self._name_to_info)} files")
+
+        return self._zip_file
+
+    def read_image(self, path):
+        """
+        Read image data with automatic caching
+
+        Returns: bytes (image file data)
+        """
+        # Check cache first
+        if path in self._cache:
+            self._hits += 1
+            return self._cache[path]
+
+        # Cache miss - read from ZIP (this triggers lazy initialization)
+        self._misses += 1
+        img_data = self.zip_file.read(path)  # Uses property getter
+
+        # Add to cache with LRU eviction
+        if len(self._cache) >= self.cache_size:
+            oldest = self._cache_order.pop(0)
+            del self._cache[oldest]
+
+        self._cache[path] = img_data
+        self._cache_order.append(path)
+
+        return img_data
+
+    def get_cache_stats(self):
+        """Return cache hit rate statistics"""
+        total = self._hits + self._misses
+        hit_rate = self._hits / total * 100 if total > 0 else 0
+        return {
+            'hits': self._hits,
+            'misses': self._misses,
+            'hit_rate': f"{hit_rate:.2f}%",
+            'cache_size': len(self._cache)
+        }
+
+    def close(self):
+        """Close ZIP file and clear cache"""
+        if self._zip_file is not None:
+            self._zip_file.close()
+            self._zip_file = None
+        self._cache.clear()
+        self._cache_order.clear()
+        self._name_to_info = None
+
+class CheXpertDataset(Dataset):
+    """
+    CheXpert Dataset class
+
+    NEW: Returns 3-channel images: (img, img*mask, mask)
+    - Channel 0: Original grayscale image
+    - Channel 1: Masked image (lung region only)
+    - Channel 2: Binary lung mask
+
+    Args:
+        csv_path (str): Path to the CSV file (train.csv or valid.csv)
+        root_dir (str): Root directory of the CheXpert dataset
+        image_size (int): Target image size (default: 384)
+        augment (bool): Whether to apply augmentations (default: False)
+        use_frontal_only (bool): If True, only use frontal view images (default: True)
+        fill_uncertain (str): How to handle uncertain labels: 'zeros', 'ones', 'ignore' (default: 'zeros')
+    """
+
+    # 14 pathology classes in CheXpert
+    PATHOLOGIES = [
+        'No Finding',
+        'Enlarged Cardiomediastinum',
+        'Cardiomegaly',
+        'Lung Opacity',
+        'Lung Lesion',
+        'Edema',
+        'Consolidation',
+        'Pneumonia',
+        'Atelectasis',
+        'Pneumothorax',
+        'Pleural Effusion',
+        'Pleural Other',
+        'Fracture',
+        'Support Devices'
+    ]
+
+    def __init__(
+        self,
+        csv_path,
+        root_dir,
+        image_size=384,
+        augment=False,
+        use_frontal_only=False,
+        fill_uncertain='ignore',
+        lmdb_path=None,
+        zip_path=None,
+        zip_cache_size=1000,
+        mask_dir=None, domask=False
+    ):
+        self.root_dir = root_dir
+        self.image_size = image_size
+        self.augment = augment
+        self.fill_uncertain = fill_uncertain
+        self.env =None #lmdb.open(lmdb_path, readonly=True, lock=False) if lmdb_path else None
+        self._zip_path = zip_path
+        self._zip_cache_size = zip_cache_size
+        self._zip_reader_instance = None
+
+
+        # Read CSV file
+        self.df = pd.read_csv(csv_path)
+        for pathology in self.PATHOLOGIES:
+            if pathology in self.df.columns:
+                self.df[pathology] = pd.to_numeric(self.df[pathology], errors='coerce')
+
+        # Filter for frontal views only if specified
+        if use_frontal_only:
+            self.df = self.df[self.df['Frontal/Lateral'] == 'Frontal'].reset_index(drop=True)
+
+        # Handle uncertain labels (-1 values)
+        self._process_uncertain_labels()
+
+        # Setup augmentations
+        self.train_transform = self._get_train_transforms()
+        self.val_transform = self._get_val_transforms()
+
+        print(f"Loaded {len(self.df)} images from {csv_path}")
+        print(f"Image size: {image_size}x{image_size}")
+        print(f"Augmentation: {augment}")
+        print(f"Uncertain labels filled with: {fill_uncertain}")
+
+        if mask_dir and domask:
+            self.precompute_all_masks(mask_dir)
+
+    # Run this ONCE before training
+    def precompute_all_masks(self, save_dir):
+        os.makedirs(save_dir, exist_ok=True)
+        for idx in tqdm(range(len(self))):
+            img_path = os.path.join(self.root_dir,self.df.iloc[idx]['Path'])
+            part_path="/".join(self.df.iloc[idx]['Path'].split("/")[1:])
+            if self.zip_reader:
+                    # Read image data from ZIP (no extraction!)
+                img_data = self.zip_reader.read_image(part_path)
+
+                # Open image from bytes in memory
+                image = Image.open(io.BytesIO(img_data)).convert('L')
+            else:
+                image = Image.open(img_path).convert('L')
+
+            image = np.array(image)
+
+            mask = chexpert_medsam_mask(image)
+            mask_path = os.path.join(save_dir, "_".join(self.df.iloc[idx]['Path'].split("/")[-3:]).replace('.jpg', '_mask.pt'))
+            os.makedirs(os.path.dirname(mask_path), exist_ok=True)
+            torch.save(mask, mask_path)
+    @property
+    def zip_reader(self):
+        """
+        Lazy property getter for ZIP reader
+
+        The ZIP file is only opened when first accessed, not during __init__.
+        This is useful when:
+        - Creating multiple dataset objects but only using some
+        - Saving memory during dataset setup
+        - Working with multiprocessing (each worker creates its own)
+        """
+        if self._zip_reader_instance is None and self._zip_path is not None:
+            self._zip_reader_instance = OptimizedZipReader(
+                self._zip_path,
+                cache_size=self._zip_cache_size
+            )
+        return self._zip_reader_instance
+
+    def _load_and_cache_image(self, img_path, idx):
+        """
+        Load image with automatic resizing and caching.
+        If resized version exists, load it. Otherwise, resize, save, and load.
+
+        Args:
+            img_path (str): Original image path from CSV
+            idx (int): Index for tracking
+
+        Returns:
+            np.ndarray: Loaded image (grayscale)
+        """
+        # Create cache directory structure
+        cache_dir = Path(self.root_dir) #/ f"cache_{self.image_size}"
+
+        # Preserve the relative path structure in cache
+        path_parts = list(Path(img_path).parts)
+        path_parts[-1]=f"{self.image_size}_{path_parts[-1]}"
+        relative_path = Path(*path_parts)
+        cached_path =relative_path.with_suffix('.jpg')
+
+        # Check if cached version exists
+        if cached_path.exists():
+            # Load cached image
+            image = Image.open(cached_path).convert('L')
+            image = np.array(image)
+
+            # Verify it's the correct size
+            if image.shape[0] == self.image_size and image.shape[1] == self.image_size:
+                return image
+
+        # Cache doesn't exist or wrong size - load original
+        original_path = img_path
+        image = Image.open(original_path).convert('L')
+
+        # Check if original is already target size
+        width, height = image.size
+
+        if width == self.image_size and height == self.image_size:
+            # Already correct size, just convert to array
+            return np.array(image)
+
+        # Resize image
+        image_resized = image.resize(
+            (self.image_size, self.image_size),
+            Image.LANCZOS
+        )
+
+        # Save to cache
+        cached_path.parent.mkdir(parents=True, exist_ok=True)
+        image_resized.save(cached_path, 'JPEG', quality=95, optimize=True)
+
+        return np.array(image_resized)
+
+    def _process_uncertain_labels(self):
+        """Process uncertain labels (-1) based on the chosen strategy."""
+        for pathology in self.PATHOLOGIES:
+            if pathology in self.df.columns:
+                if self.fill_uncertain == 'zeros':
+                    # Map uncertain (-1) to negative (0)
+                    self.df[pathology] = self.df[pathology].replace(-1, 0)
+                elif self.fill_uncertain == 'ones':
+                    # Map uncertain (-1) to positive (1)
+                    self.df[pathology] = self.df[pathology].replace(-1, 1)
+                elif self.fill_uncertain == 'ignore':
+                    # Keep -1 as is (you'll need to handle this in loss function)
+                    pass
+
+                # Fill NaN with 0 (negative)
+                self.df[pathology] = self.df[pathology].fillna(0)
+
+    def _get_train_transforms(self):
+        """Get training augmentations suitable for chest X-rays."""
+        import cv2
+        return A.Compose([
+            # Resize to target size
+            A.LongestMaxSize(max_size=self.image_size),
+            A.PadIfNeeded(self.image_size, self.image_size, border_mode=cv2.BORDER_CONSTANT, position='center'),
+
+            # Geometric augmentations (conservative for medical images)
+            A.HorizontalFlip(p=0.5),
+            A.Affine(
+                translate_percent={"x": (-0.1, 0.1), "y": (-0.1, 0.1)},
+                scale=(0.9, 1.1),
+                rotate=(-10, 10),
+                fit_output=False,
+                p=0.5
+            ),
+
+            # Intensity augmentations
+            A.OneOf([
+                A.RandomBrightnessContrast(
+                    brightness_limit=0.2,
+                    contrast_limit=0.2,
+                    p=1.0
+                ),
+                A.RandomGamma(gamma_limit=(80, 120), p=1.0),
+                A.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=1.0),
+            ], p=0.5),
+
+            # Add slight blur to simulate different imaging conditions
+            A.OneOf([
+                A.GaussianBlur(blur_limit=(3, 5), p=1.0),
+                A.MedianBlur(blur_limit=3, p=1.0),
+            ], p=0.2),
+
+            # Add noise
+            A.GaussNoise(p=0.2),
+
+            # Normalize to [0, 1]
+            A.Normalize(
+                mean=[0.5],
+                std=[0.5],
+                max_pixel_value=255.0
+            ),
+
+            ToTensorV2()
+        ])
+
+    def _get_val_transforms(self):
+        """Get validation/test transforms (no augmentation)."""
+        return A.Compose([
+            A.LongestMaxSize(max_size=self.image_size),
+            A.PadIfNeeded(self.image_size, self.image_size, border_mode=cv2.BORDER_CONSTANT, position='center'),
+            A.Normalize(
+                mean=[0.5],
+                std=[0.5],
+                max_pixel_value=255.0
+            ),
+            ToTensorV2()
+        ])
+
+    def __len__(self):
+        return len(self.df)
+
+    def __del__(self):
+        """Close ZIP when done"""
+        if hasattr(self, 'zip_reader'):
+            self.zip_reader.close()
+
+    def __getitem__(self, idx):
+        if self.env:
+            with self.env.begin() as txn:
+                # Retrieve serialized data
+                data = txn.get(str(idx).encode())
+                sample = pickle.loads(data)
+                return sample
+        else:
+            # Get image path
+            img_path = os.path.join(self.root_dir,self.df.iloc[idx]['Path'])
+            #image = self._load_and_cache_image(img_path, idx)
+            # Load image
+            #image = Image.open(img_path).convert('L')  # Convert to grayscale
+
+            part_path="/".join(self.df.iloc[idx]['Path'].split("/")[1:])
+            if self.zip_reader:
+                    # Read image data from ZIP (no extraction!)
+                img_data = self.zip_reader.read_image(part_path)
+
+                # Open image from bytes in memory
+                image = Image.open(io.BytesIO(img_data)).convert('L')
+            else:
+                image = Image.open(img_path).convert('L')
+
+            image = np.array(image)
+
+
+            # Load pre-computed mask
+            #mask_path = os.path.join(self.mask_dir, "_".join(self.df.iloc[idx]['Path'].split("/")[-3:]).replace('.jpg', '_mask.pt'))
+            #masked_img = torch.load(mask_path)
+            # Apply transforms to BOTH image and mask together
+            if self.augment:
+                # Augmentation applies to both image and mask
+                transformed = self.train_transform(image=image)
+                image_transformed = transformed['image']  # (1, H, W) tensor, normalized
+                #masked_img=transformed['mask']
+                  # (H, W) tensor
+            else:
+                transformed = self.val_transform(image=image)
+                image_transformed = transformed['image']  # (1, H, W) tensor, normalized
+                #masked_img=transformed['mask']
+
+            # Expand dimensions to match
+            image_1ch = image_transformed  # (1, H, W)
+            masked_img = image_transformed
+
+            # Get labels for all pathologies
+            labels = []
+            for pathology in self.PATHOLOGIES:
+                if pathology in self.df.columns:
+                    label = self.df.iloc[idx][pathology]
+                    labels.append(float(label) if not pd.isna(label) else 0.0)
+                else:
+                    labels.append(0.0)
+
+            labels = torch.tensor(labels, dtype=torch.float32)
+
+            # Get additional metadata
+            metadata = {
+                'patient_id': self.df.iloc[idx]['Path'].split('/')[2],  # Extract patient ID from path
+                'study_id': self.df.iloc[idx]['Path'].split('/')[3],    # Extract study ID from path
+                'view': self.df.iloc[idx]['Frontal/Lateral'],
+                'sex': self.df.iloc[idx]['Sex'] if 'Sex' in self.df.columns else 'Unknown',
+                'age': self.df.iloc[idx]['Age'] if 'Age' in self.df.columns else -1,
+                'path': self.df.iloc[idx]['Path']
+            }
+
+            return {
+                'image': image_1ch,
+                'labels': labels,
+                'metadata': metadata
+            }
+
+    def get_label_names(self):
+        """Return list of pathology label names."""
+        return self.PATHOLOGIES
+
+    def get_label_distribution(self):
+        """Get distribution of positive labels for each pathology."""
+        distribution = {}
+        for pathology in self.PATHOLOGIES:
+            if pathology in self.df.columns:
+                positive_count = (self.df[pathology] == 1.0).sum()
+                distribution[pathology] = {
+                    'positive': int(positive_count),
+                    'percentage': round(positive_count / len(self.df) * 100, 2)
+                }
+        return distribution
+
+    def get_class_weights(self):
+        """
+        OPTIMIZED: Vectorized class weights calculation
+        """
+        weights = []
+        for pathology in self.PATHOLOGIES:
+            if pathology in self.df.columns:
+                # Vectorized counting (much faster than iterating)
+                values = self.df[pathology].values
+                pos = np.sum(values == 1.0)
+                neg = np.sum(values == 0.0)
+                weight = neg / pos if pos > 0 else 1.0
+                weights.append(weight)
+        return torch.tensor(weights, dtype=torch.float32)
+
+    def get_sample_weights(self):
+        """
+        OPTIMIZED: Vectorized sample weights calculation
+
+        Performance: ~1000x faster than original
+        Original: 15-30 seconds for 200k samples
+        This: 0.01-0.05 seconds for 200k samples
+        """
+        # Get class weights as numpy array
+        class_weights = self.get_class_weights().numpy()
+
+        # Get all labels as numpy array in ONE vectorized operation
+        labels_array = self.df[self.PATHOLOGIES].values.astype(np.float32)
+
+        # Create weighted labels matrix: where label=1, use class_weight, else -inf
+        # Shape: (n_samples, n_classes)
+        weighted_labels = np.where(
+            labels_array == 1.0,
+            class_weights,
+            -np.inf  # Use -inf instead of 0 so max will only consider positive labels
+        )
+
+        # For each sample, find the maximum class weight of its positive labels
+        # If a sample has no positive labels, max will be -inf, which we'll replace with 1.0
+        sample_weights = np.max(weighted_labels, axis=1)
+        sample_weights = np.where(
+            np.isinf(sample_weights),
+            1.0,  # Samples with no positive labels get weight 1.0
+            sample_weights
+        )
+
+        return torch.tensor(sample_weights, dtype=torch.float32)

data/splitter.py

@@ -0,0 +1,347 @@
+# Standard library
+import os
+from pathlib import Path
+
+# Data handling
+import pandas as pd
+import numpy as np
+
+# Machine learning
+from sklearn.model_selection import train_test_split
+
+class CheXpertDataSplitter:
+    """
+    Advanced stratified train-validation splitter for CheXpert dataset.
+    Handles:
+    - Patient-level splitting (prevents data leakage)
+    - Multi-label stratification
+    - Class imbalance awareness
+    - Study-level grouping
+    """
+
+    PATHOLOGIES = [
+        'No Finding',
+        'Enlarged Cardiomediastinum',
+        'Cardiomegaly',
+        'Lung Opacity',
+        'Lung Lesion',
+        'Edema',
+        'Consolidation',
+        'Pneumonia',
+        'Atelectasis',
+        'Pneumothorax',
+        'Pleural Effusion',
+        'Pleural Other',
+        'Fracture',
+        'Support Devices'
+    ]
+
+    def __init__(self, csv_path, val_size=0.15,test_size=0.15, random_state=42,
+                 use_frontal_only=True, fill_uncertain='zeros',root=None):
+        """
+        Initialize the splitter.
+
+        Args:
+            csv_path: Path to train.csv from CheXpert-small
+            val_size: Validation set proportion (default: 0.15)
+            random_state: Random seed for reproducibility
+            use_frontal_only: Use only frontal view images
+            fill_uncertain: How to handle uncertain labels ('zeros', 'ones', 'ignore')
+        """
+        self.csv_path = csv_path
+        self.val_size = val_size
+        self.test_size = test_size
+        self.random_state = random_state
+        self.use_frontal_only = use_frontal_only
+        self.fill_uncertain = fill_uncertain
+        self.root=root
+
+        print("=" * 80)
+        print("CheXpert Data Splitter - Preventing Data Leakage & Class Bias")
+        print("=" * 80)
+
+    def load_and_preprocess(self):
+        """Load and preprocess the dataset."""
+        print("\n[1/5] Loading data...")
+        self.df = pd.read_csv(self.csv_path)
+        print(f"   Loaded {len(self.df)} images")
+
+        #self.df=self.df[self.df["Path"].apply(os.path.exists)]
+
+        # Filter for frontal views only
+        if self.use_frontal_only:
+            initial_count = len(self.df)
+            self.df = self.df[self.df['Frontal/Lateral'] == 'Frontal'].reset_index(drop=True)
+            print(f"   Filtered to frontal views: {len(self.df)} images ({initial_count - len(self.df)} removed)")
+
+        # Extract patient and study IDs from path
+        print("\n[2/5] Extracting patient and study IDs...")
+        self.df['patient_id'] = self.df['Path'].apply(lambda x: x.split('/')[2])
+        self.df['study_id'] = self.df['Path'].apply(lambda x: x.split('/')[3])
+
+        n_patients = self.df['patient_id'].nunique()
+        n_studies = self.df['study_id'].nunique()
+        print(f"   Unique patients: {n_patients}")
+        print(f"   Unique studies: {n_studies}")
+        print(f"   Images per patient (avg): {len(self.df) / n_patients:.2f}")
+
+        # Process uncertain labels
+        print("\n[3/5] Processing uncertain labels...")
+        self._process_uncertain_labels()
+
+        return self.df
+
+    def _process_uncertain_labels(self):
+        """Process uncertain labels (-1) based on the chosen strategy."""
+        for pathology in self.PATHOLOGIES:
+            if pathology in self.df.columns:
+                uncertain_count = (self.df[pathology] == -1).sum()
+
+                if self.fill_uncertain == 'zeros':
+                    self.df[pathology] = self.df[pathology].replace(-1, 0)
+                elif self.fill_uncertain == 'ones':
+                    self.df[pathology] = self.df[pathology].replace(-1, 1)
+                elif self.fill_uncertain == 'ignore':
+                    pass  # Keep -1 as is
+
+                # Fill NaN with 0
+                self.df[pathology] = self.df[pathology].fillna(0)
+
+        print(f"   Uncertain labels strategy: {self.fill_uncertain}")
+
+    def create_stratification_groups(self):
+        """
+        Create stratification groups based on multi-label combinations.
+        Uses patient-level aggregation to prevent data leakage.
+        """
+        print("\n[4/5] Creating stratification groups (patient-level)...")
+
+        # Group by patient and aggregate labels
+        patient_groups = self.df.groupby('patient_id').agg({
+            **{pathology: 'max' for pathology in self.PATHOLOGIES if pathology in self.df.columns},
+            'study_id': 'first',  # Keep one study_id for reference
+            'Sex': 'first',
+            'Age': 'first'
+        }).reset_index()
+
+        # Create label signature for each patient
+        # This is a binary string representing which conditions are present
+        def create_label_signature(row):
+            signature = []
+            for pathology in self.PATHOLOGIES:
+                if pathology in patient_groups.columns:
+                    signature.append(str(int(row[pathology])))
+            return ''.join(signature)
+
+        patient_groups['label_signature'] = patient_groups.apply(create_label_signature, axis=1)
+
+        # For rare combinations, group them together
+        signature_counts = patient_groups['label_signature'].value_counts()
+        rare_threshold = max(5, int(len(patient_groups) * 0.001))  # At least 5 or 0.1%
+
+        def get_stratification_group(signature):
+            if signature_counts[signature] < rare_threshold:
+                return 'RARE_COMBINATION'
+            return signature
+
+        patient_groups['stratification_group'] = patient_groups['label_signature'].apply(get_stratification_group)
+
+        # Print distribution statistics
+        print(f"\n   Patient-level label distribution:")
+        for pathology in self.PATHOLOGIES:
+            if pathology in patient_groups.columns:
+                positive_count = (patient_groups[pathology] == 1).sum()
+                percentage = positive_count / len(patient_groups) * 100
+                print(f"   {pathology:30s}: {positive_count:5d} ({percentage:5.2f}%)")
+
+        unique_groups = patient_groups['stratification_group'].nunique()
+        print(f"\n   Unique stratification groups: {unique_groups}")
+        print(f"   Rare combinations grouped: {(patient_groups['stratification_group'] == 'RARE_COMBINATION').sum()}")
+
+        return patient_groups
+
+    def perform_split(self, patient_groups):
+        """
+        Perform stratified train-validation-test split at patient level.
+        """
+        print("\n[5/5] Performing stratified patient-level split...")
+
+        stratification_labels = patient_groups['stratification_group'].values
+
+        # ---- train / (val+test) ----
+        train_patients, valtest_patients = train_test_split(
+            patient_groups['patient_id'].values,
+            test_size=self.val_size + self.test_size,          # <-- new
+            stratify=stratification_labels,
+            random_state=self.random_state
+        )
+
+        # ---- val / test from the remaining pool ----
+        remaining_labels = patient_groups.set_index('patient_id').loc[valtest_patients]['stratification_group'].values
+        val_patients, test_patients = train_test_split(
+            valtest_patients,
+            test_size=self.test_size / (self.val_size + self.test_size),   # <-- proportion of the val+test pool
+            stratify=remaining_labels,
+            random_state=self.random_state
+        )
+
+        print(f"   Train patients: {len(train_patients)}")
+        print(f"   Val   patients: {len(val_patients)}")
+        print(f"   Test  patients: {len(test_patients)}")
+
+        # Split the full dataframe
+        train_df = self.df[self.df['patient_id'].isin(train_patients)].copy()
+        val_df   = self.df[self.df['patient_id'].isin(val_patients)].copy()
+        test_df  = self.df[self.df['patient_id'].isin(test_patients)].copy()
+
+        # ---- leakage check (train vs val vs test) ----
+        sets = [('train', train_df), ('val', val_df), ('test', test_df)]
+        for i, (name_i, df_i) in enumerate(sets):
+            for j, (name_j, df_j) in enumerate(sets[i+1:]):
+                overlap = set(df_i['patient_id']).intersection(set(df_j['patient_id']))
+                if overlap:
+                    raise ValueError(f"Data leakage between {name_i} and {name_j}: {len(overlap)} patients overlap")
+        print("\n   No patient overlap – leakage prevented!")
+
+        return train_df, val_df, test_df
+
+    def run(self, output_dir='.', save_test=True):
+        self.load_and_preprocess()
+        patient_groups = self.create_stratification_groups()
+        train_df, val_df, test_df = self.perform_split(patient_groups)
+
+        self.verify_split_quality(train_df, val_df)
+        # optional: also verify train vs test (same function works with two dfs)
+        print("\n--- Train vs Test distribution check ---")
+        self.verify_split_quality(train_df, test_df)
+
+        train_path, val_path = self.save_splits(train_df, val_df, output_dir)
+        if save_test:
+            test_path = self.save_test_split(test_df, output_dir)
+        else:
+            test_path = None
+
+        print("\n" + "="*80)
+        print("Split Complete! (train / val / test)")
+        print("="*80)
+        return train_path, val_path, test_path
+
+    def save_test_split(self, test_df, output_dir):
+        output_dir = Path(output_dir)
+        output_dir.mkdir(exist_ok=True)
+        test_path = output_dir / 'test_ready.csv'
+
+        cols_to_drop = ['patient_id', 'study_id']
+        test_clean = test_df.drop(columns=[c for c in cols_to_drop if c in test_df.columns])
+        test_clean.to_csv(test_path, index=False)
+
+        print(f"Test set : {test_path} ({len(test_clean)} images)")
+        return test_path
+
+    def verify_split_quality(self, train_df, val_df):
+        """
+        Verify the quality of the split by comparing label distributions.
+        """
+        print("\n" + "=" * 80)
+        print("Split Quality Verification")
+        print("=" * 80)
+
+        print(f"\n{'Pathology':<30s} {'Train %':>10s} {'Val %':>10s} {'Difference':>12s}")
+        print("-" * 80)
+
+        max_diff = 0
+        for pathology in self.PATHOLOGIES:
+            if pathology in train_df.columns:
+                train_pos = (train_df[pathology] == 1).sum() / len(train_df) * 100
+                val_pos = (val_df[pathology] == 1).sum() / len(val_df) * 100
+                diff = abs(train_pos - val_pos)
+                max_diff = max(max_diff, diff)
+
+                print(f"{pathology:<30s} {train_pos:>9.2f}% {val_pos:>9.2f}% {diff:>11.2f}%")
+
+        print("-" * 80)
+        print(f"Maximum distribution difference: {max_diff:.2f}%")
+
+        if max_diff < 2.0:
+            print("✓ Excellent stratification (< 2% difference)")
+        elif max_diff < 5.0:
+            print("✓ Good stratification (< 5% difference)")
+        else:
+            print("⚠ Warning: Large distribution differences detected")
+
+        # Check for class imbalance
+        print("\n" + "=" * 80)
+        print("Class Imbalance Analysis (Train Set)")
+        print("=" * 80)
+
+        imbalance_ratios = []
+        for pathology in self.PATHOLOGIES:
+            if pathology in train_df.columns:
+                pos = (train_df[pathology] == 1).sum()
+                neg = (train_df[pathology] == 0).sum()
+                if pos > 0:
+                    ratio = neg / pos
+                    imbalance_ratios.append(ratio)
+                    severity = "Low" if ratio < 5 else "Medium" if ratio < 20 else "High"
+                    print(f"{pathology:<30s} Ratio: {ratio:>6.2f}:1 [{severity:>6s} imbalance]")
+
+        avg_imbalance = np.mean(imbalance_ratios)
+        print(f"\nAverage imbalance ratio: {avg_imbalance:.2f}:1")
+
+    def save_splits(self, train_df, val_df, output_dir='.'):
+        """Save train and validation splits to CSV files."""
+        output_dir = Path(output_dir)
+        output_dir.mkdir(exist_ok=True)
+
+        train_path = output_dir / 'train_ready.csv'
+        val_path = output_dir / 'val_ready.csv'
+
+        # Remove temporary columns used for splitting
+        columns_to_drop = ['patient_id', 'study_id']
+        train_df_clean = train_df.drop(columns=[col for col in columns_to_drop if col in train_df.columns])
+        val_df_clean = val_df.drop(columns=[col for col in columns_to_drop if col in val_df.columns])
+
+        train_df_clean.to_csv(train_path, index=False)
+        val_df_clean.to_csv(val_path, index=False)
+
+        print("\n" + "=" * 80)
+        print("Files Saved Successfully")
+        print("=" * 80)
+        print(f"Train set: {train_path} ({len(train_df_clean)} images)")
+        print(f"Val set:   {val_path} ({len(val_df_clean)} images)")
+
+        return train_path, val_path
+
+# Main execution
+if __name__ == "__main__":
+    root = "/content/drive/MyDrive"
+    # Configuration
+    CHEXPERT_CSV = os.path.join(root,"CheXpert-v1.0-small","train.csv")  # Adjust path as needed
+    OUTPUT_DIR = os.path.join(root,"CheXpert-v1.0-small")
+    VAL_SIZE = 0.15
+    RANDOM_STATE = 42
+    USE_FRONTAL_ONLY = True
+    FILL_UNCERTAIN = 'zeros'  # Options: 'zeros', 'ones', 'ignore'
+
+    # Create splitter
+    splitter = CheXpertDataSplitter(
+        csv_path=CHEXPERT_CSV,
+        val_size=VAL_SIZE,test_size=VAL_SIZE,
+        random_state=RANDOM_STATE,
+        use_frontal_only=USE_FRONTAL_ONLY,
+        fill_uncertain=FILL_UNCERTAIN,
+        root=OUTPUT_DIR
+    )
+
+    # Run the split
+    if os.path.exists(os.path.join(root,"CheXpert-v1.0-small","train_ready.csv")) and os.path.exists(os.path.join(root,"CheXpert-v1.0-small","val_ready.csv")):
+        train_path=os.path.join(root,"CheXpert-v1.0-small","train_ready.csv")
+        val_path=os.path.join(root,"CheXpert-v1.0-small","val_ready.csv")
+        test_path=os.path.join(root,"CheXpert-v1.0-small","test_ready.csv")
+    else:
+        train_path, val_path,test_path = splitter.run(output_dir=OUTPUT_DIR)
+
+    print("\nYou can now use these files with your CheXpertDataset class:")
+    print(f"  train_dataset = CheXpertDataset('{train_path}', root_dir='...', augment=True)")
+    print(f"  val_dataset = CheXpertDataset('{val_path}', root_dir='...', augment=False)")
+    print(f"  test_dataset = CheXpertDataset('{test_path}', root_dir='...', augment=False)")

loss/mae_loss.py

@@ -0,0 +1,10 @@
+import torch
+import torch.nn as nn
+
+def mae_loss(pred, target, mask):
+    # pred/target: (B, N, P), mask: (B, N) with 1=masked
+    B, N, P = pred.shape
+    mask = mask.unsqueeze(-1).float()            # (B, N, 1)
+    loss = (pred - target) ** 2
+    loss = (loss * mask).sum() / mask.sum().clamp_min(1.0)
+    return loss

models/mae.py

@@ -0,0 +1,177 @@
+import torch
+import torch.nn as nn
+import math
+def patchify(x,patch_size=8):
+    b,c,h,w=x.shape
+    th=h//patch_size
+    tw=w//patch_size
+    assert h%patch_size==0 and w%patch_size==0, "Image size must be divisible by patch_size"
+
+    out=x.reshape(b,c,th,patch_size,tw,patch_size)
+    out=out.permute(0,2,4,1,3,5).contiguous()
+    out=out.view(b,th*tw,c*(patch_size**2))
+    return out
+def unpatchify(x,patch_size=8):
+    b,z,p=x.shape
+    c=p//(patch_size**2)
+    th=int(math.sqrt(z))
+    tw=th
+    h=th*patch_size
+    w=tw*patch_size
+    x=x.view(b,th,tw,c,patch_size,patch_size)
+    x=x.permute(0,3,1,4,2,5).contiguous()
+    out=x.view(b,c,h,w)
+    return out
+def random_mask(x,mask_ratio=0.75):
+    b,n,p=x.shape
+    len_keep=int(n*(1-mask_ratio))
+    noise=torch.rand(b,n).to(x.device)
+    ids_shuffle=torch.argsort(noise,dim=1)
+    ids_restore=torch.argsort(ids_shuffle,dim=1)
+    ids_keep=ids_shuffle[:,:len_keep]
+    x_masked=torch.gather(x,dim=1,index=ids_keep.unsqueeze(-1).expand(-1,-1,p)).to(x.device)
+    mask=torch.ones(b,n).to(x.device)
+    mask[:,:len_keep]=0
+    mask=torch.gather(mask,dim=1,index=ids_restore).to(x.device)
+    return x_masked,mask,ids_restore,ids_keep
+
+def mae_loss(pred, target, mask):
+    # pred/target: (B, N, P), mask: (B, N) with 1=masked
+    B, N, P = pred.shape
+    mask = mask.unsqueeze(-1).float()  # (B, N, 1)
+    loss = (pred - target) ** 2
+    loss = (loss * mask).sum() / mask.sum().clamp_min(1.0)
+    return loss
+
+class PositionalEncoding(nn.Module):
+    def __init__(self,num_patches,hidden_dim=768):
+        super().__init__()
+        self.pos_embed=nn.Parameter(torch.empty(1,num_patches,hidden_dim))
+        nn.init.trunc_normal_(self.pos_embed, std=0.02)
+    def forward(self, x, visible_indices):
+        # x: (B, len_keep, D); visible_indices: (B, len_keep)
+        B, L, D = x.shape
+        # expand table to (B, N, D)
+        pos = self.pos_embed.expand(B, -1, -1)                 # (B, N, D)
+        # build gather index (B, L, D)
+        idx = visible_indices.unsqueeze(-1).expand(B, L, pos.size(-1))
+        visible_pos = torch.gather(pos, 1, idx)                # (B, L, D)
+        return x + visible_pos
+
+class TransformerBlock(nn.Module):
+    def __init__(self,hidden_dim,mlp_dim,num_heads,dropout):
+        super().__init__()
+        self.layernorm1=nn.LayerNorm(hidden_dim)
+        self.multihead=nn.MultiheadAttention(batch_first=True,embed_dim=hidden_dim,num_heads=num_heads,dropout=dropout)
+        self.layernorm2=nn.LayerNorm(hidden_dim)
+        self.mlp=nn.Sequential(
+            nn.Linear(hidden_dim,mlp_dim),nn.GELU(),nn.Dropout(dropout),nn.Linear(mlp_dim,hidden_dim),nn.Dropout(dropout)
+        )
+
+
+    def forward(self,x):
+        residual=x
+        x=self.layernorm1(x)
+        attn,_=self.multihead(x,x,x)
+        x=residual+attn
+        residual=x
+        x=self.layernorm2(x)
+        x=self.mlp(x)
+        x=residual+x
+        return x
+
+class MAEEncoder(nn.Module):
+    """
+    patch_dim-> % non-masked * no. of patches
+    """
+    def __init__(self,patch_dim,num_patches=(384//4)**2,hidden_dim=768,mlp_dim=768*4,num_heads=8,depth=12,dropout=0.25,mask_ratio=0.75,patch_size=8):
+        super().__init__()
+        self.mask_ratio=mask_ratio
+        self.patch_size=patch_size
+        self.patch_embed=nn.Linear(patch_dim,hidden_dim)
+        self.pos_embed=PositionalEncoding(num_patches=num_patches,hidden_dim=hidden_dim)
+        self.transformer=nn.ModuleList([TransformerBlock(hidden_dim=hidden_dim,mlp_dim=mlp_dim,num_heads=num_heads,dropout=dropout)
+                                  for _ in range(depth)])
+
+        self._init_weights()
+    def _init_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.trunc_normal_(m.weight, std=0.02)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self,x_in):
+        x_p=patchify(x_in,self.patch_size)
+        x_masked,mask,ids_restore,ids_keep=random_mask(x_p,self.mask_ratio)
+        x= self.patch_embed(x_masked)
+        x=self.pos_embed(x,ids_keep)
+        for attn_layer in self.transformer:x=attn_layer(x)
+        return x,mask,ids_keep,ids_restore
+
+class MAEDecoder(nn.Module):
+    def __init__(self,c,num_patches,patch_size,encoder_dim,decoder_dim,decoder_depth,mlp_dim,num_heads,dropout):
+        super().__init__()
+        self.num_patches=num_patches
+        self.encoder_dim=encoder_dim
+        self.decoder_dim=decoder_dim
+        self.mask_token=nn.Parameter(torch.empty(1,1,decoder_dim))
+        self.enc_to_dec=nn.Linear(encoder_dim,decoder_dim)
+        self.pos_embed=nn.Parameter(torch.empty(1,num_patches,decoder_dim))
+        self.transformer=nn.ModuleList([TransformerBlock(hidden_dim=decoder_dim,mlp_dim=mlp_dim,num_heads=num_heads,dropout=dropout)
+                                  for _ in range(decoder_depth)])
+        self.layernorm=nn.LayerNorm(decoder_dim)
+        self.pred=nn.Linear(decoder_dim,c*(patch_size**2))
+
+        self._init_weights()
+    def _init_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.trunc_normal_(m.weight, std=0.02)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+        nn.init.trunc_normal_(self.pos_embed, std=0.02)
+        nn.init.trunc_normal_(self.mask_token, std=0.02)
+    def forward(self,x,ids_keep,ids_restore):
+        b,n,p=x.shape
+        xdec=self.enc_to_dec(x)
+        len_keep=xdec.size(1)
+        num_patches=ids_restore.size(1)
+        num_mask=num_patches-len_keep
+
+        mask_token=self.mask_token.expand(b,num_mask,-1)
+        x_=torch.cat([xdec,mask_token],dim=1)
+        x_=torch.gather(x_,dim=1,index=ids_restore.unsqueeze(-1).expand(-1,-1,x_.size(-1)))
+        x_=x_+self.pos_embed
+        for block in self.transformer:x_=block(x_)
+        x_=self.layernorm(x_)
+        out=self.pred(x_)
+        return out
+
+class MaskedAutoEncoder(nn.Module):
+    def __init__(self,c=1,mask_ratio=0.75,dropout=0.25,img_size=384,encoder_dim=768,mlp_dim=3072,decoder_dim=512,encoder_depth=12,encoder_head=8,decoder_depth=8,decoder_head=8,patch_size=8):
+        super().__init__()
+        self.patch_size=patch_size
+        self.encoder=MAEEncoder(patch_dim=c*(patch_size**2),num_patches=(img_size//patch_size)**2
+                                ,hidden_dim=encoder_dim,mlp_dim=mlp_dim,num_heads=encoder_head
+                                ,depth=encoder_depth,dropout=dropout,mask_ratio=mask_ratio,patch_size=patch_size)
+        self.decoder=MAEDecoder(c,num_patches=(img_size//patch_size)**2,patch_size=patch_size
+                                ,encoder_dim=encoder_dim,decoder_dim=decoder_dim,decoder_depth=decoder_depth
+                                ,mlp_dim=mlp_dim,num_heads=decoder_head,dropout=dropout)
+
+    def forward(self,x):
+        b,c,h,w=x.shape
+        encoded,mask,ids_keep,ids_restore=self.encoder(x)
+        decoded=self.decoder(encoded,ids_keep,ids_restore)
+
+        xpatched=patchify(x,self.patch_size)
+        return xpatched,decoded,mask
+
+    @staticmethod
+    def testme():
+        img=torch.rand(1,1,384,384)
+        mae=MaskedAutoEncoder()
+        a,b,c=mae(img)
+        print(a.shape)
+        print(b.shape)
+        print(c.shape)

requirements.txt

@@ -0,0 +1,29 @@
+# Core Deep Learning
+torch>=2.0.0
+torchvision>=0.15.0
+
+# Data Processing
+numpy>=1.24.0
+pandas>=2.0.0
+scikit-learn>=1.3.0
+
+# Image Processing
+Pillow>=10.0.0
+opencv-python>=4.8.0
+albumentations>=1.3.1
+
+# Visualization
+matplotlib>=3.7.0
+seaborn>=0.12.0
+
+# Utilities
+tqdm>=4.65.0
+
+# Jupyter (optional - for notebooks)
+jupyter>=1.0.0
+ipykernel>=6.25.0
+ipywidgets>=8.1.0
+
+# Additional utilities (if needed)
+# lmdb>=1.4.0  # Uncomment if using LMDB for caching
+# tensorboard>=2.13.0  # Uncomment if using TensorBoard logging

trainer/trainer.py

@@ -0,0 +1,14 @@
+from trainer.utils import *
+from configs.configs import root,mae_config
+
+def main():
+    try:
+        print(f"Training mae")
+        trainer=MAETrainer(mae_config)
+        trainer.test()
+        
+    except:
+        import traceback
+        traceback.print_exc()
+        
+if __name__=="__main__":main()

trainer/utils.py

@@ -0,0 +1,348 @@
+from data.dataset import CheXpertDataset
+from loss.mae_loss import mae_loss
+from models.mae import *
+from torch.utils.data import DataLoader
+import json
+import os
+import io
+import sys
+
+class TeeFile:
+    """
+    File-like object that writes to multiple streams (e.g., stdout and a file)
+    Automatically handles string paths by opening them as files.
+
+    Usage:
+        # This now works with both file objects and paths
+        tee = TeeFile(sys.stdout, "/path/to/log.txt")
+        print("Hello", file=tee)  # Writes to both stdout and the file
+    """
+    def __init__(self, *file_objects_or_paths):
+        """
+        Args:
+            *file_objects_or_paths: Mix of file objects (like sys.stdout)
+                                   or string paths to log files
+        """
+        self.files = []
+        self.opened_files = []  # Track files we opened so we can close them later
+
+        for item in file_objects_or_paths:
+            if isinstance(item, str):
+                # It's a path string - open it as a file
+                f = open(item, 'a', buffering=1)  # Append mode, line buffered
+                self.files.append(f)
+                self.opened_files.append(f)
+            else:
+                # It's already a file-like object (e.g., sys.stdout)
+                self.files.append(item)
+
+    def write(self, data):
+        """Write data to all streams"""
+        for f in self.files:
+            try:
+                f.write(data)
+                f.flush()
+            except Exception as e:
+                # Handle closed file gracefully
+                print(f"Warning: Could not write to {f}: {e}", file=sys.stderr)
+
+    def flush(self):
+        """Flush all streams"""
+        for f in self.files:
+            try:
+                f.flush()
+            except:
+                pass
+
+    def isatty(self):
+        """Check if any stream is a terminal (for tqdm compatibility)"""
+        return any(getattr(f, "isatty", lambda: False)() for f in self.files)
+
+    def fileno(self):
+        """Get file descriptor from any real file-like stream"""
+        for f in self.files:
+            if hasattr(f, "fileno"):
+                try:
+                    return f.fileno()
+                except Exception:
+                    pass
+        raise io.UnsupportedOperation("No fileno available")
+
+    def close(self):
+        """Close any files we opened"""
+        for f in self.opened_files:
+            try:
+                f.close()
+            except:
+                pass
+        self.opened_files.clear()
+
+    def __del__(self):
+        """Cleanup on deletion"""
+        self.close()
+
+    def __enter__(self):
+        """Context manager support"""
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """Context manager cleanup"""
+        self.close()
+        return False
+    
+class MAETrainer:
+    def __init__(self,configs={}):
+
+        self.configs=configs
+        os.makedirs(configs["logdir"],exist_ok=True)
+        log_path_train = os.path.join(configs["logdir"], "training_log.txt")
+        log_path_val = os.path.join(configs["logdir"], "val_log.txt")
+        log_path_test = os.path.join(configs["logdir"], "test_log.txt")
+        #self.log_file = open(log_path, 'w', buffering=1)
+        self.traintee = TeeFile(sys.stdout, log_path_train)
+        self.valtee = TeeFile(sys.stdout, log_path_val)
+        self.testtee = TeeFile(sys.stdout, log_path_test)
+
+        for dir in self.configs["dirsToMake"]: os.makedirs(dir,exist_ok=True)
+
+        self.model=MaskedAutoEncoder(
+            c=configs["channels"],
+            mask_ratio=configs["mask_ratio"],
+            dropout=configs["dropout"],
+            img_size=configs["img_size"],
+            encoder_dim=configs["encoder_dim"],
+            mlp_dim=configs["mlp_dim"],
+            decoder_dim=configs["decoder_dim"],
+            encoder_depth=configs["encoder_depth"],
+            encoder_head=configs["encoder_head"],
+            decoder_depth=configs["decoder_depth"],
+            decoder_head=configs["decoder_head"],
+            patch_size=configs["patch_size"]
+        ).to(configs["device"])
+
+        self.criterion=mae_loss
+
+        self.optimizer=torch.optim.AdamW(self.model.parameters(),configs["lr"], weight_decay=configs["weight_decay"])
+        self.schedular1=torch.optim.lr_scheduler.LinearLR(self.optimizer,start_factor=0.1,end_factor=1.0,total_iters=configs["warmup"])
+        self.schedular2=torch.optim.lr_scheduler.CosineAnnealingLR(self.optimizer,T_max=configs["num_epochs"]-configs["warmup"])
+        self.schedular=torch.optim.lr_scheduler.SequentialLR (self.optimizer,schedulers=[self.schedular1,self.schedular2],milestones=[configs["warmup"]])
+        self.scaler=torch.amp.GradScaler()
+
+        self.train_dataset= CheXpertDataset(zip_path=configs["zip_path"],csv_path=configs["train_csv"],root_dir=configs["datadir"],augment=True,use_frontal_only=True)
+        self.val_dataset= CheXpertDataset(zip_path=configs["zip_path"],csv_path=configs["val_csv"],root_dir=configs["datadir"],augment=False,use_frontal_only=True )
+        self.class_Weights=self.train_dataset.get_class_weights().to(self.configs["device"])
+        self.sample_Weights=self.train_dataset.get_sample_weights()
+        self.sampler=torch.utils.data.WeightedRandomSampler(self.sample_Weights,num_samples=len(self.sample_Weights))
+        self.trainloader=DataLoader(self.train_dataset,batch_size=configs["batch_size"],sampler=self.sampler,num_workers=8,pin_memory=True,persistent_workers=True)
+        self.valloader=DataLoader(self.val_dataset,batch_size=configs["batch_size"],shuffle=False,num_workers=8,pin_memory=True,persistent_workers=True)
+        self.history={"train_loss":[],"val_loss":[]}
+
+        self.current_epoch=0
+
+        if os.path.exists(self.configs["resume"]):
+            loadedpickle=torch.load(self.configs["resume"],map_location=self.configs["device"])
+            self.model.load_state_dict(loadedpickle["model"],strict=False)
+            self.optimizer.load_state_dict(loadedpickle["optimizer"])
+            self.schedular.load_state_dict(loadedpickle["schedular"])
+            self.schedular1.load_state_dict(loadedpickle["schedular1"])
+            self.schedular2.load_state_dict(loadedpickle["schedular2"])
+            self.scaler.load_state_dict(loadedpickle["scaler"])
+            self.current_epoch=loadedpickle["epoch"]+1
+
+
+
+        self.test_dataset = None
+        self.testloader   = None
+        if configs.get("test_csv"):
+            self.test_dataset = CheXpertDataset(
+                zip_path=configs["zip_path"],
+                csv_path=configs["test_csv"],
+                root_dir=configs["datadir"],
+                augment=False,
+                use_frontal_only=True
+            )
+            self.testloader = DataLoader(
+                self.test_dataset,
+                batch_size=configs["batch_size"],
+                shuffle=False,
+                num_workers=8,
+                pin_memory=True,
+                persistent_workers=True
+            )
+            print(f"Test loader ready – {len(self.test_dataset)} images")
+
+            torch.backends.cudnn.benchmark = True
+            torch.backends.cudnn.enabled = True
+
+            # FIX: Set memory allocator settings
+            os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'
+
+            # FIX: Enable gradient checkpointing if model supports it
+            if hasattr(self.model, 'enable_gradient_checkpointing'):
+                self.model.enable_gradient_checkpointing()
+    @staticmethod
+    def plot_training_metrics(metrics, epoch,figs_path):
+        import matplotlib.pyplot as plt
+        """
+        Plot loss and AUC curves from training metrics.
+
+        Args:
+            metrics (dict): Dictionary containing lists for each metric key:
+                {
+                    "train_loss": [...],
+                    "val_loss": [...]
+                }
+            epoch (int): Current epoch number (used for title or axis scaling)
+        """
+        epochs = list(range(1, epoch + 1))
+
+        #Compute the common length across all series
+        keys = ["train_loss","val_loss"]
+        lengths = [len(metrics[k]) for k in keys if k in metrics]
+        if not lengths:
+            return
+        n = min(lengths)
+
+        # Slice everything to the same length
+        m = {k: metrics[k][:n] for k in keys if k in metrics}
+        epochs = list(range(1, n + 1))
+
+        plt.figure(figsize=(14, 6))
+
+
+        # ---- Loss subplot ----
+        plt.subplot(1, 2, 1)
+        plt.plot(epochs, metrics["train_loss"], label="Train Loss", marker='o')
+        plt.plot(epochs, metrics["val_loss"], label="Val Loss", marker='s')
+        plt.xlabel("Epoch")
+        plt.ylabel("Loss")
+        plt.title("Training & Validation Loss")
+        plt.legend()
+        plt.grid(True, linestyle='--', alpha=0.6)
+
+
+        plt.tight_layout()
+        os.makedirs(os.path.join(figs_path,str(epoch)),exist_ok=True)
+        plt.savefig(os.path.join(figs_path,str(epoch),"metrics.png"))
+        plt.show()
+
+    def train_epoch(self, epoch, looper):
+        self.model.train()
+        running_loss = 0.0
+        all_preds = []
+        all_targets = []
+        current_loss=0
+        total_batches = len(self.trainloader)
+
+        for batch_idx, data in looper:
+            image = data['image'].to(self.configs["device"], non_blocking=True)
+            target = data['labels'].to(self.configs["device"], non_blocking=True)
+
+            with torch.autocast(device_type=self.configs["device"].type, dtype=torch.float16):
+                img,preds,mask = self.model(image)
+                loss = self.criterion(img,preds,mask)
+
+            loss_back = loss / self.configs["accumulation"]
+            running_loss += loss.item()
+
+            if torch.isfinite(loss):
+                #loss_back.backward()
+                self.scaler.scale(loss_back).backward()
+            else:
+                self.optimizer.zero_grad(set_to_none=True)
+                continue
+
+            if (batch_idx + 1) % self.configs["accumulation"] == 0 or batch_idx == total_batches - 1:
+                self.scaler.unscale_(self.optimizer)
+                torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)
+                self.scaler.step(self.optimizer)
+                self.scaler.update()
+                #self.optimizer.step()
+                self.schedular.step()
+                self.optimizer.zero_grad(set_to_none=True)
+
+
+            # === LIVE METRICS (every batch) ===
+            current_loss = running_loss / (batch_idx + 1)
+            if (batch_idx + 1) % 10 == 0:
+                current_lr = self.optimizer.param_groups[0]['lr']
+                looper.set_postfix({
+                    "lr": f"{current_lr:.2e}","batch":f"{batch_idx}/{total_batches}",
+                    "epoch": f"{epoch}/{self.configs['num_epochs']}",
+                    "loss": f"{current_loss:.3f}",
+                })
+
+        return current_loss
+    def validate(self, epoch, looper):
+        self.model.eval()
+        val_loss = 0.0
+        all_preds = []
+        all_targets = []
+        lenloader=len(self.valloader)
+        current_loss=0
+        with torch.no_grad():
+            for batch_idx, data in looper:
+                image = data["image"].to(self.configs["device"], non_blocking=True)
+                target = data["labels"].to(self.configs["device"], non_blocking=True)
+
+                with torch.autocast(device_type=self.configs["device"].type, dtype=torch.float16):
+                    img,preds,mask = self.model(image)
+                    loss = self.criterion(img,preds,mask)
+
+                val_loss += loss.item()
+
+                # === LIVE METRICS ===
+                current_loss = val_loss / (batch_idx + 1)
+                if (batch_idx + 1) % 10 == 0 :
+
+                    looper.set_postfix({
+                        "epoch": f"{epoch}/{self.configs['num_epochs']}",
+                        "batch":f"{batch_idx}/{lenloader}",
+                        "loss": f"{current_loss:.3f}",
+                    })
+
+        return current_loss
+    def train(self):
+
+        for epoch in range(self.current_epoch,self.configs["num_epochs"]):
+            trainlooper=tqdm(enumerate(self.trainloader),desc="training: ", leave=False,file=self.traintee)
+            vallooper=tqdm(enumerate(self.valloader),desc="validating: ",leave=False,file=self.valtee)
+
+
+            self.model.train()
+            self.optimizer.zero_grad(set_to_none=True)
+
+            running_loss=self.train_epoch(epoch,trainlooper)
+
+            torch.cuda.synchronize()
+            torch.cuda.empty_cache()
+
+            val_loss=self.validate(epoch,vallooper)
+
+            torch.cuda.synchronize()
+            torch.cuda.empty_cache()
+
+            gc.collect()
+
+            if (self.history["val_loss"] and (val_loss<min(self.history["val_loss"]))) :
+                checkpoint={"model":self.model.state_dict(),"optimizer":self.optimizer.state_dict(),"schedular":self.schedular.state_dict(),"schedular1":self.schedular1.state_dict(),"schedular2":self.schedular2.state_dict(),"scaler":self.scaler.state_dict(),"epoch":epoch}
+                torch.save(checkpoint, self.configs["resume"])
+
+            print(f"train loss {running_loss} val loss {val_loss}")
+
+            self.history["train_loss"].append(float(running_loss))
+            self.history["val_loss"].append(float(val_loss))
+
+            if epoch%10==0:
+                historyfile=os.path.join(self.configs["logdir"],"history.json")
+                if os.path.exists(historyfile):
+                    with open(historyfile,"r") as f:
+                      history=json.load(f)
+                      history["train_loss"]+=self.history["train_loss"]
+                      history["val_loss"]+=self.history["val_loss"]
+                with open(historyfile,"w") as f:
+                    json.dump(self.history,f)
+                    f.close()
+                MAETrainer.plot_training_metrics(self.history,epoch+1,self.configs["logdir"])
+
+            self.current_epoch=epoch

training logs/mae/history.json

@@ -0,0 +1 @@
+{"train_loss": [61.882596965502664, 44.471386281221996, 34.87869473939301, 27.54925524051899, 24.282669028415476, 22.451558465855097, 21.147514946913635, 20.99175854891432, 20.213269732075354, 20.001737736117455, 19.91450946014842, 19.47891389648547, 19.18175616794162, 18.748555672895098, 19.04294045150921, 18.45393469068739, 18.838688672315264, 18.155403604131447, 18.37196905074581, 17.72255533279911, 18.3023255009805, 17.781992453742625, 18.199740923262837, 17.737673626834773, 18.033451236875255, 17.423879869522587, 17.789273495144315, 17.71034824754175, 17.673153098253366, 17.505879045301867, 17.242074561717263, 17.15336024376654, 16.90521138372387, 16.933767681053464, 17.133027586714768, 17.32885531353694, 17.239427022352867, 17.082445266340795, 17.198882104333585, 16.797237842655523, 16.80550524462081, 16.855286646060193, 16.820361117564648, 16.804262694492135, 16.60529948887432, 16.617685430109713, 16.49680861811484, 16.534883691247646, 16.66711660952551, 16.68186878833292, 16.830572265365216, 16.551616942156173, 16.58498664637193, 16.517176605552756, 16.586636129673238, 16.610006309153786, 16.467580378269208, 16.335460134390008, 16.45028829335312, 16.415938852508436, 16.657625508052046, 16.622210603119225, 16.319772068146737, 16.374398973253037, 16.252494745015245, 16.311780229520625, 16.422519305102714, 16.033588693133392, 16.024791444757934, 16.087340885422137, 16.039379536208287, 16.26473860928662, 16.34161920308212, 15.996231590804234, 16.295430011817633, 16.445986707407087, 16.343918548775402, 16.409462545251333, 16.49581729998298, 16.137155871921117, 16.05842663481244, 16.16612617533694, 16.27624153595244, 16.29507503646249, 16.29731023747434, 16.399930175316378, 16.08117872668851, 16.119801326464582, 16.0585214939596, 15.990199300977919, 16.033912498036592, 16.225505850050183, 16.006062768095283, 16.016956458553192, 15.915514986318499, 16.111989719978798, 15.927976318414066, 16.02773256541153, 15.936725686186103, 15.84361021441798, 15.960153004089136], "val_loss": [31.570541223418278, 22.954840935741945, 14.442683501893104, 9.339503538568946, 8.24503136790076, 6.798558349229173, 7.171370674209341, 6.004779509927744, 6.1726217491682185, 5.98896356120062, 5.595671336912238, 5.563971098079238, 5.132711923795681, 5.617544600337843, 4.961312991044054, 5.048685019990534, 5.025379383682808, 4.725488581134631, 4.978387813631482, 4.562931300793771, 4.850497535692893, 4.521047054335129, 4.649226747081921, 4.5786320276038595, 4.333143504355041, 4.4852356395848165, 4.235697840535363, 4.414844192935779, 4.24225838635847, 4.289312726239429, 4.266240818555965, 4.077501735021902, 4.319235841301192, 4.025318107731715, 4.189943225676831, 4.086437865349145, 4.028652789980867, 4.119319666263669, 3.907484631205714, 4.0297732788859015, 3.884794211466843, 4.035012978651991, 3.96724297437953, 3.852650080804413, 3.9237460725727273, 3.7834923750538367, 3.9779289901454584, 3.7871727095885928, 3.838316381967741, 3.852536988020735, 3.7242261183222265, 3.889924947605577, 3.681634605920988, 3.851548166370075, 3.699327661349528, 3.693325303321661, 3.7589161586127804, 3.61661579759414, 3.7395904396855553, 3.5934903867220958, 3.6865477134223, 3.6255838419511863, 3.5961770504416024, 3.650220192152004, 3.5313091785012687, 3.658342431153966, 3.5214638036746915, 3.5825591047736896, 3.535893441830759, 3.4996724699026722, 3.600268395636169, 3.4633193554672292, 3.5713670332962493, 3.4522710076202188, 3.5278821134092007, 3.5031054748649217, 3.426014715650945, 3.5055409483735347, 3.403536310227606, 3.5005639573664364, 3.434894419983772, 3.439712012725019, 3.4542255861022544, 3.3616595402904523, 3.4835145457638457, 3.351115513481571, 3.4225067292337004, 3.3756711308742284, 3.367003402044607, 3.418841015064835, 3.3144125257219588, 3.4290886646093326, 3.2998156674280517, 3.368062291826521, 3.337226649851498, 3.2932771796799973, 3.3656438268300306, 3.266660438423537, 3.3932779888774074, 3.2590805264406426, 3.298969637119889]}