Upload folder using huggingface_hub

DELULU_MODEL_CARD.md

@@ -0,0 +1,326 @@
+---
+license: cc-by-nc-nd-4.0
+language:
+  - en
+library_name: transformers
+tags:
+  - speaker-verification
+  - speaker-diarization
+  - speaker-profiling
+  - speech
+  - audio
+  - self-supervised-learning
+  - ssl
+  - hubert
+  - speech-representation
+  - pytorch
+  - deep-learning
+datasets:
+  - librispeech_asr
+metrics:
+  - eer
+pipeline_tag: audio-classification
+model-index:
+  - name: DELULU
+    results:
+      - task:
+          type: speaker-verification
+          name: Speaker Verification
+        dataset:
+          type: VoxCeleb1-O
+          name: VoxCeleb1-O
+        metrics:
+          - type: eer
+            value: 13.52
+            name: Equal Error Rate (Upstream)
+---
+
+# DELULU: Discriminative Embedding Learning Using Latent Units
+
+<div align="center">
+
+**A Speaker-Aware Self-Supervised Speech Foundational Model**
+
+[![Paper](https://img.shields.io/badge/arXiv-2510.17662-b31b1b.svg)](https://arxiv.org/abs/2510.17662)
+[![License](https://img.shields.io/badge/License-CC%20BY--NC--ND%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-nd/4.0/)
+
+</div>
+
+## Model Description
+
+**DELULU** (Discriminative Embedding Learning Using Latent Units) is a speaker-aware self-supervised speech foundational model that addresses a critical limitation of existing SSL models: their inability to capture speaker-discriminative features essential for verification, diarization, and profiling applications.
+
+While conventional SSL models like HuBERT, wav2vec 2.0, and WavLM excel at content-driven tasks (ASR, speech recognition), they learn representations optimized for phonetic/linguistic content, inadvertently discarding speaker identity information. DELULU bridges this gap by integrating external speaker supervision into the pseudo-label generation process.
+
+### Key Innovation
+
+DELULU introduces a novel approach to self-supervised speech learning by leveraging **frame-level embeddings from ReDimNet**, a state-of-the-art speaker verification model, to guide the k-means clustering step during pre-training. This introduces a strong **speaker-discriminative inductive bias** that aligns representation learning with speaker identity—a fundamental shift from content-focused SSL paradigms.
+
+## Architecture
+
+DELULU is based on the HuBERT architecture with a **modified convolutional feature extractor** optimized for speaker verification:
+
+### Convolutional Feature Extractor
+
+| Layer | Channels | Kernel Size | Stride |
+|-------|----------|-------------|--------|
+| 1     | 512      | 10          | **4**  |
+| 2     | 512      | 3           | 2      |
+| 3     | 512      | 3           | 2      |
+| 4     | 512      | 3           | 2      |
+| 5     | 512      | 3           | 2      |
+| 6     | 512      | 2           | 2      |
+| 7     | 512      | 2           | 2      |
+
+> **Key Difference**: The first layer uses stride **4** (vs. stride 5 in standard HuBERT), resulting in a **16ms frame shift** optimized for speaker verification tasks.
+
+### Transformer Encoder
+
+- **Hidden size**: 768
+- **Attention heads**: 12
+- **Layers**: 12
+- **Intermediate size**: 3,072
+- **Frame shift**: 16ms (vs. 20ms in HuBERT)
+
+### Training Configuration
+
+- **Clustering**: ReDimNet-guided k-means with k=256 clusters
+- **Feature dimension**: 2,304 (ReDimNet frame-level embeddings)
+- **Training objective**: Dual objective combining masked prediction + denoising
+- **Pre-training data**: LibriSpeech 960h
+- **Training steps**: 400k updates
+
+## Performance
+
+### Upstream Speaker Verification (Zero-Shot)
+
+| Model | VoxCeleb1-O EER (%) |
+|-------|---------------------|
+| wav2vec 2.0 | 37.21 |
+| HuBERT | 34.05 |
+| WavLM | 29.84 |
+| **DELULU** | **13.52** |
+
+> **62% relative improvement** over standard HuBERT in equal error rate.
+
+### Ablation: Why ReDimNet-Guided Clustering?
+
+| Clustering Features | k | EER (%) |
+|---------------------|---|---------|
+| MFCC | 100 | 37.73 |
+| HuBERT (pretrained) | 500 | 34.05 |
+| **ReDimNet** | 256 | **13.53** |
+
+ReDimNet-guided pseudo-labels provide a **60% relative improvement** over HuBERT's acoustic-only approach.
+
+### Demographic Robustness
+
+DELULU consistently outperforms baselines across all demographic groups, with particularly strong improvements for challenging subgroups:
+
+| Demographic | HuBERT EER (%) | DELULU EER (%) | Improvement |
+|-------------|----------------|----------------|-------------|
+| Male 36-45  | 39.47 | 24.53 | 38% |
+| All groups  | Varies | Consistent | ✓ |
+
+### Zero-Shot Speaker Profiling (DynamicSUPERB)
+
+DELULU excels on multiple speaker-related tasks without fine-tuning:
+- Gender classification
+- Age estimation
+- Accent recognition
+- Speaker counting
+- Spoof detection
+
+## Intended Uses
+
+### Primary Use Cases
+
+1. **Speaker Verification**: Verify whether two speech samples are from the same speaker
+2. **Speaker Diarization**: Segment and cluster speech by speaker identity
+3. **Speaker Profiling**: Extract demographic attributes (age, gender, accent)
+4. **Forensic Audio Analysis**: Speaker identification in investigative contexts
+
+### Downstream Applications
+
+- Voice biometrics and authentication systems
+- Meeting transcription with speaker labels
+- Call center analytics
+- Content personalization based on speaker identity
+- Multi-speaker dialogue systems
+
+## How to Use
+
+### Installation
+
+```bash
+pip install transformers torch torchaudio
+```
+
+### Loading the Model
+
+```python
+import torch
+from transformers import AutoModel, AutoConfig
+
+# Load DELULU model
+model = AutoModel.from_pretrained("username/DELULU", trust_remote_code=True)
+model.eval()
+```
+
+### Feature Extraction
+
+```python
+import torchaudio
+
+# Load audio (16kHz sampling rate required)
+waveform, sample_rate = torchaudio.load("audio.wav")
+if sample_rate != 16000:
+    resampler = torchaudio.transforms.Resample(sample_rate, 16000)
+    waveform = resampler(waveform)
+
+# Extract features
+with torch.no_grad():
+    outputs = model(waveform)
+    # Use last hidden state for downstream tasks
+    features = outputs.last_hidden_state  # [batch, time, 768]
+    
+    # For speaker verification, typically use mean pooling
+    speaker_embedding = features.mean(dim=1)  # [batch, 768]
+```
+
+### Speaker Verification Example
+
+```python
+import torch.nn.functional as F
+
+def compute_similarity(embedding1, embedding2):
+    """Compute cosine similarity between two speaker embeddings."""
+    return F.cosine_similarity(embedding1, embedding2, dim=-1)
+
+# Extract embeddings for two audio samples
+emb1 = extract_embedding(model, audio1)
+emb2 = extract_embedding(model, audio2)
+
+# Compute similarity score
+similarity = compute_similarity(emb1, emb2)
+print(f"Similarity score: {similarity.item():.4f}")
+
+# Threshold-based decision (tune threshold on validation data)
+threshold = 0.7
+same_speaker = similarity > threshold
+```
+
+### Fine-Tuning for Downstream Tasks
+
+```python
+from transformers import Trainer, TrainingArguments
+
+# Add task-specific head
+class SpeakerVerificationModel(torch.nn.Module):
+    def __init__(self, base_model, embedding_dim=256):
+        super().__init__()
+        self.base = base_model
+        self.projector = torch.nn.Linear(768, embedding_dim)
+        
+    def forward(self, x):
+        features = self.base(x).last_hidden_state
+        pooled = features.mean(dim=1)
+        return self.projector(pooled)
+
+# Fine-tune with your speaker verification dataset
+model = SpeakerVerificationModel(base_model)
+```
+
+## Training Details
+
+### Pre-training Process
+
+1. **Pseudo-Label Generation**:
+   - Extract frame-level embeddings using ReDimNet (dimension: 2,304)
+   - Apply k-means clustering with k=256 to create speaker-aware pseudo-labels
+   - ReDimNet stride modified to match encoder stride (16ms)
+
+2. **Training Objective**:
+   - **Masked Prediction**: Predict pseudo-labels for masked frames
+   - **Denoising**: Additional denoising objective for robustness
+
+3. **Optimization**:
+   - Training data: LibriSpeech 960 hours
+   - Training steps: 400k updates
+   - Batch size: 87.5 seconds of audio per GPU
+   - Hardware: 32 GPUs
+
+### Why 16ms Frame Shift?
+
+Ablation studies showed that **16ms stride achieves optimal EER (13.52%)**, while both lower (≤15ms) and higher (≥20ms) strides resulted in EER >14%. This precise temporal resolution balances:
+- Fine-grained speaker characteristics capture
+- Computational efficiency
+- Training stability
+
+## Limitations
+
+1. **Domain Shift**: Performance may degrade on audio with characteristics significantly different from LibriSpeech (e.g., noisy environments, non-English speech, telephony audio)
+
+2. **Computational Requirements**: As a transformer-based model, DELULU requires substantial computational resources for inference on long audio
+
+3. **Fine-tuning May Be Required**: While DELULU provides strong zero-shot speaker representations, task-specific fine-tuning typically improves performance
+
+4. **Language**: Pre-trained on English speech; cross-lingual transfer may be limited
+
+## Ethical Considerations
+
+### Potential Misuse
+
+Speaker verification technology can be misused for:
+- Unauthorized surveillance
+- Privacy violations
+- Identity fraud
+- Discriminatory profiling
+
+### Recommended Safeguards
+
+- Obtain explicit consent before processing voice data
+- Implement robust access controls
+- Follow data protection regulations (GDPR, CCPA)
+- Conduct bias audits across demographic groups
+- Maintain transparency about system capabilities and limitations
+
+### Bias Evaluation
+
+DELULU was evaluated across demographic subgroups and shows consistent improvements without introducing systematic biases. However, users should validate performance on their specific populations.
+
+## Citation
+
+If you use DELULU in your research, please cite:
+
+```bibtex
+@article{baali2025delulu,
+  title={DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Supervised Speech Foundational Model},
+  author={Baali, Massa and Singh, Rita and Raj, Bhiksha},
+  journal={arXiv preprint arXiv:2510.17662},
+  year={2025}
+}
+```
+
+## Related Work
+
+- **HuBERT**: [Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447)
+- **ReDimNet**: State-of-the-art speaker verification model used for pseudo-label generation
+- **WavLM**: [Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://arxiv.org/abs/2110.13900)
+
+## Acknowledgments
+
+This work was conducted at Carnegie Mellon University's Language Technologies Institute. We thank the speech processing community for foundational work on self-supervised learning and speaker verification.
+
+## Contact
+
+For questions about the model or paper:
+- **Author**: Massa Baali
+- **Advisors**: Prof. Rita Singh, Prof. Bhiksha Raj
+- **Institution**: Carnegie Mellon University, Language Technologies Institute
+
+---
+
+<div align="center">
+<i>DELULU: Where Self-Supervised Learning Meets Speaker Identity</i>
+</div>

config.json

@@ -1,39 +1,13 @@
 {
   "model_type": "delulu",
-  "architectures": [
-    "DELULUModel"
-  ],
+  "architectures": ["DELULUModel"],
   "auto_map": {
     "AutoConfig": "configuration_delulu.DELULUConfig",
     "AutoModel": "modeling_delulu.DELULUModel"
   },
-  "conv_dim": [
-    512,
-    512,
-    512,
-    512,
-    512,
-    512,
-    512
-  ],
-  "conv_kernel": [
-    10,
-    3,
-    3,
-    3,
-    3,
-    2,
-    2
-  ],
-  "conv_stride": [
-    4,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2
-  ],
+  "conv_dim": [512, 512, 512, 512, 512, 512, 512],
+  "conv_kernel": [10, 3, 3, 3, 3, 2, 2],
+  "conv_stride": [4, 2, 2, 2, 2, 2, 2],
   "conv_bias": false,
   "extractor_mode": "group_norm",
   "hidden_size": 768,
@@ -44,7 +18,7 @@
   "attention_dropout": 0.1,
   "final_dropout": 0.1,
   "feat_proj_dropout": 0.1,
-  "layer_norm_eps": 1e-05,
+  "layer_norm_eps": 1e-5,
   "layer_drop": 0.05,
   "num_conv_pos_embeddings": 128,
   "num_conv_pos_embedding_groups": 16,
@@ -56,5 +30,6 @@
   "pad_token_id": 0,
   "bos_token_id": 1,
   "eos_token_id": 2,
+  "transformers_version": "4.36.0",
   "torch_dtype": "float32"
-}
+}

configuration_delulu.py

@@ -1,20 +1,80 @@
-"""DELULU Configuration"""
+"""
+DELULU Configuration
+
+Configuration class for DELULU (Discriminative Embedding Learning Using Latent Units),
+a speaker-aware self-supervised speech foundational model.
+
+Paper: https://arxiv.org/abs/2510.17662
+Authors: Massa Baali, Rita Singh, Bhiksha Raj
+"""
 
 from transformers import PretrainedConfig
 
 
 class DELULUConfig(PretrainedConfig):
-    """Configuration class for DELULU model."""
+    r"""
+    Configuration class for DELULU model.
+    
+    DELULU is based on HuBERT architecture with modified convolutional strides
+    optimized for speaker verification (16ms frame shift).
+    
+    Args:
+        conv_dim (`List[int]`, *optional*, defaults to `[512, 512, 512, 512, 512, 512, 512]`):
+            Dimensions of each convolutional layer in the feature extractor.
+        conv_kernel (`List[int]`, *optional*, defaults to `[10, 3, 3, 3, 3, 2, 2]`):
+            Kernel sizes of each convolutional layer in the feature extractor.
+        conv_stride (`List[int]`, *optional*, defaults to `[4, 2, 2, 2, 2, 2, 2]`):
+            Stride sizes of each convolutional layer. Note: first stride is 4 (vs 5 in HuBERT)
+            for 16ms frame shift optimized for speaker verification.
+        conv_bias (`bool`, *optional*, defaults to `False`):
+            Whether to use bias in convolutional layers.
+        hidden_size (`int`, *optional*, defaults to 768):
+            Dimensionality of the encoder layers and pooler layer.
+        num_hidden_layers (`int`, *optional*, defaults to 12):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 12):
+            Number of attention heads for each attention layer.
+        intermediate_size (`int`, *optional*, defaults to 3072):
+            Dimensionality of the feed-forward layer in the Transformer encoder.
+        hidden_dropout (`float`, *optional*, defaults to 0.1):
+            Dropout probability for all fully connected layers.
+        attention_dropout (`float`, *optional*, defaults to 0.1):
+            Dropout probability for attention weights.
+        feat_proj_dropout (`float`, *optional*, defaults to 0.1):
+            Dropout probability for feature projection layer.
+        layer_drop (`float`, *optional*, defaults to 0.05):
+            Layer drop probability during training.
+        num_conv_pos_embeddings (`int`, *optional*, defaults to 128):
+            Number of convolutional positional embeddings.
+        num_conv_pos_embedding_groups (`int`, *optional*, defaults to 16):
+            Number of groups for convolutional positional embeddings.
+        sampling_rate (`int`, *optional*, defaults to 16000):
+            Audio sampling rate in Hz.
+        
+    Example:
+        ```python
+        from transformers import AutoConfig, AutoModel
+        
+        # Load config
+        config = AutoConfig.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)
+        
+        # Load model
+        model = AutoModel.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)
+        ```
+    """
     
     model_type = "delulu"
     
     def __init__(
         self,
+        # Convolutional feature extractor
         conv_dim=None,
         conv_kernel=None,
         conv_stride=None,
         conv_bias=False,
         extractor_mode="group_norm",
+        
+        # Transformer encoder
         hidden_size=768,
         num_hidden_layers=12,
         num_attention_heads=12,
@@ -25,15 +85,24 @@ class DELULUConfig(PretrainedConfig):
         feat_proj_dropout=0.1,
         layer_norm_eps=1e-5,
         layer_drop=0.05,
+        
+        # Positional encoding
         num_conv_pos_embeddings=128,
         num_conv_pos_embedding_groups=16,
+        
+        # Audio settings
         sampling_rate=16000,
         do_stable_layer_norm=False,
+        
+        # DELULU-specific settings
         num_clusters=256,
         feature_type="redimnet",
+        
+        # Pad token for compatibility
         pad_token_id=0,
         bos_token_id=1,
         eos_token_id=2,
+        
         **kwargs
     ):
         super().__init__(
@@ -43,10 +112,18 @@ class DELULUConfig(PretrainedConfig):
             **kwargs
         )
         
-        # DELULU conv config: [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
-        self.conv_dim = conv_dim or [512, 512, 512, 512, 512, 512, 512]
-        self.conv_kernel = conv_kernel or [10, 3, 3, 3, 3, 2, 2]
-        self.conv_stride = conv_stride or [4, 2, 2, 2, 2, 2, 2]
+        # Set default DELULU conv configuration
+        # Key difference from HuBERT: first stride is 4 instead of 5
+        if conv_dim is None:
+            conv_dim = [512, 512, 512, 512, 512, 512, 512]
+        if conv_kernel is None:
+            conv_kernel = [10, 3, 3, 3, 3, 2, 2]
+        if conv_stride is None:
+            conv_stride = [4, 2, 2, 2, 2, 2, 2]
+        
+        self.conv_dim = conv_dim
+        self.conv_kernel = conv_kernel
+        self.conv_stride = conv_stride
         self.conv_bias = conv_bias
         self.extractor_mode = extractor_mode
         
@@ -70,4 +147,13 @@ class DELULUConfig(PretrainedConfig):
         self.num_clusters = num_clusters
         self.feature_type = feature_type
         
+        # Computed properties
         self.num_feat_extract_layers = len(self.conv_dim)
+    
+    @property
+    def inputs_to_logits_ratio(self):
+        """Compute the ratio between input samples and output frames."""
+        ratio = 1
+        for stride in self.conv_stride:
+            ratio *= stride
+        return ratio

convert_delulu_fixed.py

@@ -0,0 +1,426 @@
+#!/usr/bin/env python3
+"""
+DELULU Checkpoint Converter - Fixed Version
+
+Converts DELULU model checkpoints from torchaudio/PyTorch Lightning format
+to Hugging Face compatible format with proper metadata.
+
+Usage:
+    python convert_delulu_fixed.py \
+        --checkpoint /path/to/epoch=45-step=400000.ckpt \
+        --output-dir ./delulu_hf_model
+
+Author: Massa Baali
+"""
+
+import argparse
+import json
+import os
+import sys
+from collections import OrderedDict
+from pathlib import Path
+
+import torch
+
+try:
+    from safetensors.torch import save_file as save_safetensors
+    SAFETENSORS_AVAILABLE = True
+except ImportError:
+    SAFETENSORS_AVAILABLE = False
+    print("Warning: safetensors not installed. Install with: pip install safetensors")
+
+
+def load_lightning_checkpoint(checkpoint_path: str) -> dict:
+    """Load and clean PyTorch Lightning checkpoint."""
+    print(f"Loading checkpoint: {checkpoint_path}")
+    
+    checkpoint = torch.load(checkpoint_path, map_location="cpu")
+    
+    # Extract state dict
+    if "state_dict" in checkpoint:
+        state_dict = checkpoint["state_dict"]
+    else:
+        state_dict = checkpoint
+    
+    # Clean up state dict keys
+    cleaned_state_dict = OrderedDict()
+    
+    for key, value in state_dict.items():
+        new_key = key
+        
+        # Remove Lightning prefixes
+        if key.startswith("model.wav2vec2."):
+            new_key = key.replace("model.wav2vec2.", "")
+        elif key.startswith("model."):
+            new_key = key.replace("model.", "")
+        
+        # Skip auxiliary heads
+        if "aux" in new_key:
+            print(f"  Skipping: {key}")
+            continue
+        
+        cleaned_state_dict[new_key] = value
+    
+    print(f"Loaded {len(cleaned_state_dict)} parameters")
+    return cleaned_state_dict
+
+
+def save_pytorch_model_bin(state_dict: dict, output_path: Path):
+    """
+    Save state dict as pytorch_model.bin with proper format.
+    
+    This saves ONLY the state dict (not a full checkpoint with metadata),
+    which is what HuggingFace expects.
+    """
+    print(f"Saving pytorch_model.bin to: {output_path}")
+    
+    # Convert all tensors to contiguous for safety
+    clean_state_dict = OrderedDict()
+    for key, value in state_dict.items():
+        if isinstance(value, torch.Tensor):
+            clean_state_dict[key] = value.contiguous()
+        else:
+            clean_state_dict[key] = value
+    
+    # Save just the state dict (NOT a checkpoint dict)
+    torch.save(clean_state_dict, output_path)
+    
+    print(f"  Saved {len(clean_state_dict)} tensors")
+    print(f"  File size: {output_path.stat().st_size / 1024 / 1024:.2f} MB")
+
+
+def save_safetensors_model(state_dict: dict, output_path: Path):
+    """Save state dict in safetensors format."""
+    if not SAFETENSORS_AVAILABLE:
+        print("Skipping safetensors (not installed)")
+        return
+    
+    print(f"Saving model.safetensors to: {output_path}")
+    
+    # Safetensors requires contiguous tensors
+    clean_state_dict = {}
+    for key, value in state_dict.items():
+        if isinstance(value, torch.Tensor):
+            clean_state_dict[key] = value.contiguous()
+    
+    save_safetensors(clean_state_dict, str(output_path))
+    print(f"  File size: {output_path.stat().st_size / 1024 / 1024:.2f} MB")
+
+
+def create_config_json(output_dir: Path):
+    """Create config.json with DELULU configuration."""
+    config = {
+        "model_type": "delulu",
+        "architectures": ["DELULUModel"],
+        "auto_map": {
+            "AutoConfig": "configuration_delulu.DELULUConfig",
+            "AutoModel": "modeling_delulu.DELULUModel"
+        },
+        "conv_dim": [512, 512, 512, 512, 512, 512, 512],
+        "conv_kernel": [10, 3, 3, 3, 3, 2, 2],
+        "conv_stride": [4, 2, 2, 2, 2, 2, 2],
+        "conv_bias": False,
+        "extractor_mode": "group_norm",
+        "hidden_size": 768,
+        "num_hidden_layers": 12,
+        "num_attention_heads": 12,
+        "intermediate_size": 3072,
+        "hidden_dropout": 0.1,
+        "attention_dropout": 0.1,
+        "final_dropout": 0.1,
+        "feat_proj_dropout": 0.1,
+        "layer_norm_eps": 1e-5,
+        "layer_drop": 0.05,
+        "num_conv_pos_embeddings": 128,
+        "num_conv_pos_embedding_groups": 16,
+        "sampling_rate": 16000,
+        "do_stable_layer_norm": False,
+        "num_clusters": 256,
+        "feature_type": "redimnet",
+        "num_feat_extract_layers": 7,
+        "pad_token_id": 0,
+        "bos_token_id": 1,
+        "eos_token_id": 2,
+        "torch_dtype": "float32"
+    }
+    
+    config_path = output_dir / "config.json"
+    with open(config_path, "w") as f:
+        json.dump(config, f, indent=2)
+    print(f"Created config.json")
+
+
+def create_configuration_delulu(output_dir: Path):
+    """Create configuration_delulu.py file."""
+    code = '''"""DELULU Configuration"""
+
+from transformers import PretrainedConfig
+
+
+class DELULUConfig(PretrainedConfig):
+    """Configuration class for DELULU model."""
+    
+    model_type = "delulu"
+    
+    def __init__(
+        self,
+        conv_dim=None,
+        conv_kernel=None,
+        conv_stride=None,
+        conv_bias=False,
+        extractor_mode="group_norm",
+        hidden_size=768,
+        num_hidden_layers=12,
+        num_attention_heads=12,
+        intermediate_size=3072,
+        hidden_dropout=0.1,
+        attention_dropout=0.1,
+        final_dropout=0.1,
+        feat_proj_dropout=0.1,
+        layer_norm_eps=1e-5,
+        layer_drop=0.05,
+        num_conv_pos_embeddings=128,
+        num_conv_pos_embedding_groups=16,
+        sampling_rate=16000,
+        do_stable_layer_norm=False,
+        num_clusters=256,
+        feature_type="redimnet",
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        **kwargs
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            **kwargs
+        )
+        
+        # DELULU conv config: [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+        self.conv_dim = conv_dim or [512, 512, 512, 512, 512, 512, 512]
+        self.conv_kernel = conv_kernel or [10, 3, 3, 3, 3, 2, 2]
+        self.conv_stride = conv_stride or [4, 2, 2, 2, 2, 2, 2]
+        self.conv_bias = conv_bias
+        self.extractor_mode = extractor_mode
+        
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        self.hidden_dropout = hidden_dropout
+        self.attention_dropout = attention_dropout
+        self.final_dropout = final_dropout
+        self.feat_proj_dropout = feat_proj_dropout
+        self.layer_norm_eps = layer_norm_eps
+        self.layer_drop = layer_drop
+        
+        self.num_conv_pos_embeddings = num_conv_pos_embeddings
+        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
+        
+        self.sampling_rate = sampling_rate
+        self.do_stable_layer_norm = do_stable_layer_norm
+        
+        self.num_clusters = num_clusters
+        self.feature_type = feature_type
+        
+        self.num_feat_extract_layers = len(self.conv_dim)
+'''
+    
+    with open(output_dir / "configuration_delulu.py", "w") as f:
+        f.write(code)
+    print("Created configuration_delulu.py")
+
+
+def create_modeling_delulu(output_dir: Path):
+    """Create modeling_delulu.py file."""
+    code = '''"""DELULU Model"""
+
+import torch
+import torch.nn as nn
+from typing import Optional, Tuple, Union
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import BaseModelOutput
+from .configuration_delulu import DELULUConfig
+
+try:
+    from torchaudio.models.wav2vec2 import wav2vec2_model
+    TORCHAUDIO_AVAILABLE = True
+except ImportError:
+    TORCHAUDIO_AVAILABLE = False
+
+
+class DELULUModel(PreTrainedModel):
+    """
+    DELULU Model for speaker-aware speech representation learning.
+    
+    Example:
+        ```python
+        from transformers import AutoModel
+        import torch
+        
+        model = AutoModel.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)
+        waveform = torch.randn(1, 16000)  # 1 second at 16kHz
+        outputs = model(waveform)
+        features = outputs.last_hidden_state
+        ```
+    """
+    
+    config_class = DELULUConfig
+    base_model_prefix = "delulu"
+    main_input_name = "input_values"
+    
+    def __init__(self, config: DELULUConfig):
+        super().__init__(config)
+        self.config = config
+        
+        if not TORCHAUDIO_AVAILABLE:
+            raise ImportError("torchaudio is required. Install with: pip install torchaudio")
+        
+        # Build conv config
+        conv_layer_config = list(zip(
+            config.conv_dim,
+            config.conv_kernel,
+            config.conv_stride
+        ))
+        
+        # Create torchaudio model
+        self.wav2vec2 = wav2vec2_model(
+            extractor_mode=config.extractor_mode,
+            extractor_conv_layer_config=conv_layer_config,
+            extractor_conv_bias=config.conv_bias,
+            encoder_embed_dim=config.hidden_size,
+            encoder_projection_dropout=config.feat_proj_dropout,
+            encoder_pos_conv_kernel=config.num_conv_pos_embeddings,
+            encoder_pos_conv_groups=config.num_conv_pos_embedding_groups,
+            encoder_num_layers=config.num_hidden_layers,
+            encoder_num_heads=config.num_attention_heads,
+            encoder_attention_dropout=config.attention_dropout,
+            encoder_ff_interm_features=config.intermediate_size,
+            encoder_ff_interm_dropout=config.hidden_dropout,
+            encoder_dropout=config.hidden_dropout,
+            encoder_layer_norm_first=config.do_stable_layer_norm,
+            encoder_layer_drop=config.layer_drop,
+            aux_num_out=None,
+        )
+        
+        self.post_init()
+    
+    def _init_weights(self, module):
+        """Initialize weights."""
+        pass  # Handled by torchaudio
+    
+    def forward(
+        self,
+        input_values: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, BaseModelOutput]:
+        """
+        Args:
+            input_values: Audio waveform (batch, samples) at 16kHz
+            attention_mask: Optional attention mask
+            output_hidden_states: Whether to return all hidden states
+            return_dict: Whether to return BaseModelOutput
+        """
+        return_dict = return_dict if return_dict is not None else True
+        output_hidden_states = output_hidden_states if output_hidden_states is not None else False
+        
+        if input_values.dim() == 1:
+            input_values = input_values.unsqueeze(0)
+        
+        lengths = None
+        if attention_mask is not None:
+            lengths = attention_mask.sum(dim=-1)
+        
+        if output_hidden_states:
+            features, _ = self.wav2vec2.extract_features(input_values, lengths=lengths)
+            hidden_states = tuple(features)
+            last_hidden_state = features[-1]
+        else:
+            last_hidden_state, _ = self.wav2vec2(input_values, lengths=lengths)
+            hidden_states = None
+        
+        if not return_dict:
+            return (last_hidden_state, hidden_states) if hidden_states else (last_hidden_state,)
+        
+        return BaseModelOutput(
+            last_hidden_state=last_hidden_state,
+            hidden_states=hidden_states,
+        )
+    
+    def extract_features(self, input_values: torch.Tensor):
+        """Extract features from all layers."""
+        if input_values.dim() == 1:
+            input_values = input_values.unsqueeze(0)
+        features, _ = self.wav2vec2.extract_features(input_values)
+        return tuple(features)
+    
+    @classmethod
+    def _load_pretrained_model_low_mem(cls, *args, **kwargs):
+        """Override to handle custom loading."""
+        return super()._load_pretrained_model_low_mem(*args, **kwargs)
+'''
+    
+    with open(output_dir / "modeling_delulu.py", "w") as f:
+        f.write(code)
+    print("Created modeling_delulu.py")
+
+
+def convert_checkpoint(checkpoint_path: str, output_dir: str):
+    """Main conversion function."""
+    output_path = Path(output_dir)
+    output_path.mkdir(parents=True, exist_ok=True)
+    
+    print("=" * 60)
+    print("DELULU Checkpoint Converter")
+    print("=" * 60)
+    
+    # Step 1: Load checkpoint
+    state_dict = load_lightning_checkpoint(checkpoint_path)
+    
+    # Step 2: Print some keys for verification
+    print("\nSample keys in state dict:")
+    for i, key in enumerate(list(state_dict.keys())[:10]):
+        print(f"  {key}")
+    print(f"  ... and {len(state_dict) - 10} more")
+    
+    # Step 3: Save weights
+    save_pytorch_model_bin(state_dict, output_path / "pytorch_model.bin")
+    
+    if SAFETENSORS_AVAILABLE:
+        save_safetensors_model(state_dict, output_path / "model.safetensors")
+    
+    # Step 4: Create config and code files
+    create_config_json(output_path)
+    create_configuration_delulu(output_path)
+    create_modeling_delulu(output_path)
+    
+    # Step 5: Summary
+    print("\n" + "=" * 60)
+    print("Conversion Complete!")
+    print("=" * 60)
+    print(f"\nOutput directory: {output_path}")
+    print("\nFiles created:")
+    for f in sorted(output_path.iterdir()):
+        size_mb = f.stat().st_size / 1024 / 1024
+        print(f"  {f.name}: {size_mb:.2f} MB")
+    
+    print("\nNext steps:")
+    print("  1. Upload all files to huggingface.co/cmu-mlsp/DELULU")
+    print("  2. Test with:")
+    print('     model = AutoModel.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)')
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Convert DELULU checkpoint to HuggingFace format")
+    parser.add_argument("--checkpoint", "-c", required=True, help="Path to .ckpt file")
+    parser.add_argument("--output-dir", "-o", required=True, help="Output directory")
+    args = parser.parse_args()
+    
+    convert_checkpoint(args.checkpoint, args.output_dir)
+
+
+if __name__ == "__main__":
+    main()

convert_delulu_to_hf.py

@@ -0,0 +1,697 @@
+#!/usr/bin/env python3
+"""
+DELULU Checkpoint Converter
+===========================
+
+Converts DELULU model checkpoints from torchaudio/PyTorch Lightning format
+to Hugging Face compatible format (config.json + model weights).
+
+Usage:
+    python convert_delulu_to_hf.py \
+        --checkpoint /path/to/epoch=45-step=400000.ckpt \
+        --output-dir ./delulu_hf_model
+
+Author: Massa Baali
+Model: DELULU - Speaker-Aware Self-Supervised Speech Foundational Model
+"""
+
+import argparse
+import json
+import logging
+import os
+import sys
+from pathlib import Path
+from typing import Optional, Tuple, List
+from collections import OrderedDict
+
+import torch
+import torch.nn as nn
+
+try:
+    from safetensors.torch import save_file as save_safetensors
+    SAFETENSORS_AVAILABLE = True
+except ImportError:
+    SAFETENSORS_AVAILABLE = False
+    print("Warning: safetensors not installed. Will save as pytorch_model.bin only.")
+    print("Install with: pip install safetensors")
+
+# Configure logging
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s - %(levelname)s - %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+logger = logging.getLogger(__name__)
+
+
+# =============================================================================
+# DELULU Configuration
+# =============================================================================
+
+class DELULUConfig:
+    """
+    Configuration class for DELULU model.
+    
+    DELULU uses HuBERT architecture with modified convolutional strides
+    for 16ms frame shift, optimized for speaker verification.
+    """
+    
+    # Model architecture identifier
+    model_type = "delulu"
+    architectures = ["DELULUModel"]
+    
+    def __init__(
+        self,
+        # Convolutional feature extractor config
+        # DELULU: [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+        conv_dim: List[int] = None,
+        conv_kernel: List[int] = None,
+        conv_stride: List[int] = None,
+        conv_bias: bool = False,
+        extractor_mode: str = "group_norm",
+        
+        # Transformer encoder config
+        hidden_size: int = 768,
+        num_hidden_layers: int = 12,
+        num_attention_heads: int = 12,
+        intermediate_size: int = 3072,
+        hidden_dropout: float = 0.1,
+        attention_dropout: float = 0.1,
+        final_dropout: float = 0.1,
+        feat_proj_dropout: float = 0.1,
+        layer_norm_eps: float = 1e-5,
+        layer_drop: float = 0.05,
+        
+        # Positional encoding
+        num_conv_pos_embeddings: int = 128,
+        num_conv_pos_embedding_groups: int = 16,
+        
+        # Audio config
+        sampling_rate: int = 16000,
+        do_stable_layer_norm: bool = False,
+        
+        # Training config (for reference)
+        num_clusters: int = 256,
+        feature_type: str = "redimnet",
+        
+        **kwargs
+    ):
+        # Set default conv config for DELULU
+        if conv_dim is None:
+            conv_dim = [512, 512, 512, 512, 512, 512, 512]
+        if conv_kernel is None:
+            conv_kernel = [10, 3, 3, 3, 3, 2, 2]
+        if conv_stride is None:
+            conv_stride = [4, 2, 2, 2, 2, 2, 2]  # Key difference from HuBERT!
+        
+        self.conv_dim = conv_dim
+        self.conv_kernel = conv_kernel
+        self.conv_stride = conv_stride
+        self.conv_bias = conv_bias
+        self.extractor_mode = extractor_mode
+        
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        self.hidden_dropout = hidden_dropout
+        self.attention_dropout = attention_dropout
+        self.final_dropout = final_dropout
+        self.feat_proj_dropout = feat_proj_dropout
+        self.layer_norm_eps = layer_norm_eps
+        self.layer_drop = layer_drop
+        
+        self.num_conv_pos_embeddings = num_conv_pos_embeddings
+        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
+        
+        self.sampling_rate = sampling_rate
+        self.do_stable_layer_norm = do_stable_layer_norm
+        
+        self.num_clusters = num_clusters
+        self.feature_type = feature_type
+        
+        # Store any additional kwargs
+        for key, value in kwargs.items():
+            setattr(self, key, value)
+    
+    def to_dict(self) -> dict:
+        """Convert config to dictionary for JSON serialization."""
+        return {
+            # Model identification
+            "model_type": self.model_type,
+            "architectures": self.architectures,
+            
+            # Convolutional feature extractor
+            "conv_dim": self.conv_dim,
+            "conv_kernel": self.conv_kernel,
+            "conv_stride": self.conv_stride,
+            "conv_bias": self.conv_bias,
+            "extractor_mode": self.extractor_mode,
+            
+            # Transformer encoder
+            "hidden_size": self.hidden_size,
+            "num_hidden_layers": self.num_hidden_layers,
+            "num_attention_heads": self.num_attention_heads,
+            "intermediate_size": self.intermediate_size,
+            "hidden_dropout": self.hidden_dropout,
+            "attention_dropout": self.attention_dropout,
+            "final_dropout": self.final_dropout,
+            "feat_proj_dropout": self.feat_proj_dropout,
+            "layer_norm_eps": self.layer_norm_eps,
+            "layer_drop": self.layer_drop,
+            
+            # Positional encoding
+            "num_conv_pos_embeddings": self.num_conv_pos_embeddings,
+            "num_conv_pos_embedding_groups": self.num_conv_pos_embedding_groups,
+            
+            # Audio config
+            "sampling_rate": self.sampling_rate,
+            "do_stable_layer_norm": self.do_stable_layer_norm,
+            
+            # Training reference
+            "num_clusters": self.num_clusters,
+            "feature_type": self.feature_type,
+            
+            # Transformers compatibility
+            "transformers_version": "4.36.0",
+            "torch_dtype": "float32",
+            
+            # Auto-mapping for custom code
+            "auto_map": {
+                "AutoConfig": "configuration_delulu.DELULUConfig",
+                "AutoModel": "modeling_delulu.DELULUModel"
+            }
+        }
+    
+    def save_pretrained(self, save_directory: str):
+        """Save config to directory."""
+        os.makedirs(save_directory, exist_ok=True)
+        config_path = os.path.join(save_directory, "config.json")
+        with open(config_path, "w") as f:
+            json.dump(self.to_dict(), f, indent=2)
+        logger.info(f"Config saved to: {config_path}")
+
+
+# =============================================================================
+# Weight Mapping: torchaudio -> Hugging Face
+# =============================================================================
+
+def create_weight_mapping() -> dict:
+    """
+    Create mapping from torchaudio wav2vec2_model keys to Hugging Face format.
+    
+    torchaudio structure:
+        feature_extractor.conv_layers.{i}.{0,1,2}...
+        encoder.feature_projection.{projection,layer_norm}...
+        encoder.transformer.pos_conv_embed...
+        encoder.transformer.layers.{i}.{attention,feed_forward,layer_norms}...
+        encoder.transformer.layer_norm...
+    
+    HuggingFace structure:
+        feature_extractor.conv_layers.{i}.{conv,layer_norm}...
+        feature_projection.{projection,layer_norm}...
+        encoder.pos_conv_embed...
+        encoder.layers.{i}.{attention,feed_forward,layer_norm}...
+        encoder.layer_norm...
+    """
+    
+    # This will be populated dynamically based on actual keys
+    mapping = {}
+    return mapping
+
+
+def convert_torchaudio_to_hf(state_dict: dict) -> dict:
+    """
+    Convert torchaudio wav2vec2_model state dict to Hugging Face format.
+    
+    Args:
+        state_dict: State dict from torchaudio model
+        
+    Returns:
+        Converted state dict in HuggingFace format
+    """
+    new_state_dict = OrderedDict()
+    
+    for key, value in state_dict.items():
+        new_key = key
+        
+        # Feature extractor conv layers
+        # torchaudio: feature_extractor.conv_layers.0.0.weight -> hf: feature_extractor.conv_layers.0.conv.weight
+        if "feature_extractor.conv_layers" in key:
+            # Handle conv layer structure: .{layer_idx}.0. -> .{layer_idx}.conv.
+            # Handle norm layer structure: .{layer_idx}.2.1. -> .{layer_idx}.layer_norm.
+            parts = key.split(".")
+            layer_idx = parts[2]
+            
+            if ".0." in key and "weight" in key:
+                # Convolution weight
+                new_key = f"delulu.feature_extractor.conv_layers.{layer_idx}.conv.weight"
+            elif ".2.1." in key or (".1." in key and "layer_norm" not in key):
+                # Group norm / layer norm
+                if "weight" in key:
+                    new_key = f"delulu.feature_extractor.conv_layers.{layer_idx}.layer_norm.weight"
+                elif "bias" in key:
+                    new_key = f"delulu.feature_extractor.conv_layers.{layer_idx}.layer_norm.bias"
+            else:
+                new_key = f"delulu.{key}"
+        
+        # Feature projection
+        elif "encoder.feature_projection" in key:
+            new_key = key.replace("encoder.feature_projection", "delulu.feature_projection")
+        
+        # Positional conv embedding
+        elif "encoder.transformer.pos_conv_embed" in key:
+            new_key = key.replace("encoder.transformer.pos_conv_embed", "delulu.encoder.pos_conv_embed")
+        
+        # Transformer layers
+        elif "encoder.transformer.layers" in key:
+            new_key = key.replace("encoder.transformer.layers", "delulu.encoder.layers")
+            
+            # Attention mappings
+            new_key = new_key.replace(".attention.k_proj", ".attention.k_proj")
+            new_key = new_key.replace(".attention.v_proj", ".attention.v_proj")
+            new_key = new_key.replace(".attention.q_proj", ".attention.q_proj")
+            new_key = new_key.replace(".attention.out_proj", ".attention.out_proj")
+            
+            # Feed forward mappings
+            new_key = new_key.replace(".feed_forward.intermediate_dense", ".feed_forward.intermediate_dense")
+            new_key = new_key.replace(".feed_forward.output_dense", ".feed_forward.output_dense")
+            
+            # Layer norm mappings
+            new_key = new_key.replace(".layer_norms.0", ".layer_norm")
+            new_key = new_key.replace(".layer_norms.1", ".final_layer_norm")
+        
+        # Final layer norm
+        elif "encoder.transformer.layer_norm" in key:
+            new_key = key.replace("encoder.transformer.layer_norm", "delulu.encoder.layer_norm")
+        
+        # Mask embedding (if present)
+        elif "mask_emb" in key:
+            new_key = f"delulu.{key}"
+        
+        # Auxiliary head (if present)
+        elif "aux" in key:
+            new_key = key  # Keep as is for now
+        
+        else:
+            # Default: add delulu prefix
+            new_key = f"delulu.{key}"
+        
+        new_state_dict[new_key] = value
+        
+        if new_key != key:
+            logger.debug(f"Mapped: {key} -> {new_key}")
+    
+    return new_state_dict
+
+
+def convert_simple_format(state_dict: dict) -> dict:
+    """
+    Simple conversion that just renames keys minimally.
+    Suitable for direct loading with torchaudio models.
+    """
+    new_state_dict = OrderedDict()
+    
+    for key, value in state_dict.items():
+        # Just add a model prefix for organization
+        new_key = f"model.{key}" if not key.startswith("model.") else key
+        new_state_dict[new_key] = value
+    
+    return new_state_dict
+
+
+# =============================================================================
+# Checkpoint Loading
+# =============================================================================
+
+def load_lightning_checkpoint(checkpoint_path: str) -> Tuple[dict, dict]:
+    """
+    Load PyTorch Lightning checkpoint and extract model state dict.
+    
+    Args:
+        checkpoint_path: Path to .ckpt file
+        
+    Returns:
+        Tuple of (state_dict, hyperparameters)
+    """
+    logger.info(f"Loading checkpoint: {checkpoint_path}")
+    
+    checkpoint = torch.load(checkpoint_path, map_location="cpu")
+    
+    # Extract state dict
+    if "state_dict" in checkpoint:
+        state_dict = checkpoint["state_dict"]
+    else:
+        state_dict = checkpoint
+    
+    # Extract hyperparameters if available
+    hparams = checkpoint.get("hyper_parameters", {})
+    
+    # Clean up state dict keys (remove Lightning prefixes)
+    cleaned_state_dict = OrderedDict()
+    for key, value in state_dict.items():
+        new_key = key
+        
+        # Remove common Lightning prefixes
+        if key.startswith("model.wav2vec2."):
+            new_key = key.replace("model.wav2vec2.", "")
+        elif key.startswith("model."):
+            new_key = key.replace("model.", "")
+        
+        # Skip auxiliary heads unless needed
+        if "aux" in new_key:
+            logger.debug(f"Skipping auxiliary layer: {key}")
+            continue
+        
+        cleaned_state_dict[new_key] = value
+    
+    logger.info(f"Loaded {len(cleaned_state_dict)} parameters")
+    
+    return cleaned_state_dict, hparams
+
+
+def verify_state_dict(state_dict: dict) -> bool:
+    """
+    Verify the state dict has expected DELULU components.
+    """
+    expected_prefixes = [
+        "feature_extractor",
+        "encoder",
+    ]
+    
+    found_prefixes = set()
+    for key in state_dict.keys():
+        for prefix in expected_prefixes:
+            if prefix in key:
+                found_prefixes.add(prefix)
+    
+    missing = set(expected_prefixes) - found_prefixes
+    if missing:
+        logger.warning(f"Missing expected components: {missing}")
+        return False
+    
+    logger.info("✓ State dict contains expected components")
+    return True
+
+
+# =============================================================================
+# Main Conversion
+# =============================================================================
+
+def convert_checkpoint(
+    checkpoint_path: str,
+    output_dir: str,
+    save_safetensors_format: bool = True,
+    save_bin_format: bool = True,
+    verify: bool = True
+) -> None:
+    """
+    Convert DELULU checkpoint to Hugging Face format.
+    
+    Args:
+        checkpoint_path: Path to input .ckpt file
+        output_dir: Output directory for converted model
+        save_safetensors_format: Save in safetensors format
+        save_bin_format: Save in pytorch_model.bin format
+        verify: Verify the conversion
+    """
+    output_path = Path(output_dir)
+    output_path.mkdir(parents=True, exist_ok=True)
+    
+    # Step 1: Load checkpoint
+    state_dict, hparams = load_lightning_checkpoint(checkpoint_path)
+    
+    # Step 2: Verify state dict
+    if verify:
+        verify_state_dict(state_dict)
+    
+    # Step 3: Create and save config
+    logger.info("Creating DELULU config...")
+    config = DELULUConfig(
+        # Use DELULU's custom conv config
+        conv_dim=[512, 512, 512, 512, 512, 512, 512],
+        conv_kernel=[10, 3, 3, 3, 3, 2, 2],
+        conv_stride=[4, 2, 2, 2, 2, 2, 2],  # Key difference!
+    )
+    config.save_pretrained(output_dir)
+    
+    # Step 4: Convert state dict format (minimal conversion)
+    logger.info("Converting state dict format...")
+    # Keep the original format since it's compatible with torchaudio loading
+    converted_state_dict = state_dict
+    
+    # Step 5: Save weights
+    if save_safetensors_format and SAFETENSORS_AVAILABLE:
+        safetensors_path = output_path / "model.safetensors"
+        logger.info(f"Saving safetensors to: {safetensors_path}")
+        save_safetensors(converted_state_dict, str(safetensors_path))
+    
+    if save_bin_format:
+        bin_path = output_path / "pytorch_model.bin"
+        logger.info(f"Saving pytorch_model.bin to: {bin_path}")
+        torch.save(converted_state_dict, str(bin_path))
+    
+    # Step 6: Create additional files
+    create_additional_files(output_path, config)
+    
+    # Step 7: Print summary
+    print_conversion_summary(checkpoint_path, output_dir, converted_state_dict)
+
+
+def create_additional_files(output_path: Path, config: DELULUConfig) -> None:
+    """Create additional files needed for Hugging Face model."""
+    
+    # Create preprocessor_config.json
+    preprocessor_config = {
+        "do_normalize": True,
+        "feature_extractor_type": "Wav2Vec2FeatureExtractor",
+        "feature_size": 1,
+        "padding_side": "right",
+        "padding_value": 0.0,
+        "return_attention_mask": True,
+        "sampling_rate": config.sampling_rate,
+    }
+    
+    with open(output_path / "preprocessor_config.json", "w") as f:
+        json.dump(preprocessor_config, f, indent=2)
+    logger.info("Created preprocessor_config.json")
+    
+    # Create a simple modeling file for reference
+    modeling_code = '''"""
+DELULU Model - Minimal Loading Example
+
+This file shows how to load DELULU weights with torchaudio.
+For full Hugging Face Transformers integration, see the modeling_delulu.py file.
+"""
+
+import torch
+from torchaudio.models.wav2vec2 import wav2vec2_model
+
+# DELULU configuration
+DELULU_CONV_CONFIG = [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+
+def load_delulu(checkpoint_path: str = None, weights_path: str = None):
+    """
+    Load DELULU model.
+    
+    Args:
+        checkpoint_path: Path to original .ckpt file (PyTorch Lightning format)
+        weights_path: Path to pytorch_model.bin (Hugging Face format)
+    
+    Returns:
+        DELULU model ready for inference
+    """
+    model = wav2vec2_model(
+        extractor_mode="group_norm",
+        extractor_conv_layer_config=DELULU_CONV_CONFIG,
+        extractor_conv_bias=False,
+        encoder_embed_dim=768,
+        encoder_projection_dropout=0.1,
+        encoder_pos_conv_kernel=128,
+        encoder_pos_conv_groups=16,
+        encoder_num_layers=12,
+        encoder_num_heads=12,
+        encoder_attention_dropout=0.1,
+        encoder_ff_interm_features=3072,
+        encoder_ff_interm_dropout=0.1,
+        encoder_dropout=0.1,
+        encoder_layer_norm_first=False,
+        encoder_layer_drop=0.05,
+        aux_num_out=None,
+    )
+    
+    if checkpoint_path:
+        # Load from original Lightning checkpoint
+        checkpoint = torch.load(checkpoint_path, map_location="cpu")
+        state_dict = checkpoint.get("state_dict", checkpoint)
+        
+        # Clean keys
+        new_state_dict = {}
+        for k, v in state_dict.items():
+            if "model.wav2vec2" in k:
+                new_state_dict[k.replace("model.wav2vec2.", "")] = v
+            elif not k.startswith("aux"):
+                new_state_dict[k] = v
+        
+        model.load_state_dict(new_state_dict, strict=False)
+    
+    elif weights_path:
+        # Load from Hugging Face format
+        state_dict = torch.load(weights_path, map_location="cpu")
+        model.load_state_dict(state_dict, strict=False)
+    
+    return model
+
+
+def extract_features(model, waveform: torch.Tensor) -> torch.Tensor:
+    """
+    Extract speaker features from audio waveform.
+    
+    Args:
+        model: DELULU model
+        waveform: Audio tensor of shape (batch, samples) at 16kHz
+    
+    Returns:
+        Features of shape (batch, time, 768)
+    """
+    model.eval()
+    with torch.no_grad():
+        features, _ = model.extract_features(waveform)
+        # Return last layer features
+        return features[-1]
+
+
+if __name__ == "__main__":
+    # Example usage
+    import sys
+    
+    if len(sys.argv) > 1:
+        model = load_delulu(weights_path=sys.argv[1])
+        print(f"Model loaded successfully!")
+        print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
+    else:
+        print("Usage: python load_delulu.py path/to/pytorch_model.bin")
+'''
+    
+    with open(output_path / "load_delulu.py", "w") as f:
+        f.write(modeling_code)
+    logger.info("Created load_delulu.py")
+
+
+def print_conversion_summary(
+    input_path: str,
+    output_dir: str,
+    state_dict: dict
+) -> None:
+    """Print summary of the conversion."""
+    
+    total_params = sum(p.numel() for p in state_dict.values())
+    
+    print("\n" + "=" * 60)
+    print("DELULU Checkpoint Conversion Complete!")
+    print("=" * 60)
+    print(f"\nInput:  {input_path}")
+    print(f"Output: {output_dir}")
+    print(f"\nModel Statistics:")
+    print(f"  - Total parameters: {total_params:,}")
+    print(f"  - Parameter tensors: {len(state_dict)}")
+    print(f"\nOutput Files:")
+    
+    output_path = Path(output_dir)
+    for f in sorted(output_path.iterdir()):
+        size_mb = f.stat().st_size / 1024 / 1024
+        print(f"  - {f.name}: {size_mb:.2f} MB")
+    
+    print(f"\nNext Steps:")
+    print(f"  1. Test loading: python {output_dir}/load_delulu.py {output_dir}/pytorch_model.bin")
+    print(f"  2. Upload to HF: python upload_delulu_to_hf.py --checkpoint-dir {output_dir} --repo-id YOUR_USERNAME/DELULU")
+    print("=" * 60 + "\n")
+
+
+# =============================================================================
+# CLI Interface
+# =============================================================================
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Convert DELULU checkpoint to Hugging Face format",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Basic conversion
+  python convert_delulu_to_hf.py \\
+      --checkpoint /path/to/epoch=45-step=400000.ckpt \\
+      --output-dir ./delulu_hf_model
+
+  # Save only safetensors format
+  python convert_delulu_to_hf.py \\
+      --checkpoint /path/to/checkpoint.ckpt \\
+      --output-dir ./delulu_hf_model \\
+      --no-bin
+
+  # Skip verification
+  python convert_delulu_to_hf.py \\
+      --checkpoint /path/to/checkpoint.ckpt \\
+      --output-dir ./delulu_hf_model \\
+      --no-verify
+        """
+    )
+    
+    parser.add_argument(
+        "--checkpoint", "-c",
+        type=str,
+        required=True,
+        help="Path to DELULU checkpoint (.ckpt file)"
+    )
+    
+    parser.add_argument(
+        "--output-dir", "-o",
+        type=str,
+        required=True,
+        help="Output directory for converted model"
+    )
+    
+    parser.add_argument(
+        "--no-safetensors",
+        action="store_true",
+        help="Don't save in safetensors format"
+    )
+    
+    parser.add_argument(
+        "--no-bin",
+        action="store_true",
+        help="Don't save pytorch_model.bin"
+    )
+    
+    parser.add_argument(
+        "--no-verify",
+        action="store_true",
+        help="Skip state dict verification"
+    )
+    
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        help="Enable verbose logging"
+    )
+    
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    
+    if args.verbose:
+        logging.getLogger().setLevel(logging.DEBUG)
+    
+    convert_checkpoint(
+        checkpoint_path=args.checkpoint,
+        output_dir=args.output_dir,
+        save_safetensors_format=not args.no_safetensors,
+        save_bin_format=not args.no_bin,
+        verify=not args.no_verify
+    )
+
+
+if __name__ == "__main__":
+    main()

delulu_hf_model/config.json

@@ -0,0 +1,60 @@
+{
+  "model_type": "delulu",
+  "architectures": [
+    "DELULUModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_delulu.DELULUConfig",
+    "AutoModel": "modeling_delulu.DELULUModel"
+  },
+  "conv_dim": [
+    512,
+    512,
+    512,
+    512,
+    512,
+    512,
+    512
+  ],
+  "conv_kernel": [
+    10,
+    3,
+    3,
+    3,
+    3,
+    2,
+    2
+  ],
+  "conv_stride": [
+    4,
+    2,
+    2,
+    2,
+    2,
+    2,
+    2
+  ],
+  "conv_bias": false,
+  "extractor_mode": "group_norm",
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "hidden_dropout": 0.1,
+  "attention_dropout": 0.1,
+  "final_dropout": 0.1,
+  "feat_proj_dropout": 0.1,
+  "layer_norm_eps": 1e-05,
+  "layer_drop": 0.05,
+  "num_conv_pos_embeddings": 128,
+  "num_conv_pos_embedding_groups": 16,
+  "sampling_rate": 16000,
+  "do_stable_layer_norm": false,
+  "num_clusters": 256,
+  "feature_type": "redimnet",
+  "num_feat_extract_layers": 7,
+  "pad_token_id": 0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "torch_dtype": "float32"
+}

delulu_hf_model/configuration_delulu.py

@@ -0,0 +1,73 @@
+"""DELULU Configuration"""
+
+from transformers import PretrainedConfig
+
+
+class DELULUConfig(PretrainedConfig):
+    """Configuration class for DELULU model."""
+    
+    model_type = "delulu"
+    
+    def __init__(
+        self,
+        conv_dim=None,
+        conv_kernel=None,
+        conv_stride=None,
+        conv_bias=False,
+        extractor_mode="group_norm",
+        hidden_size=768,
+        num_hidden_layers=12,
+        num_attention_heads=12,
+        intermediate_size=3072,
+        hidden_dropout=0.1,
+        attention_dropout=0.1,
+        final_dropout=0.1,
+        feat_proj_dropout=0.1,
+        layer_norm_eps=1e-5,
+        layer_drop=0.05,
+        num_conv_pos_embeddings=128,
+        num_conv_pos_embedding_groups=16,
+        sampling_rate=16000,
+        do_stable_layer_norm=False,
+        num_clusters=256,
+        feature_type="redimnet",
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        **kwargs
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            **kwargs
+        )
+        
+        # DELULU conv config: [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+        self.conv_dim = conv_dim or [512, 512, 512, 512, 512, 512, 512]
+        self.conv_kernel = conv_kernel or [10, 3, 3, 3, 3, 2, 2]
+        self.conv_stride = conv_stride or [4, 2, 2, 2, 2, 2, 2]
+        self.conv_bias = conv_bias
+        self.extractor_mode = extractor_mode
+        
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        self.hidden_dropout = hidden_dropout
+        self.attention_dropout = attention_dropout
+        self.final_dropout = final_dropout
+        self.feat_proj_dropout = feat_proj_dropout
+        self.layer_norm_eps = layer_norm_eps
+        self.layer_drop = layer_drop
+        
+        self.num_conv_pos_embeddings = num_conv_pos_embeddings
+        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
+        
+        self.sampling_rate = sampling_rate
+        self.do_stable_layer_norm = do_stable_layer_norm
+        
+        self.num_clusters = num_clusters
+        self.feature_type = feature_type
+        
+        self.num_feat_extract_layers = len(self.conv_dim)

delulu_hf_model/modeling_delulu.py

@@ -0,0 +1,127 @@
+"""DELULU Model"""
+
+import torch
+import torch.nn as nn
+from typing import Optional, Tuple, Union
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import BaseModelOutput
+from .configuration_delulu import DELULUConfig
+
+try:
+    from torchaudio.models.wav2vec2 import wav2vec2_model
+    TORCHAUDIO_AVAILABLE = True
+except ImportError:
+    TORCHAUDIO_AVAILABLE = False
+
+
+class DELULUModel(PreTrainedModel):
+    """
+    DELULU Model for speaker-aware speech representation learning.
+    
+    Example:
+        ```python
+        from transformers import AutoModel
+        import torch
+        
+        model = AutoModel.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)
+        waveform = torch.randn(1, 16000)  # 1 second at 16kHz
+        outputs = model(waveform)
+        features = outputs.last_hidden_state
+        ```
+    """
+    
+    config_class = DELULUConfig
+    base_model_prefix = "delulu"
+    main_input_name = "input_values"
+    
+    def __init__(self, config: DELULUConfig):
+        super().__init__(config)
+        self.config = config
+        
+        if not TORCHAUDIO_AVAILABLE:
+            raise ImportError("torchaudio is required. Install with: pip install torchaudio")
+        
+        # Build conv config
+        conv_layer_config = list(zip(
+            config.conv_dim,
+            config.conv_kernel,
+            config.conv_stride
+        ))
+        
+        # Create torchaudio model
+        self.wav2vec2 = wav2vec2_model(
+            extractor_mode=config.extractor_mode,
+            extractor_conv_layer_config=conv_layer_config,
+            extractor_conv_bias=config.conv_bias,
+            encoder_embed_dim=config.hidden_size,
+            encoder_projection_dropout=config.feat_proj_dropout,
+            encoder_pos_conv_kernel=config.num_conv_pos_embeddings,
+            encoder_pos_conv_groups=config.num_conv_pos_embedding_groups,
+            encoder_num_layers=config.num_hidden_layers,
+            encoder_num_heads=config.num_attention_heads,
+            encoder_attention_dropout=config.attention_dropout,
+            encoder_ff_interm_features=config.intermediate_size,
+            encoder_ff_interm_dropout=config.hidden_dropout,
+            encoder_dropout=config.hidden_dropout,
+            encoder_layer_norm_first=config.do_stable_layer_norm,
+            encoder_layer_drop=config.layer_drop,
+            aux_num_out=None,
+        )
+        
+        self.post_init()
+    
+    def _init_weights(self, module):
+        """Initialize weights."""
+        pass  # Handled by torchaudio
+    
+    def forward(
+        self,
+        input_values: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, BaseModelOutput]:
+        """
+        Args:
+            input_values: Audio waveform (batch, samples) at 16kHz
+            attention_mask: Optional attention mask
+            output_hidden_states: Whether to return all hidden states
+            return_dict: Whether to return BaseModelOutput
+        """
+        return_dict = return_dict if return_dict is not None else True
+        output_hidden_states = output_hidden_states if output_hidden_states is not None else False
+        
+        if input_values.dim() == 1:
+            input_values = input_values.unsqueeze(0)
+        
+        lengths = None
+        if attention_mask is not None:
+            lengths = attention_mask.sum(dim=-1)
+        
+        if output_hidden_states:
+            features, _ = self.wav2vec2.extract_features(input_values, lengths=lengths)
+            hidden_states = tuple(features)
+            last_hidden_state = features[-1]
+        else:
+            last_hidden_state, _ = self.wav2vec2(input_values, lengths=lengths)
+            hidden_states = None
+        
+        if not return_dict:
+            return (last_hidden_state, hidden_states) if hidden_states else (last_hidden_state,)
+        
+        return BaseModelOutput(
+            last_hidden_state=last_hidden_state,
+            hidden_states=hidden_states,
+        )
+    
+    def extract_features(self, input_values: torch.Tensor):
+        """Extract features from all layers."""
+        if input_values.dim() == 1:
+            input_values = input_values.unsqueeze(0)
+        features, _ = self.wav2vec2.extract_features(input_values)
+        return tuple(features)
+    
+    @classmethod
+    def _load_pretrained_model_low_mem(cls, *args, **kwargs):
+        """Override to handle custom loading."""
+        return super()._load_pretrained_model_low_mem(*args, **kwargs)

load_delulu.py

@@ -1,94 +0,0 @@
-"""
-DELULU Model - Minimal Loading Example
-
-This file shows how to load DELULU weights with torchaudio.
-For full Hugging Face Transformers integration, see the modeling_delulu.py file.
-"""
-
-import torch
-from torchaudio.models.wav2vec2 import wav2vec2_model
-
-# DELULU configuration
-DELULU_CONV_CONFIG = [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
-
-def load_delulu(checkpoint_path: str = None, weights_path: str = None):
-    """
-    Load DELULU model.
-    
-    Args:
-        checkpoint_path: Path to original .ckpt file (PyTorch Lightning format)
-        weights_path: Path to pytorch_model.bin (Hugging Face format)
-    
-    Returns:
-        DELULU model ready for inference
-    """
-    model = wav2vec2_model(
-        extractor_mode="group_norm",
-        extractor_conv_layer_config=DELULU_CONV_CONFIG,
-        extractor_conv_bias=False,
-        encoder_embed_dim=768,
-        encoder_projection_dropout=0.1,
-        encoder_pos_conv_kernel=128,
-        encoder_pos_conv_groups=16,
-        encoder_num_layers=12,
-        encoder_num_heads=12,
-        encoder_attention_dropout=0.1,
-        encoder_ff_interm_features=3072,
-        encoder_ff_interm_dropout=0.1,
-        encoder_dropout=0.1,
-        encoder_layer_norm_first=False,
-        encoder_layer_drop=0.05,
-        aux_num_out=None,
-    )
-    
-    if checkpoint_path:
-        # Load from original Lightning checkpoint
-        checkpoint = torch.load(checkpoint_path, map_location="cpu")
-        state_dict = checkpoint.get("state_dict", checkpoint)
-        
-        # Clean keys
-        new_state_dict = {}
-        for k, v in state_dict.items():
-            if "model.wav2vec2" in k:
-                new_state_dict[k.replace("model.wav2vec2.", "")] = v
-            elif not k.startswith("aux"):
-                new_state_dict[k] = v
-        
-        model.load_state_dict(new_state_dict, strict=False)
-    
-    elif weights_path:
-        # Load from Hugging Face format
-        state_dict = torch.load(weights_path, map_location="cpu")
-        model.load_state_dict(state_dict, strict=False)
-    
-    return model
-
-
-def extract_features(model, waveform: torch.Tensor) -> torch.Tensor:
-    """
-    Extract speaker features from audio waveform.
-    
-    Args:
-        model: DELULU model
-        waveform: Audio tensor of shape (batch, samples) at 16kHz
-    
-    Returns:
-        Features of shape (batch, time, 768)
-    """
-    model.eval()
-    with torch.no_grad():
-        features, _ = model.extract_features(waveform)
-        # Return last layer features
-        return features[-1]
-
-
-if __name__ == "__main__":
-    # Example usage
-    import sys
-    
-    if len(sys.argv) > 1:
-        model = load_delulu(weights_path=sys.argv[1])
-        print(f"Model loaded successfully!")
-        print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
-    else:
-        print("Usage: python load_delulu.py path/to/pytorch_model.bin")

modeling_delulu.py

@@ -1,12 +1,27 @@
-"""DELULU Model"""
+"""
+DELULU Model
+
+DELULU (Discriminative Embedding Learning Using Latent Units) is a speaker-aware
+self-supervised speech foundational model based on HuBERT architecture.
+
+Paper: https://arxiv.org/abs/2510.17662
+Authors: Massa Baali, Rita Singh, Bhiksha Raj
+
+This implementation wraps torchaudio's wav2vec2_model for compatibility with
+Hugging Face's AutoModel interface.
+"""
 
 import torch
 import torch.nn as nn
 from typing import Optional, Tuple, Union
+from dataclasses import dataclass
+
 from transformers import PreTrainedModel
 from transformers.modeling_outputs import BaseModelOutput
+
 from .configuration_delulu import DELULUConfig
 
+# Try to import torchaudio
 try:
     from torchaudio.models.wav2vec2 import wav2vec2_model
     TORCHAUDIO_AVAILABLE = True
@@ -14,41 +29,79 @@ except ImportError:
     TORCHAUDIO_AVAILABLE = False
 
 
+@dataclass
+class DELULUOutput(BaseModelOutput):
+    """
+    Output class for DELULU model.
+    
+    Args:
+        last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
+            Sequence of hidden-states at the output of the last layer of the model.
+        hidden_states (`tuple(torch.FloatTensor)`, *optional*):
+            Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for each layer)
+            of shape `(batch_size, sequence_length, hidden_size)`.
+        attentions (`tuple(torch.FloatTensor)`, *optional*):
+            Attention weights (not available for torchaudio backend).
+        extract_features (`torch.FloatTensor` of shape `(batch_size, sequence_length, conv_dim[-1])`):
+            Features from the convolutional feature extractor.
+    """
+    last_hidden_state: torch.FloatTensor = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
+    extract_features: Optional[torch.FloatTensor] = None
+
+
 class DELULUModel(PreTrainedModel):
     """
     DELULU Model for speaker-aware speech representation learning.
     
+    This model wraps torchaudio's wav2vec2_model with DELULU's custom configuration
+    (modified convolutional strides for 16ms frame shift).
+    
     Example:
         ```python
         from transformers import AutoModel
         import torch
         
+        # Load model
         model = AutoModel.from_pretrained("cmu-mlsp/DELULU", trust_remote_code=True)
-        waveform = torch.randn(1, 16000)  # 1 second at 16kHz
-        outputs = model(waveform)
-        features = outputs.last_hidden_state
+        model.eval()
+        
+        # Process audio (16kHz, mono)
+        waveform = torch.randn(1, 16000)  # 1 second of audio
+        
+        with torch.no_grad():
+            outputs = model(waveform)
+            features = outputs.last_hidden_state  # [1, T, 768]
+            
+            # For speaker verification, use mean pooling
+            speaker_embedding = features.mean(dim=1)  # [1, 768]
         ```
     """
     
     config_class = DELULUConfig
     base_model_prefix = "delulu"
     main_input_name = "input_values"
+    supports_gradient_checkpointing = False
     
     def __init__(self, config: DELULUConfig):
         super().__init__(config)
         self.config = config
         
         if not TORCHAUDIO_AVAILABLE:
-            raise ImportError("torchaudio is required. Install with: pip install torchaudio")
+            raise ImportError(
+                "torchaudio is required for DELULU model. "
+                "Install with: pip install torchaudio"
+            )
         
-        # Build conv config
+        # Build convolutional layer config from DELULU config
         conv_layer_config = list(zip(
             config.conv_dim,
             config.conv_kernel,
             config.conv_stride
         ))
         
-        # Create torchaudio model
+        # Create the underlying torchaudio model
         self.wav2vec2 = wav2vec2_model(
             extractor_mode=config.extractor_mode,
             extractor_conv_layer_config=conv_layer_config,
@@ -68,60 +121,213 @@ class DELULUModel(PreTrainedModel):
             aux_num_out=None,
         )
         
+        # Initialize weights
         self.post_init()
     
-    def _init_weights(self, module):
-        """Initialize weights."""
-        pass  # Handled by torchaudio
-    
     def forward(
         self,
         input_values: torch.Tensor,
         attention_mask: Optional[torch.Tensor] = None,
         output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
         return_dict: Optional[bool] = None,
-    ) -> Union[Tuple, BaseModelOutput]:
+    ) -> Union[Tuple, DELULUOutput]:
         """
+        Forward pass of DELULU model.
+        
         Args:
-            input_values: Audio waveform (batch, samples) at 16kHz
-            attention_mask: Optional attention mask
-            output_hidden_states: Whether to return all hidden states
-            return_dict: Whether to return BaseModelOutput
+            input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`):
+                Raw audio waveform at 16kHz sampling rate.
+            attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Mask to avoid performing attention on padding. Not used in current implementation.
+            output_hidden_states (`bool`, *optional*):
+                Whether to return all hidden states.
+            output_attentions (`bool`, *optional*):
+                Whether to return attention weights. Not supported with torchaudio backend.
+            return_dict (`bool`, *optional*):
+                Whether to return a `DELULUOutput` instead of a tuple.
+        
+        Returns:
+            `DELULUOutput` or `tuple`: Model outputs.
         """
-        return_dict = return_dict if return_dict is not None else True
-        output_hidden_states = output_hidden_states if output_hidden_states is not None else False
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None 
+            else self.config.output_hidden_states if hasattr(self.config, 'output_hidden_states')
+            else False
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict if hasattr(self.config, 'use_return_dict') else True
         
+        # Ensure input is 2D: (batch, samples)
         if input_values.dim() == 1:
             input_values = input_values.unsqueeze(0)
         
+        # Handle lengths for torchaudio model
         lengths = None
         if attention_mask is not None:
             lengths = attention_mask.sum(dim=-1)
         
+        # Extract features using torchaudio model
         if output_hidden_states:
-            features, _ = self.wav2vec2.extract_features(input_values, lengths=lengths)
+            # Get all layer outputs
+            features, lengths_out = self.wav2vec2.extract_features(
+                input_values, 
+                lengths=lengths
+            )
+            # features is a list of tensors, one per layer
             hidden_states = tuple(features)
             last_hidden_state = features[-1]
         else:
-            last_hidden_state, _ = self.wav2vec2(input_values, lengths=lengths)
+            # Just get final output
+            outputs, lengths_out = self.wav2vec2(input_values, lengths=lengths)
+            last_hidden_state = outputs
             hidden_states = None
         
+        # Get convolutional features (before transformer)
+        extract_features = self.wav2vec2.feature_extractor(input_values, lengths)[0]
+        
         if not return_dict:
-            return (last_hidden_state, hidden_states) if hidden_states else (last_hidden_state,)
+            outputs = (last_hidden_state,)
+            if output_hidden_states:
+                outputs = outputs + (hidden_states,)
+            return outputs
         
-        return BaseModelOutput(
+        return DELULUOutput(
             last_hidden_state=last_hidden_state,
             hidden_states=hidden_states,
+            attentions=None,  # torchaudio doesn't expose attention weights
+            extract_features=extract_features,
         )
     
-    def extract_features(self, input_values: torch.Tensor):
-        """Extract features from all layers."""
+    def extract_features(
+        self, 
+        input_values: torch.Tensor,
+        lengths: Optional[torch.Tensor] = None
+    ) -> Tuple[torch.Tensor, ...]:
+        """
+        Extract features from all layers.
+        
+        Args:
+            input_values: Audio waveform of shape (batch, samples)
+            lengths: Optional lengths for each sample in batch
+            
+        Returns:
+            Tuple of tensors, one per layer (including CNN output)
+        """
         if input_values.dim() == 1:
             input_values = input_values.unsqueeze(0)
-        features, _ = self.wav2vec2.extract_features(input_values)
+        
+        features, _ = self.wav2vec2.extract_features(input_values, lengths=lengths)
         return tuple(features)
     
-    @classmethod
-    def _load_pretrained_model_low_mem(cls, *args, **kwargs):
-        """Override to handle custom loading."""
-        return super()._load_pretrained_model_low_mem(*args, **kwargs)
+    def get_speaker_embedding(
+        self,
+        input_values: torch.Tensor,
+        pooling: str = "mean"
+    ) -> torch.Tensor:
+        """
+        Extract speaker embedding from audio.
+        
+        Args:
+            input_values: Audio waveform of shape (batch, samples)
+            pooling: Pooling method - "mean", "max", or "first"
+            
+        Returns:
+            Speaker embedding of shape (batch, hidden_size)
+        """
+        outputs = self.forward(input_values, return_dict=True)
+        features = outputs.last_hidden_state
+        
+        if pooling == "mean":
+            return features.mean(dim=1)
+        elif pooling == "max":
+            return features.max(dim=1).values
+        elif pooling == "first":
+            return features[:, 0, :]
+        else:
+            raise ValueError(f"Unknown pooling method: {pooling}")
+    
+    def _init_weights(self, module):
+        """Initialize weights - mostly handled by torchaudio."""
+        pass
+
+
+class DELULUForSequenceClassification(PreTrainedModel):
+    """
+    DELULU with a classification head for speaker verification and other tasks.
+    
+    Example:
+        ```python
+        from transformers import AutoModel
+        
+        model = AutoModel.from_pretrained(
+            "cmu-mlsp/DELULU", 
+            trust_remote_code=True,
+            num_labels=1251  # Number of speakers in VoxCeleb2
+        )
+        ```
+    """
+    
+    config_class = DELULUConfig
+    base_model_prefix = "delulu"
+    
+    def __init__(self, config: DELULUConfig):
+        super().__init__(config)
+        
+        self.delulu = DELULUModel(config)
+        self.projector = nn.Linear(config.hidden_size, config.hidden_size)
+        
+        num_labels = getattr(config, 'num_labels', None)
+        if num_labels:
+            self.classifier = nn.Linear(config.hidden_size, num_labels)
+        else:
+            self.classifier = None
+        
+        self.post_init()
+    
+    def forward(
+        self,
+        input_values: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        return_dict: Optional[bool] = None,
+    ):
+        return_dict = return_dict if return_dict is not None else True
+        
+        outputs = self.delulu(
+            input_values,
+            attention_mask=attention_mask,
+            return_dict=True
+        )
+        
+        # Pool features
+        hidden_states = outputs.last_hidden_state
+        pooled = hidden_states.mean(dim=1)
+        
+        # Project
+        embeddings = self.projector(pooled)
+        
+        # Classify if head exists
+        logits = None
+        if self.classifier is not None:
+            logits = self.classifier(embeddings)
+        
+        loss = None
+        if labels is not None and logits is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(logits, labels)
+        
+        if not return_dict:
+            output = (logits, embeddings) + (outputs.last_hidden_state,)
+            return ((loss,) + output) if loss is not None else output
+        
+        return {
+            "loss": loss,
+            "logits": logits,
+            "embeddings": embeddings,
+            "last_hidden_state": outputs.last_hidden_state,
+        }
+
+
+# Register for auto classes
+DELULUConfig.register_for_auto_class()
+DELULUModel.register_for_auto_class("AutoModel")

preprocessor_config.json

@@ -1,9 +0,0 @@
-{
-  "do_normalize": true,
-  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
-  "feature_size": 1,
-  "padding_side": "right",
-  "padding_value": 0.0,
-  "return_attention_mask": true,
-  "sampling_rate": 16000
-}

upload_delulu_to_hf.py

@@ -0,0 +1,593 @@
+#!/usr/bin/env python3
+"""
+DELULU Model Upload Script for Hugging Face Hub
+================================================
+
+Production-ready script to upload DELULU (Discriminative Embedding Learning Using
+Latent Units) model checkpoints to Hugging Face with safety checks, versioning,
+and best practices.
+
+Author: Massa Baali
+Model: DELULU - Speaker-Aware Self-Supervised Speech Foundational Model
+Paper: https://arxiv.org/abs/2510.17662
+
+Usage:
+    python upload_delulu_to_hf.py --checkpoint-dir ./checkpoints --repo-id username/DELULU
+    
+    # With all options:
+    python upload_delulu_to_hf.py \
+        --checkpoint-dir ./checkpoints \
+        --repo-id username/DELULU \
+        --version v1.0.0 \
+        --tags speaker-verification speech-ssl hubert \
+        --private \
+        --dry-run
+"""
+
+import argparse
+import hashlib
+import json
+import logging
+import os
+import sys
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+try:
+    from huggingface_hub import (
+        HfApi,
+        create_repo,
+        upload_folder,
+        login,
+        whoami,
+        RepoUrl,
+    )
+    from huggingface_hub.utils import RepositoryNotFoundError, HfHubHTTPError
+except ImportError:
+    print("Error: huggingface_hub not installed. Install with: pip install huggingface_hub")
+    sys.exit(1)
+
+# Configure logging
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s - %(levelname)s - %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+logger = logging.getLogger(__name__)
+
+
+# =============================================================================
+# Configuration
+# =============================================================================
+
+@dataclass
+class DELULUConfig:
+    """Configuration for DELULU model architecture.
+    
+    DELULU uses HuBERT architecture with modified convolutional feature extractor
+    strides for 16ms frame shift (optimized for speaker verification).
+    """
+    # Model architecture (HuBERT-based)
+    model_type: str = "hubert"
+    
+    # Modified convolutional feature extractor configuration
+    # Standard HuBERT: [(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+    # DELULU:          [(512, 10, 4)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2
+    conv_dim: list = field(default_factory=lambda: [512, 512, 512, 512, 512, 512, 512])
+    conv_kernel: list = field(default_factory=lambda: [10, 3, 3, 3, 3, 2, 2])
+    conv_stride: list = field(default_factory=lambda: [4, 2, 2, 2, 2, 2, 2])  # Key difference!
+    
+    # Transformer configuration
+    hidden_size: int = 768
+    num_hidden_layers: int = 12
+    num_attention_heads: int = 12
+    intermediate_size: int = 3072
+    
+    # Training configuration
+    frame_shift_ms: int = 16  # Optimal for speaker verification
+    sampling_rate: int = 16000
+    
+    # Clustering configuration (ReDimNet-guided)
+    num_clusters: int = 256
+    cluster_feature_dim: int = 2304  # ReDimNet frame-level embedding dimension
+    
+    def to_dict(self) -> dict:
+        """Convert config to dictionary for serialization."""
+        return {
+            "model_type": self.model_type,
+            "conv_dim": self.conv_dim,
+            "conv_kernel": self.conv_kernel,
+            "conv_stride": self.conv_stride,
+            "hidden_size": self.hidden_size,
+            "num_hidden_layers": self.num_hidden_layers,
+            "num_attention_heads": self.num_attention_heads,
+            "intermediate_size": self.intermediate_size,
+            "frame_shift_ms": self.frame_shift_ms,
+            "sampling_rate": self.sampling_rate,
+            "num_clusters": self.num_clusters,
+            "cluster_feature_dim": self.cluster_feature_dim,
+            "architectures": ["DELULUModel"],
+            "auto_map": {
+                "AutoModel": "modeling_delulu.DELULUModel",
+                "AutoConfig": "configuration_delulu.DELULUConfig"
+            }
+        }
+
+
+@dataclass
+class UploadConfig:
+    """Configuration for the upload process."""
+    checkpoint_dir: Path
+    repo_id: str
+    version: Optional[str] = None
+    tags: list = field(default_factory=list)
+    private: bool = False
+    dry_run: bool = False
+    create_if_missing: bool = True
+    commit_message: Optional[str] = None
+    
+    # Safety settings
+    verify_checksums: bool = True
+    max_file_size_gb: float = 10.0
+    required_files: list = field(default_factory=lambda: ["pytorch_model.bin", "config.json"])
+    
+    def __post_init__(self):
+        self.checkpoint_dir = Path(self.checkpoint_dir)
+
+
+# =============================================================================
+# Safety Checks
+# =============================================================================
+
+class SafetyChecker:
+    """Performs safety checks before upload."""
+    
+    def __init__(self, config: UploadConfig):
+        self.config = config
+        self.errors: list[str] = []
+        self.warnings: list[str] = []
+    
+    def check_all(self) -> bool:
+        """Run all safety checks. Returns True if all pass."""
+        self._check_directory_exists()
+        self._check_required_files()
+        self._check_file_sizes()
+        self._check_no_sensitive_data()
+        self._check_checkpoint_integrity()
+        
+        # Log results
+        for warning in self.warnings:
+            logger.warning(f"⚠️  {warning}")
+        for error in self.errors:
+            logger.error(f"❌ {error}")
+        
+        if self.errors:
+            logger.error(f"Safety checks failed with {len(self.errors)} error(s)")
+            return False
+        
+        logger.info("✅ All safety checks passed")
+        return True
+    
+    def _check_directory_exists(self):
+        """Verify checkpoint directory exists and is accessible."""
+        if not self.config.checkpoint_dir.exists():
+            self.errors.append(f"Checkpoint directory not found: {self.config.checkpoint_dir}")
+        elif not self.config.checkpoint_dir.is_dir():
+            self.errors.append(f"Path is not a directory: {self.config.checkpoint_dir}")
+    
+    def _check_required_files(self):
+        """Check that required model files exist."""
+        if not self.config.checkpoint_dir.exists():
+            return
+        
+        for required_file in self.config.required_files:
+            file_path = self.config.checkpoint_dir / required_file
+            # Also check for .safetensors variant
+            safetensors_variant = required_file.replace(".bin", ".safetensors")
+            safetensors_path = self.config.checkpoint_dir / safetensors_variant
+            
+            if not file_path.exists() and not safetensors_path.exists():
+                # Special handling for model weights - either .bin or .safetensors is fine
+                if "model" in required_file:
+                    self.warnings.append(
+                        f"Model file not found: {required_file} or {safetensors_variant}. "
+                        "Will look for alternative formats."
+                    )
+                else:
+                    self.errors.append(f"Required file not found: {required_file}")
+    
+    def _check_file_sizes(self):
+        """Verify no files exceed maximum size limit."""
+        if not self.config.checkpoint_dir.exists():
+            return
+        
+        max_size_bytes = self.config.max_file_size_gb * 1024 * 1024 * 1024
+        
+        for file_path in self.config.checkpoint_dir.rglob("*"):
+            if file_path.is_file():
+                size = file_path.stat().st_size
+                if size > max_size_bytes:
+                    self.errors.append(
+                        f"File exceeds {self.config.max_file_size_gb}GB limit: "
+                        f"{file_path.name} ({size / 1024 / 1024 / 1024:.2f}GB)"
+                    )
+    
+    def _check_no_sensitive_data(self):
+        """Check for potentially sensitive files that shouldn't be uploaded."""
+        sensitive_patterns = [
+            ".env", ".secret", "credentials", "password", "api_key", "token",
+            ".git", "__pycache__", ".pyc", ".DS_Store"
+        ]
+        
+        if not self.config.checkpoint_dir.exists():
+            return
+        
+        for file_path in self.config.checkpoint_dir.rglob("*"):
+            file_name = file_path.name.lower()
+            for pattern in sensitive_patterns:
+                if pattern in file_name:
+                    self.warnings.append(
+                        f"Potentially sensitive file detected: {file_path.name}. "
+                        "Consider adding to .gitignore or removing before upload."
+                    )
+                    break
+    
+    def _check_checkpoint_integrity(self):
+        """Basic integrity check for PyTorch checkpoint files."""
+        if not self.config.checkpoint_dir.exists():
+            return
+        
+        try:
+            import torch
+            
+            for file_path in self.config.checkpoint_dir.glob("*.bin"):
+                try:
+                    # Just try to load metadata, not full weights
+                    torch.load(file_path, map_location="cpu", weights_only=False)
+                    logger.info(f"✓ Checkpoint integrity verified: {file_path.name}")
+                except Exception as e:
+                    self.errors.append(f"Corrupted checkpoint file: {file_path.name} - {e}")
+        except ImportError:
+            self.warnings.append("PyTorch not installed, skipping checkpoint integrity check")
+
+
+# =============================================================================
+# Checksum Utilities
+# =============================================================================
+
+def compute_file_checksum(file_path: Path, algorithm: str = "sha256") -> str:
+    """Compute checksum for a file."""
+    hash_func = hashlib.new(algorithm)
+    
+    with open(file_path, "rb") as f:
+        for chunk in iter(lambda: f.read(8192), b""):
+            hash_func.update(chunk)
+    
+    return hash_func.hexdigest()
+
+
+def generate_checksums(directory: Path) -> dict:
+    """Generate checksums for all files in directory."""
+    checksums = {}
+    
+    for file_path in directory.rglob("*"):
+        if file_path.is_file():
+            relative_path = file_path.relative_to(directory)
+            checksums[str(relative_path)] = {
+                "sha256": compute_file_checksum(file_path, "sha256"),
+                "size_bytes": file_path.stat().st_size
+            }
+    
+    return checksums
+
+
+def save_checksums(checksums: dict, output_path: Path):
+    """Save checksums to JSON file."""
+    with open(output_path, "w") as f:
+        json.dump(checksums, f, indent=2)
+    logger.info(f"Checksums saved to: {output_path}")
+
+
+# =============================================================================
+# Upload Manager
+# =============================================================================
+
+class DELULUUploader:
+    """Handles uploading DELULU model to Hugging Face Hub."""
+    
+    def __init__(self, upload_config: UploadConfig):
+        self.config = upload_config
+        self.api = HfApi()
+        self.model_config = DELULUConfig()
+    
+    def authenticate(self) -> bool:
+        """Verify authentication with Hugging Face Hub."""
+        try:
+            user_info = whoami()
+            logger.info(f"✅ Authenticated as: {user_info['name']}")
+            return True
+        except Exception as e:
+            logger.error(f"❌ Authentication failed: {e}")
+            logger.info("Run 'huggingface-cli login' or set HF_TOKEN environment variable")
+            return False
+    
+    def prepare_upload_directory(self) -> Path:
+        """Prepare files for upload, including config and checksums."""
+        upload_dir = self.config.checkpoint_dir
+        
+        # Generate and save config.json if not present
+        config_path = upload_dir / "config.json"
+        if not config_path.exists():
+            logger.info("Generating config.json...")
+            with open(config_path, "w") as f:
+                json.dump(self.model_config.to_dict(), f, indent=2)
+        
+        # Generate checksums
+        if self.config.verify_checksums:
+            logger.info("Generating checksums...")
+            checksums = generate_checksums(upload_dir)
+            save_checksums(checksums, upload_dir / "checksums.json")
+        
+        # Create upload metadata
+        metadata = {
+            "upload_timestamp": datetime.utcnow().isoformat(),
+            "version": self.config.version,
+            "uploader_script_version": "1.0.0",
+            "model_type": "DELULU",
+            "base_architecture": "HuBERT"
+        }
+        
+        metadata_path = upload_dir / "upload_metadata.json"
+        with open(metadata_path, "w") as f:
+            json.dump(metadata, f, indent=2)
+        
+        return upload_dir
+    
+    def create_or_verify_repo(self) -> bool:
+        """Create repository if it doesn't exist, or verify access."""
+        try:
+            # Check if repo exists
+            self.api.repo_info(repo_id=self.config.repo_id, repo_type="model")
+            logger.info(f"✅ Repository exists: {self.config.repo_id}")
+            return True
+            
+        except RepositoryNotFoundError:
+            if self.config.create_if_missing:
+                logger.info(f"Creating repository: {self.config.repo_id}")
+                
+                if self.config.dry_run:
+                    logger.info("[DRY RUN] Would create repository")
+                    return True
+                
+                try:
+                    repo_url: RepoUrl = create_repo(
+                        repo_id=self.config.repo_id,
+                        repo_type="model",
+                        private=self.config.private,
+                        exist_ok=True
+                    )
+                    logger.info(f"✅ Repository created: {repo_url}")
+                    return True
+                except HfHubHTTPError as e:
+                    logger.error(f"❌ Failed to create repository: {e}")
+                    return False
+            else:
+                logger.error(f"❌ Repository not found: {self.config.repo_id}")
+                return False
+                
+        except Exception as e:
+            logger.error(f"❌ Error accessing repository: {e}")
+            return False
+    
+    def upload(self) -> bool:
+        """Execute the upload process."""
+        logger.info("=" * 60)
+        logger.info("DELULU Model Upload to Hugging Face Hub")
+        logger.info("=" * 60)
+        
+        # Step 1: Authenticate
+        if not self.authenticate():
+            return False
+        
+        # Step 2: Safety checks
+        safety_checker = SafetyChecker(self.config)
+        if not safety_checker.check_all():
+            return False
+        
+        # Step 3: Create/verify repository
+        if not self.create_or_verify_repo():
+            return False
+        
+        # Step 4: Prepare upload directory
+        upload_dir = self.prepare_upload_directory()
+        
+        # Step 5: Generate commit message
+        commit_message = self.config.commit_message or self._generate_commit_message()
+        
+        # Step 6: Execute upload
+        if self.config.dry_run:
+            logger.info("[DRY RUN] Would upload the following files:")
+            for file_path in upload_dir.rglob("*"):
+                if file_path.is_file():
+                    size_mb = file_path.stat().st_size / 1024 / 1024
+                    logger.info(f"  - {file_path.relative_to(upload_dir)} ({size_mb:.2f} MB)")
+            logger.info(f"[DRY RUN] Commit message: {commit_message}")
+            return True
+        
+        logger.info("Starting upload...")
+        try:
+            upload_folder(
+                folder_path=str(upload_dir),
+                repo_id=self.config.repo_id,
+                repo_type="model",
+                commit_message=commit_message,
+                ignore_patterns=[
+                    "*.pyc", "__pycache__", ".git", ".DS_Store",
+                    "*.log", "wandb", "runs"
+                ]
+            )
+            
+            logger.info("✅ Upload complete!")
+            logger.info(f"View model at: https://huggingface.co/{self.config.repo_id}")
+            
+            # Create version tag if specified
+            if self.config.version:
+                self._create_version_tag()
+            
+            return True
+            
+        except Exception as e:
+            logger.error(f"❌ Upload failed: {e}")
+            return False
+    
+    def _generate_commit_message(self) -> str:
+        """Generate a descriptive commit message."""
+        parts = ["Upload DELULU model checkpoint"]
+        
+        if self.config.version:
+            parts.append(f"(version {self.config.version})")
+        
+        parts.append(f"\n\nModel: DELULU - Speaker-Aware Self-Supervised Speech Model")
+        parts.append(f"Architecture: HuBERT with modified stride configuration")
+        parts.append(f"Frame shift: 16ms (optimized for speaker verification)")
+        
+        if self.config.tags:
+            parts.append(f"\nTags: {', '.join(self.config.tags)}")
+        
+        return " ".join(parts[:2]) + "".join(parts[2:])
+    
+    def _create_version_tag(self):
+        """Create a Git tag for the version."""
+        try:
+            self.api.create_tag(
+                repo_id=self.config.repo_id,
+                tag=self.config.version,
+                tag_message=f"DELULU {self.config.version}",
+                repo_type="model"
+            )
+            logger.info(f"✅ Created version tag: {self.config.version}")
+        except Exception as e:
+            logger.warning(f"⚠️  Could not create version tag: {e}")
+
+
+# =============================================================================
+# CLI Interface
+# =============================================================================
+
+def parse_args() -> argparse.Namespace:
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(
+        description="Upload DELULU model checkpoints to Hugging Face Hub",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Basic upload
+  python upload_delulu_to_hf.py --checkpoint-dir ./checkpoints --repo-id username/DELULU
+
+  # Upload with version and tags
+  python upload_delulu_to_hf.py \\
+      --checkpoint-dir ./checkpoints \\
+      --repo-id username/DELULU \\
+      --version v1.0.0 \\
+      --tags speaker-verification speech-ssl hubert
+
+  # Dry run (no actual upload)
+  python upload_delulu_to_hf.py --checkpoint-dir ./checkpoints --repo-id username/DELULU --dry-run
+
+  # Private repository
+  python upload_delulu_to_hf.py --checkpoint-dir ./checkpoints --repo-id username/DELULU --private
+        """
+    )
+    
+    # Required arguments
+    parser.add_argument(
+        "--checkpoint-dir", "-c",
+        type=str,
+        required=True,
+        help="Path to directory containing model checkpoints"
+    )
+    parser.add_argument(
+        "--repo-id", "-r",
+        type=str,
+        required=True,
+        help="Hugging Face repository ID (e.g., username/DELULU)"
+    )
+    
+    # Optional arguments
+    parser.add_argument(
+        "--version", "-v",
+        type=str,
+        default=None,
+        help="Version tag for this upload (e.g., v1.0.0)"
+    )
+    parser.add_argument(
+        "--tags", "-t",
+        nargs="+",
+        default=["speaker-verification", "speech-ssl", "hubert", "self-supervised"],
+        help="Tags to add to the model (space-separated)"
+    )
+    parser.add_argument(
+        "--private",
+        action="store_true",
+        help="Create as private repository"
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="Simulate upload without actually uploading"
+    )
+    parser.add_argument(
+        "--commit-message", "-m",
+        type=str,
+        default=None,
+        help="Custom commit message"
+    )
+    parser.add_argument(
+        "--no-verify-checksums",
+        action="store_true",
+        help="Skip checksum generation and verification"
+    )
+    parser.add_argument(
+        "--max-file-size",
+        type=float,
+        default=10.0,
+        help="Maximum file size in GB (default: 10.0)"
+    )
+    parser.add_argument(
+        "--no-create",
+        action="store_true",
+        help="Don't create repository if it doesn't exist"
+    )
+    
+    return parser.parse_args()
+
+
+def main():
+    """Main entry point."""
+    args = parse_args()
+    
+    # Create upload configuration
+    upload_config = UploadConfig(
+        checkpoint_dir=args.checkpoint_dir,
+        repo_id=args.repo_id,
+        version=args.version,
+        tags=args.tags,
+        private=args.private,
+        dry_run=args.dry_run,
+        commit_message=args.commit_message,
+        verify_checksums=not args.no_verify_checksums,
+        max_file_size_gb=args.max_file_size,
+        create_if_missing=not args.no_create
+    )
+    
+    # Create uploader and execute
+    uploader = DELULUUploader(upload_config)
+    success = uploader.upload()
+    
+    sys.exit(0 if success else 1)
+
+
+if __name__ == "__main__":
+    main()