Update custom model files, README, and requirements

README.md

@@ -1,199 +1,54 @@
 ---
-library_name: transformers
-tags: []
+license: mit
+language:
+- en
+datasets:
+- speechbrain/LoquaciousSet
+base_model:
+- facebook/hubert-xlarge-ls960-ft
+- HuggingFaceTB/SmolLM3-3B
+pipeline_tag: automatic-speech-recognition
+tags:
+- asr
+- speech-recognition
+- audio
+- smollm
+- hubert
 ---
 
-# Model Card for Model ID
+# Tiny Audio Model Card
 
-<!-- Provide a quick summary of what the model is/does. -->
+This model was born from a simple idea: what if anyone could train a powerful, modern speech recognition model for the price of a few coffees? This model is the result of the [Tiny Audio course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md), a free, hands-on guide to building your own ASR system from scratch.
 
+## The Story of this Model
 
+This model isn't the product of a massive research lab with an unlimited budget. It's the result of a 24-hour training run on a single GPU, made possible by an efficient projector-only training approach. By combining the strengths of a massive pretrained audio encoder (`facebook/hubert-xlarge-ls960-ft`) and a powerful language model (`HuggingFaceTB/SmolLM3-3B`), and only training a small projector between them, we can create a high-quality ASR model with minimal resources.
 
-## Model Details
+This model is a testament to the power of open-source and the incredible tools and models that are now available to everyone.
 
-### Model Description
+## Intended Use
 
-<!-- Provide a longer summary of what this model is. -->
+This model is for you. It's for the curious, the builders, the learners. It's for anyone who wants to understand how modern AI works by getting their hands dirty. Use it to transcribe your podcasts, your meetings, your voice memos. But more importantly, use it as a starting point. Fork it, fine-tune it, break it, and make it your own.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+## Performance
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+This model achieves a Word Error Rate (WER) of **12.14%** on the LoquaciousSet test set. It's not perfect, but it's a solid baseline that you can build on. See how it compares to other models on the [community leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard).
 
-### Model Sources [optional]
+## How to Use
 
-<!-- Provide the basic links for the model. -->
+```python
+from transformers import pipeline
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+pipe = pipeline("automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True)
 
-## Uses
+result = pipe("path/to/audio.wav")
+print(result["text"])
+```
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+## How to Get Involved
 
-### Direct Use
+This project is more than just a model; it's a community. Here's how you can get involved:
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+- **Take the course**: The best way to start is to go through the [free 6-hour course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md) and train your own model.
+- **Share your results**: Add your model to the [leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard) and share what you've learned.
+- **Join the conversation**: Ask questions, share your ideas, and connect with other builders in the [GitHub Discussions](https://github.com/alexkroman/tiny-audio/discussions).

asr_modeling.py

@@ -573,6 +573,9 @@ class ASRModel(PreTrainedModel):
             if audio_fill_value is None:
                 audio_fill_value = fill_value
 
+        # At this point tensor_to_expand is guaranteed to be a Tensor
+        assert tensor_to_expand is not None
+
         # Create output tensor
         expanded = torch.full(
             (batch_size, new_seq_len),

asr_pipeline.py

@@ -27,6 +27,7 @@ class ASRPipeline(transformers.AutomaticSpeechRecognitionPipeline):
         else:
             # Fallback to whisper-tiny tokenizer for its normalize() method only
             from transformers import WhisperTokenizer
+
             self.text_normalizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny")
 
     def __call__(self, inputs, **kwargs):

asr_processing.py

@@ -57,6 +57,12 @@ class ASRProcessor(ProcessorMixin):
         else:
             processor_config = {}
 
+        # Filter out any non-serializable objects that might have been added
+        processor_config = {
+            k: v for k, v in processor_config.items()
+            if isinstance(v, (str, int, float, bool, list, dict, type(None)))
+        }
+
         # Add/update processor metadata while preserving feature extractor settings
         feature_extractor_type = self.feature_extractor.__class__.__name__
         processor_config.update(

handler.py

@@ -0,0 +1,152 @@
+"""Custom inference handler for HuggingFace Inference Endpoints."""
+
+from typing import Any, Dict, List, Union
+
+import torch
+
+try:
+    # For remote execution, imports are relative
+    from .asr_modeling import ASRModel
+    from .asr_pipeline import ASRPipeline
+except ImportError:
+    # For local execution, imports are not relative
+    from asr_modeling import ASRModel  # type: ignore[no-redef]
+    from asr_pipeline import ASRPipeline  # type: ignore[no-redef]
+
+
+class EndpointHandler:
+    def __init__(self, path: str = ""):
+        # Set environment variables for PyTorch/CUDA (must be before imports/operations)
+        import os
+
+        # Enable expandable segments to reduce fragmentation
+        os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+        # Enable TF32 for faster matmul on A40/A100
+        torch.backends.cuda.matmul.allow_tf32 = True
+        torch.backends.cudnn.allow_tf32 = True
+
+        # Set device and dtype
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.dtype = torch.bfloat16 if self.device == "cuda" else torch.float32
+
+        # Enable CUDA optimizations
+        if torch.cuda.is_available():
+            torch.backends.cudnn.benchmark = True
+
+        # Prepare model kwargs for pipeline
+        model_kwargs = {
+            "dtype": self.dtype,
+            "low_cpu_mem_usage": True,
+        }
+        if torch.cuda.is_available():
+            model_kwargs["attn_implementation"] = (
+                "flash_attention_2" if self._is_flash_attn_available() else "sdpa"
+            )
+
+        # Load model (this loads the model, tokenizer, and feature extractor)
+        self.model = ASRModel.from_pretrained(path, **model_kwargs)
+
+        # Instantiate custom pipeline - it will get feature_extractor and tokenizer from model
+        self.pipe = ASRPipeline(
+            model=self.model,
+            feature_extractor=self.model.feature_extractor,
+            tokenizer=self.model.tokenizer,
+            device=self.device,
+        )
+
+        # Apply torch.compile if enabled (after model is loaded by pipeline)
+        # Enable by default for significant speedup (20-40%)
+        if torch.cuda.is_available() and os.getenv("ENABLE_TORCH_COMPILE", "1") == "1":
+            compile_mode = os.getenv("TORCH_COMPILE_MODE", "reduce-overhead")
+            self.model = torch.compile(self.model, mode=compile_mode)
+            # Update the pipeline with the compiled model
+            self.pipe.model = self.model
+
+        # Warmup the model
+        if torch.cuda.is_available():
+            self._warmup()
+
+    def _is_flash_attn_available(self):
+        """Check if flash attention is available."""
+        import importlib.util
+
+        return importlib.util.find_spec("flash_attn") is not None
+
+    def _warmup(self):
+        """Warmup to trigger model compilation and allocate GPU memory."""
+        try:
+            # Create dummy audio (1 second at config sample rate)
+            sample_rate = self.pipe.model.config.audio_sample_rate
+            dummy_audio = torch.randn(sample_rate, dtype=torch.float32)
+
+            # The pipeline now handles GPU optimization internally
+            with torch.inference_mode():
+                warmup_tokens = self.pipe.model.config.inference_warmup_tokens
+                _ = self.pipe(
+                    {"raw": dummy_audio, "sampling_rate": sample_rate}, max_new_tokens=warmup_tokens
+                )
+
+            # Force CUDA synchronization to ensure kernels are compiled
+            if torch.cuda.is_available():
+                torch.cuda.synchronize()
+                # Clear cache after warmup to free memory
+                torch.cuda.empty_cache()
+
+        except Exception as e:
+            print(f"Warmup skipped due to: {e}")
+
+    def __call__(self, data: Dict[str, Any]) -> Union[Dict[str, Any], List[Dict[str, Any]]]:
+        """Process audio transcription request.
+
+        Supports both single and batch inputs for efficient concurrent processing.
+        The endpoint infrastructure can batch multiple concurrent requests automatically.
+        """
+        inputs = data.get("inputs")
+        if inputs is None:
+            raise ValueError("Missing 'inputs' in request data")
+
+        # Get generation parameters (matching SLAM-ASR paper defaults)
+        params = data.get("parameters", {})
+        max_new_tokens = params.get("max_new_tokens", 200)  # Longer transcripts
+
+        # Beam search for better quality (5 beams for higher quality)
+        # Use num_beams=1 for faster inference at cost of ~2-3% WER increase
+        num_beams = params.get("num_beams", 5)
+
+        do_sample = params.get("do_sample", False)
+
+        # Length penalty encourages appropriate transcript length
+        # >1.0 = prefer longer outputs, <1.0 = prefer shorter
+        # Slight positive bias helps avoid truncated transcripts
+        length_penalty = params.get("length_penalty", 1.1)
+
+        # Repetition penalty to prevent loops (1.1-1.2 is good for ASR)
+        repetition_penalty = params.get("repetition_penalty", 1.15)
+
+        # Alternative: use no_repeat_ngram_size to prevent exact n-gram repetition
+        no_repeat_ngram_size = params.get("no_repeat_ngram_size", 3)
+
+        # Early stopping for beam search: stop when all beams end
+        # "never" = generate full max_new_tokens (more accurate but slower)
+        # True = stop when all beams reach EOS (faster)
+        early_stopping = params.get("early_stopping", True)
+
+        # Diversity penalty encourages different beams (helps with rare words)
+        # 0.0 = no diversity, 0.5-1.0 = good diversity
+        default_diversity = self.pipe.model.config.inference_diversity_penalty
+        diversity_penalty = params.get("diversity_penalty", default_diversity)
+
+        # The pipeline's __call__ method handles both single and batch inputs
+        # as well as automatic chunking for long audio files
+        return self.pipe(
+            inputs,
+            max_new_tokens=max_new_tokens,
+            num_beams=num_beams,
+            do_sample=do_sample,
+            length_penalty=length_penalty,
+            repetition_penalty=repetition_penalty,
+            no_repeat_ngram_size=no_repeat_ngram_size,
+            early_stopping=early_stopping,
+            diversity_penalty=diversity_penalty,
+        )

requirements.txt

@@ -0,0 +1,5 @@
+# Core dependencies for tiny-audio model inference
+# This file is pushed to HuggingFace for model repository
+
+# Transformers - main library for model loading and inference
+transformers>=4.57.0