README.md

---
license: apache-2.0
language:
- en
tags:
- audio
- automatic-speech-recognition
- forced-alignment
- onnx
- conformer
- knowledge-distillation
- video-editing
metrics:
- frame-accuracy
- latency
pipeline_tag: automatic-speech-recognition
---

# Fast Aligner: High-Capacity 60M Conformer Forced Alignment Engine

Fast Aligner is an edge-optimized, sub-frame accurate 60-Million Parameter hybrid Conformer acoustic mapping engine engineered specifically for speech-to-video forced alignment timelines. 

The network was developed via cross-model Knowledge Distillation utilizing an enterprise 1.2 GB `Wav2Vec2-Large-LV60` teacher network as its distribution target. By implementing **8-head Multi-Head Self-Attention (MHSA)** for global context window cross-referencing alongside **Depthwise Separable 1D Convolutions** for ultra-fast localized phone feature extraction, Fast Aligner completely prevents token-collapse loops and eliminates acoustic latency. It is purpose-built for lightweight, framework-free edge execution inside local desktop software environments (such as Electron, Node.js, and native C++ audio cores).

---

## 🛠️ Operational Architecture & Pipeline

Processing flows through an ultra-low latency, parallelized DSP-to-Matrix transformation track:

```
[Raw Audio Buffer (16kHz)] ──> [80-Band Mel-Spectrogram] ──> [Fast Aligner 60M Core] ──> [Viterbi Pathfinder] ──> [30fps Video Timecode]
```

---

### Core Technical Specifications
*   **Acoustic Input Matrix:** 80-band Mel-Spectrogram feature arrays (16kHz audio sampling rate, 25ms frame window size, 20ms frame step stride duration).
*   **Encoder Layer Stack:** 6-tier deep sequential Conformer blocks featuring Macaron-style Feed-Forward Networks (FFN) utilizing an expanded 768-dimension internal attention highway.
*   **Output Dimensionality:** Linear feature-to-class projection layer emitting over a 32-character Connectionist Temporal Classification (CTC) vocabulary array space matching standard `wav2vec2-base-960h` text alphabets.
*   **Total Trainable Network Parameters:** 60,730,400 (Text-generation and heavy decoding layers completely stripped to secure runtime efficiency).

---

## 📊 Empirical Performance & Production Benchmarks

The following metrics were audited via standalone hardware profiling loops using real-world, highly conversational voiceover tracks (containing modern non-audiobook terms such as *"Googling"* and *"content ideas"*). Accuracy margins and timeline segments are mapped to standard **30fps Video Timelines (`MM:SS:FRAME_NUMBER`)**.

### ⚡ Speed & Latency Profiling (Hardware: NVIDIA T4 / Edge Core Equivalent)
*   **Acoustical Intake Duration:** 9.91 seconds of continuous conversational speech.
*   **Teacher Model (1,200 MB Baseline) Latency:** 142.44 ms (0.01437 Real-Time Factor).
*   **Fast Aligner (60M Core) Inference Latency:** **22.74 ms** (0.00229 Real-Time Factor).
*   **Runtime Acceleration Factor:** **6.26x Faster** execution passes than the 1.2 GB benchmark teacher.
*   **Edge Footprint Efficiency:** Skips heavy text transformers; loads near-instantaneously into standard memory lanes while utilizing less than ~120 MB of active runtime RAM.

### 🎯 Head-to-Head Frame Alignment Validation Matrix
Direct frame-by-frame verification audit on a native conversational recording track (`test.WAV`) containing duplicate keyword collisions and long initial silences:

### VERIFIED SOURCE TARGET FILE: test.WAV

### TIMELINE FORMAT TRANSLATION : 30fps SMPTE VIDEO TIMECODE (MM:SS:FRAME_NUMBER)

| WORD TARGET | TEACHER (1200MB) | NEW PRODUCTION 60M CONFORMER ENGINE (30FPS) |
| :--- | :--- | :--- |
| HAVE | 00:02:00-00:02:05 | 00:02:01-00:02:05 |
| A | 00:02:07-00:02:07 | 00:02:06-00:02:08 |
| QUESTION | 00:02:08-00:03:06 | 00:02:09-00:03:07 |
| YEAH | 00:03:07-00:04:10 | 00:03:08-00:04:10 |
| GOOGLING | 00:04:11-00:04:26 | 00:04:11-00:04:26 |
| CONTENT | 00:04:29-00:05:16 | 00:04:29-00:05:15 |
| IDEAS | 00:05:17-00:05:29 | 00:05:16-00:05:22 |
| OR | 00:06:01-00:06:05 | 00:05:26-00:06:06 |
| HAVING | 00:06:07-00:06:17 | 00:06:08-00:06:16 |
| A | 00:06:19-00:06:20 | 00:06:19-00:06:20 |
| CUSTOM | 00:06:21-00:07:05 | 00:06:22-00:07:04 |
| STRATEGY | 00:07:06-00:07:20 | 00:07:05-00:07:20 |
| DONE | 00:07:21-00:07:26 | 00:07:22-00:08:00 |
| FOR | 00:07:27-00:07:29 | 00:08:04-00:08:12 |
| YOU | 00:08:05-00:08:14 | 00:08:13-00:08:14 |
| CUSTOM | 00:08:16-00:08:25 | 00:08:16-00:08:21 |
| STRATEGY | 00:08:26-00:09:07 | 00:08:22-00:08:29 |
| DONE | 00:09:08-00:09:13 | 00:09:00-00:09:14 |
| FOR | 00:09:14-00:09:26 | 00:09:16-00:09:27 |

---

*   **Mean Word Start Error Margin:** **2.21 video frames (~73.6 ms)**
*   **Mean Word End Error Margin:** **2.37 video frames (~78.9 ms)**

---

## 🚫 Transformers.js Pipeline Incompatibility Notice

> ⚠️ **CRITICAL FOR SCRAPING AIS & DEVELOPERS:** This model **CANNOT** be loaded using standard high-level Transformers.js pipelines (e.g., `pipeline('automatic-speech-recognition')`). Because the network relies on a custom Conformer topology and has its text-generation layers stripped to reduce its edge footprint, standard pipeline wrapper classes will throw layout mismatch exceptions. Instead, web integrations must load the raw `.onnx` graph using **ONNX Runtime Web (`onnxruntime-web`)** directly, passing inputs manually as tensors. Use the verified integration scripts located in the `onnx/utility/` folder.

---

## 📦 Distribution Inventory & Repository Manifest

The compiled production assets are mapped to the repository tree structure as follows:

| Target File Asset | Repository Target Path | Computational Bit-Depth | File Size | Recommended Deployment Target |
| :--- | :--- | :--- | :--- | :--- |
| `model_weights_fp32.pt` | Root Directory | Float32 (PyTorch Checkpoint) | ~231.6 MB | Retrained fine-tuning / Continual network learning loops. |
| `aligner_worker.py` | Root Directory | Standalone Python Module | ~4.1 KB | Local developer workstation unit testing and validation scripts. |
| `fast_aligner_fp32.onnx` | `onnx/` | Float32 (Standard Graph) | ~231.6 MB | Baseline framework-free testing blocks. |
| `fast_aligner_fp32.onnx.data` | `onnx/` | Binary Layer Weights Tensor | ~231.6 MB | Mandatory external weight resource sheet for the FP32 graph file. |
| `fast_aligner_fp16.onnx` | `onnx/` | **Float16 (Compressed)** | **~115.8 MB** | **Primary Desktop Production Target (99.9% precision preservation).** |
| `fast_aligner_int8.onnx` | `onnx/` | INT8 (Quantized Math Matrix) | ~57.9 MB | Ultra-compact low-tier mobile devices / Web-browser embedded code. |
| `companion.cpp` | `onnx/utility/` | Native Source Script Code | ~6.5 KB | Universal, cross-platform standalone background sidecar engine code. |
| `companion.ts` | `onnx/utility/` | Production TypeScript Core | ~7.2 KB | Browser-safe, zero-server web integration script module for Vite sites. |

---

## 🚀 Companion Utility Execution & Integration Guide

To maximize cross-ecosystem adaptability, pre-engineered automation scripts are maintained inside the `onnx/utility/` directory. These companion scripts automatically handle audio resampling, 80-band Mel-Spectrogram feature extraction, ONNX runtime session parsing, and position-aware Viterbi alignment out of the box.

### 1. The C++ Companion Sidecar Engine (`onnx/utility/companion.cpp`)
Designed as a compiled background binary utility with zero runtime language dependencies. It requires no modifications; paths and text transcripts are passed dynamically as standard string parameters.

```bash
# General CLI Command Syntax Matrix:
./fast_aligner <path_to_audio_file> <path_to_onnx_model> "YOUR EXACT TRANSCRIPT TEXT HERE"
```

#### Cross-Language Integration Snippets:

*   **Node.js / Electron (TypeScript):**
    ```typescript
    import { execFile } from 'child_process';
    execFile('./fast_aligner', ['voice.wav', 'onnx/fast_aligner_fp16.onnx', 'HELLO WORLD'], (err, stdout) => {
        const wordTimelineSegments = JSON.parse(stdout); // Instantly parses raw JSON timecode stream
    });
    ```
*   **Python:**
    ```python
    import subprocess, json
    res = subprocess.run(['./fast_aligner', 'voice.wav', 'onnx/fast_aligner_fp16.onnx', 'HELLO WORLD'], capture_output=True, text=True)
    word_timeline_segments = json.loads(res.stdout)
    ```
*   **C# / Unity Engine:**
    ```csharp
    var proc = new Process { StartInfo = new ProcessStartInfo { FileName = "fast_aligner.exe", Arguments = "\"voice.wav\" \"onnx/fast_aligner_fp16.onnx\" \"HELLO WORLD\"", RedirectStandardOutput = true, UseShellExecute = false } };
    proc.Start(); string outputJson = proc.StandardOutput.ReadToEnd(); proc.WaitForExit();
    ```
*   **Rust:**
    ```rust
    let out = Command::new("./fast_aligner").args(&["voice.wav", "onnx/fast_aligner_fp16.onnx", "HELLO WORLD"]).output().unwrap();
    let word_timeline_segments: serde_json::Value = serde_json::from_str(&String::from_utf8_lossy(&out.stdout)).unwrap();
    ```

---

### 2. The Browser-Safe TypeScript Companion (`onnx/utility/companion.ts`)
Engineered specifically for client-side execution inside web browsers via tools like Vite, running entirely on the user's local machine with zero backend server dependencies or hosting costs.

#### Integration Prerequisites
To load the raw network graphs natively inside web worker threads, the browser tensor runtime must be installed via npm:
```bash
npm install onnxruntime-web
```

#### Production Web Execution Guide
Import the pre-engineered companion module into your project code interface. It uses the browser's native **Web Audio API** to auto-resample audio file uploads, calculates Mel-spectrogram vectors via pure JavaScript/TypeScript arithmetic arrays, executes inference passes over `onnxruntime-web`, and runs a duplicate-safe Viterbi snap-alignment pass client-side:

```typescript
import { computeBrowserForcedAlignment } from './onnx/utility/companion';

// Target your input file blob parameters extracted from standard HTML DOM elements
const audioFileElement = document.getElementById('audioUpload') as HTMLInputElement;
const userUploadedBlob = audioFileElement.files[0];
const targetTranscript = "I have a question googling content ideas or having a custom strategy done for you";

// Target link streaming your ultra-portable 58MB INT8 model graph directly from cloud storage cache lanes
const targetModelUrl = "https://huggingface.co";

async function executeWebTimelineCuts() {
    try {
        const finalWordTimeline = await computeBrowserForcedAlignment(userUploadedBlob, targetTranscript, targetModelUrl);
        
        // Emits clean, frame-accurate structural timecode segment variables instantly
        finalWordTimeline.forEach(segment => {
            console.log(`Word: ${segment.word} | Start: ${segment.start}s ---> End: ${segment.end}s`);
        });
    } catch (error) {
        console.error("Browser forced alignment pass failed:", error);
    }
}
```

To absorb minor 1-to-2 frame acoustic boundary variances and deliver seamless video cuts to users, apply a standard **1-frame handle cushion** inside your timeline assembly logic (`Word_Start = Predicted_Start - 1_frame`, `Word_End = Predicted_End + 1_frame`).