Spaces:
Configuration error
Configuration error
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -9,27 +9,19 @@ app_file: app.py
|
|
| 9 |
pinned: false
|
| 10 |
---
|
| 11 |
|
| 12 |
-
# Anvaya: Multi-Modal Speech
|
| 13 |
|
| 14 |
-
## Abstract
|
| 15 |
|
| 16 |
-
Anvaya is an open-source
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
1. **Neural Disfluency Classifier**: A fine-tuned Wav2Vec 2.0 architecture with Low-Rank Adaptation (LoRA) and Focal Loss ($\gamma = 2.0$) for detecting syllable repetitions, sound prolongations, and glottal blocks.
|
| 21 |
-
2. **Phonetic Goodness of Pronunciation (GOP) & Dynamic Alignment**: An acoustic CTC decoding engine utilizing dynamic programming (Needleman-Wunsch with human phonetic tolerance) to pinpoint exact word-level substitutions, omissions, and insertions.
|
| 22 |
-
3. **Specific Phonological Disorder Classifiers**: Clinical rule-based acoustic analyzers that explicitly identify:
|
| 23 |
-
- **Rhotacism**: Substitution of rhotic /r/ with /w/ or /l/ (e.g., *red* to *wed*, *rabbit* to *wabbit*).
|
| 24 |
-
- **Sigmatism (Lisping)**: Substitution or distortion of sibilants /s/, /z/, /sh/ with dental fricatives /th/, /f/ (e.g., *sun* to *thun*, *sweet* to *thweet*).
|
| 25 |
-
4. **Biomechanical Phonation Acoustics (Praat)**: Quantitative vocal fold dynamics evaluating Pitch ($F_0$), cycle-to-cycle Period Jitter, Amplitude Shimmer, and Harmonics-to-Noise Ratio (HNR).
|
| 26 |
-
5. **Multi-Modal Decision Fusion**: A continuous non-linear fusion engine that computes a 0 to 100 Fluency Index and allows single-sample per-speaker calibration ("My Normal").
|
| 27 |
|
| 28 |
---
|
| 29 |
|
| 30 |
-
## 1. System Architecture
|
| 31 |
|
| 32 |
-
The
|
| 33 |
|
| 34 |
```
|
| 35 |
===================================================================================
|
|
@@ -40,10 +32,10 @@ The following diagram illustrates the end-to-end data flow from raw acoustic inp
|
|
| 40 |
v
|
| 41 |
+---------------------------------------------------------------------------------+
|
| 42 |
| ACOUSTIC SIGNAL PRECONDITIONING |
|
| 43 |
-
|
|
| 44 |
-
|
|
| 45 |
-
|
|
| 46 |
-
|
|
| 47 |
+---------------------------------------------------------------------------------+
|
| 48 |
|
|
| 49 |
+-----------------------------+-----------------------------+
|
|
@@ -65,227 +57,105 @@ The following diagram illustrates the end-to-end data flow from raw acoustic inp
|
|
| 65 |
v v
|
| 66 |
+-----------------------------------+ +-----------------------------------+
|
| 67 |
| EXPERT MODULE 3: | | EXPERT MODULE 4: |
|
| 68 |
-
|
|
| 69 |
-
| -
|
| 70 |
-
| -
|
| 71 |
-
| -
|
| 72 |
-
| Outputs:
|
| 73 |
+-----------------------------------+ +-----------------------------------+
|
| 74 |
|
|
| 75 |
v
|
| 76 |
+---------------------------------------------------------------------------------+
|
| 77 |
-
| MULTI-MODAL
|
| 78 |
-
| -
|
| 79 |
-
| -
|
| 80 |
-
| -
|
| 81 |
-
| - Evidence Audit Trail
|
| 82 |
+---------------------------------------------------------------------------------+
|
| 83 |
```
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
-
## 2.
|
| 88 |
-
|
| 89 |
-
A core requirement of clinical machine learning is reproducibility and verifiable provenance. Anvaya trains on a unified hybrid dataset of 10,326 audio clips combining clinical speech archives with physically generated synthetic disfluency lattices.
|
| 90 |
-
|
| 91 |
-
### 2.1 Component Data Sources
|
| 92 |
-
|
| 93 |
-
| Corpus Name | Source / Institution | Sample Count | Audio Format | Primary Characteristics |
|
| 94 |
-
| :--- | :--- | :--- | :--- | :--- |
|
| 95 |
-
| **UCLASS** | University College London | 3,124 clips | 16 kHz Mono WAV | Authentic clinical monologue & reading from individuals who stutter |
|
| 96 |
-
| **SEP-28k** | Apple ML / Univ. Wisconsin | 2,202 clips | 16 kHz Mono WAV | Real-world conversational audio with labeled disfluency subtypes |
|
| 97 |
-
| **Kisinga / LibriStutter**| Academic Repositories | 1,000 clips | 16 kHz Mono WAV | Clean fluent reference reading passages & paired disfluencies |
|
| 98 |
-
| **Synthetic Lattice** | Locally Generated (DSP) | 4,000 clips | 16 kHz PCM-16 WAV | Physiologically synthesized repetitions, prolongations & blocks |
|
| 99 |
-
| **Total Composite Corpus**| **Anvaya Unified Dataset**| **10,326 clips** | **16 kHz PCM-16 WAV**| **Balanced benchmark covering all speech pathology classes** |
|
| 100 |
-
|
| 101 |
-
### 2.2 Why Synthetic Acoustic Lattices Were Introduced
|
| 102 |
-
|
| 103 |
-
Real-world clinical stuttering datasets (such as SEP-28k and UCLASS) present three major challenges:
|
| 104 |
-
1. **Severe Class Imbalance**: Natural clinical recordings contain far fewer blocks and prolongations than fluent filler words.
|
| 105 |
-
2. **Transcription Ambiguity**: Transcribers often omit stuttered syllables or approximate them with inconsistent punctuation.
|
| 106 |
-
3. **Background Acoustic Noise**: Audio recorded across different clinics introduces confounding acoustic variables (reverberation, microphone types).
|
| 107 |
-
|
| 108 |
-
To solve this, Anvaya implements a **Physiologically Grounded Disfluency Synthesizer** (`ml/data/lattice_synth.py`). It takes pristine baseline speech (LibriSpeech / CMU ARCTIC) and physically constructs disfluent events using acoustic digital signal processing.
|
| 109 |
-
|
| 110 |
-
### 2.3 Local Physical Storage Layout
|
| 111 |
-
|
| 112 |
-
All 4,000 synthesized audio clips are stored as real physical `.wav` files on local disk:
|
| 113 |
-
|
| 114 |
-
```
|
| 115 |
-
speech-model/
|
| 116 |
-
|-- data/
|
| 117 |
-
| |-- synthetic_lattice/
|
| 118 |
-
| | |-- metadata.csv # Full CSV with audio paths, transcripts, labels
|
| 119 |
-
| | |-- audio/
|
| 120 |
-
| | |-- synth_000000_fluent_control.wav
|
| 121 |
-
| | |-- synth_001000_stutter_repetition.wav
|
| 122 |
-
| | |-- synth_002000_stutter_prolongation.wav
|
| 123 |
-
| | |-- synth_003000_stutter_block.wav
|
| 124 |
-
| | |-- ... (4,000 physical WAV files)
|
| 125 |
-
| |-- metadata/
|
| 126 |
-
| |-- hybrid_dataset/ # Arrow / Parquet unified dataset (10,326 clips)
|
| 127 |
-
| |-- train/
|
| 128 |
-
| |-- validation/
|
| 129 |
-
| |-- test/
|
| 130 |
-
```
|
| 131 |
-
|
| 132 |
-
---
|
| 133 |
-
|
| 134 |
-
## 3. Digital Signal Processing & Lattice Synthesis Methodology
|
| 135 |
-
|
| 136 |
-
The synthetic lattice pipeline models speech production anatomy through four distinct digital signal processing transformations:
|
| 137 |
-
|
| 138 |
-
### 3.1 Zero-Crossing and Hann Splice Windows
|
| 139 |
-
To eliminate audible clicks, phase jumps, and spectral discontinuities at cut boundaries, cut points are snapped to rising zero-crossings:
|
| 140 |
-
|
| 141 |
-
$$\text{ZC}(x) = \{ i \mid x[i-1] < 0 \land x[i] \ge 0 \}$$
|
| 142 |
-
|
| 143 |
-
Splices are blended using a symmetric Hann window of length $L = 160$ samples (10 ms at 16,000 Hz):
|
| 144 |
-
|
| 145 |
-
$$w_{\text{out}}[n] = 0.5 \left(1 - \cos\left(\frac{2\pi (n + L/2)}{L-1}\right)\right), \quad n \in [0, L/2]$$
|
| 146 |
-
|
| 147 |
-
$$w_{\text{in}}[n] = 0.5 \left(1 - \cos\left(\frac{2\pi n}{L-1}\right)\right), \quad n \in [0, L/2]$$
|
| 148 |
-
|
| 149 |
-
$$\text{Overlap}[n] = (x_A[n] \cdot w_{\text{out}}[n]) + (x_B[n] \cdot w_{\text{in}}[n])$$
|
| 150 |
-
|
| 151 |
-
### 3.2 Part-Word Syllable Repetition Engine
|
| 152 |
-
Onset syllables (100 to 180 ms duration) are isolated and repeated $k \in \{2, 3, 4\}$ times. Human speech repetitions naturally exhibit muscular decay and micro-pitch instability, modeled as:
|
| 153 |
|
| 154 |
-
|
| 155 |
|
| 156 |
-
|
| 157 |
|
| 158 |
-
|
| 159 |
-
Sound prolongations occur on continuants (vowels and fricatives). To extend a phoneme by $\alpha \in [3.0, 6.0]$ times without altering pitch or introducing robotic metallic artifacts, Waveform Similarity Overlap-Add (WSOLA) is applied. This preserves formant trajectories ($F_1, F_2, F_3$) while expanding the temporal envelope.
|
| 160 |
-
|
| 161 |
-
### 3.4 Glottal Tension Block Engine
|
| 162 |
-
Laryngeal blocks (spasmodic closure of vocal folds prior to phonation) are synthesized via:
|
| 163 |
-
1. Complete acoustic energy attenuation ($< -48\text{ dBFS}$) for 200 to 700 ms.
|
| 164 |
-
2. An explosive glottal burst (transient high-frequency energy release).
|
| 165 |
-
3. Post-block pitch perturbation settling over 80 ms.
|
| 166 |
-
|
| 167 |
-
---
|
| 168 |
-
|
| 169 |
-
## 4. Formal Evaluation & Benchmark Results
|
| 170 |
-
|
| 171 |
-
The system was evaluated on a held-out test split of 1,666 clips, standardized phonetic reading passages, and clinical validation test suites.
|
| 172 |
-
|
| 173 |
-
### 4.1 Quantitative Performance Summary
|
| 174 |
-
|
| 175 |
-
| Evaluation Domain | Metric Name | Measured Score | Clinical Meaning / Interpretation |
|
| 176 |
| :--- | :--- | :--- | :--- |
|
| 177 |
-
| **
|
| 178 |
-
| | **
|
| 179 |
-
| | **
|
| 180 |
-
| | **
|
| 181 |
-
|
|
| 182 |
-
| | **
|
| 183 |
-
| | **
|
| 184 |
-
|
|
| 185 |
-
|
|
| 186 |
-
| | **
|
| 187 |
-
|
|
| 188 |
-
| | **
|
| 189 |
-
|
|
| 190 |
-
| | **Silence False Positives**| **0.00%** | Silence guard completely prevents false pathology flags |
|
| 191 |
|
| 192 |
---
|
| 193 |
|
| 194 |
-
##
|
| 195 |
|
| 196 |
-
|
|
|
|
| 197 |
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
+-----------------------------------------------------------------------------------------+
|
| 202 |
-
| 1. Pronunciation Word Accuracy | 95.80% | Accuracy = (1.0 - WER) * 100% |
|
| 203 |
-
| 2. 'R' Phoneme Sound Accuracy | 96.40% | Precision on detecting R-to-W/L substitutions |
|
| 204 |
-
| 3. 'S' Phoneme Sound Accuracy | 94.80% | Precision on detecting S-to-TH/F lisps |
|
| 205 |
-
| 4. Continuous Fluency Index | 0..100 | Multi-Modal Fused Composite Speech Score |
|
| 206 |
-
+-----------------------------------------------------------------------------------------+
|
| 207 |
-
```
|
| 208 |
-
|
| 209 |
-
1. **Pronunciation Word Accuracy (95.80%)**:
|
| 210 |
-
$$\text{Pronunciation Accuracy} = \left(1.0 - \text{WER}\right) \times 100\% = \left(\frac{N_{\text{ref}} - \text{Errors}}{N_{\text{ref}}}\right) \times 100\%$$
|
| 211 |
-
Measures the exact proportion of target words correctly articulated without substitutions, omissions, or distortions. Displayed directly in the web dashboard header.
|
| 212 |
|
| 213 |
-
|
| 214 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 215 |
|
| 216 |
-
|
| 217 |
-
Measures the specific detection accuracy for Sigmatism (sibilant lisps, e.g., saying *thun* instead of *sun*, *thweet* instead of *sweet*). Evaluates high-frequency spectral energy distribution above 3.5 kHz.
|
| 218 |
-
|
| 219 |
-
4. **Continuous Overall Speech Accuracy (Fluency Index 0 to 100)**:
|
| 220 |
-
The unified composite score combining **Pronunciation Accuracy (45% weight)**, **Acoustic Disfluency Probability (40% weight)**, and **Biomechanical Vocal Stability (15% weight)**.
|
| 221 |
-
|
| 222 |
-
---
|
| 223 |
|
| 224 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
|
| 226 |
-
|
| 227 |
-
To align what the user said (Hypothesis $H$) against what they were supposed to say (Reference $R$), dynamic programming computes the optimal Levenshtein alignment matrix:
|
| 228 |
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
D(i-1, j) + 1 & (\text{Omission / Unspoken Word}) \\
|
| 232 |
-
D(i, j-1) + 1 & (\text{Insertion / Extra Spoken Word})
|
| 233 |
-
\end{cases}$$
|
| 234 |
|
| 235 |
-
|
| 236 |
|
| 237 |
-
|
| 238 |
-
0 & \text{if } R_i = H_j \lor \text{SequenceMatcher}(R_i, H_j) \ge 0.78 \\
|
| 239 |
-
1 & \text{otherwise}
|
| 240 |
-
\end{cases}$$
|
| 241 |
|
| 242 |
-
|
| 243 |
-
Pronunciation accuracy is computed by combining word-level precision with character-level Levenshtein similarity:
|
| 244 |
-
|
| 245 |
-
$$\text{GOP} = 0.75 \cdot \left( \frac{N_{\text{correct}}}{N_{\text{ref}}} \right) + 0.25 \cdot \text{LevenshteinRatio}(R, H)$$
|
| 246 |
-
|
| 247 |
-
When all target words match, $\text{GOP} = 1.0$.
|
| 248 |
-
|
| 249 |
-
### 5.3 Biomechanical Praat Phonation Formulations
|
| 250 |
-
|
| 251 |
-
Acoustic stability is computed on voiced periodic frames ($F_0 \in [75, 500]\text{ Hz}$):
|
| 252 |
-
|
| 253 |
-
- **Local Jitter (Pitch Instability)**:
|
| 254 |
-
$$\text{Jitter} = \frac{\frac{1}{N-1} \sum_{i=1}^{N-1} |T_i - T_{i+1}|}{\frac{1}{N} \sum_{i=1}^{N} T_i}$$
|
| 255 |
-
*Where $T_i$ is the duration of the $i$-th glottal pitch period.*
|
| 256 |
-
|
| 257 |
-
- **Local Shimmer (Amplitude Instability)**:
|
| 258 |
-
$$\text{Shimmer} = \frac{\frac{1}{N-1} \sum_{i=1}^{N-1} |A_i - A_{i+1}|}{\frac{1}{N} \sum_{i=1}^{N} A_i}$$
|
| 259 |
-
*Where $A_i$ is the peak amplitude of the $i$-th glottal pulse.*
|
| 260 |
-
|
| 261 |
-
- **Harmonics-to-Noise Ratio (HNR)**:
|
| 262 |
-
$$\text{HNR} = 10 \cdot \log_{10} \left( \frac{r_{AC}(T_0)}{1 - r_{AC}(T_0)} \right) \text{ dB}$$
|
| 263 |
-
*Where $r_{AC}(T_0)$ is the normalized autocorrelation at fundamental period $T_0$.*
|
| 264 |
-
|
| 265 |
-
### 5.4 Multi-Modal Fluency Index (0 to 100)
|
| 266 |
-
The continuous Fluency Index integrates all modalities into a clinically intuitive score:
|
| 267 |
|
| 268 |
-
|
| 269 |
|
| 270 |
-
|
| 271 |
-
-
|
| 272 |
-
-
|
| 273 |
-
|
|
|
|
|
|
|
| 274 |
|
| 275 |
---
|
| 276 |
|
| 277 |
-
##
|
| 278 |
-
|
| 279 |
-
The codebase is organized into modular packages:
|
| 280 |
|
| 281 |
```
|
| 282 |
speech-model/
|
| 283 |
|-- ml/
|
| 284 |
| |-- model/
|
| 285 |
| | |-- engine.py # Singleton high-speed inference pipeline
|
| 286 |
-
| | |-- infer.py #
|
| 287 |
| | |-- pron_eval.py # CTC ASR decoding, word alignment, 'r'/'s' flaw rules
|
| 288 |
-
| | |-- fusion.py # Multi-modal fusion, self-calibration
|
| 289 |
| | |-- stutter_trainer.py # Wav2Vec2 + LoRA training script with Focal Loss
|
| 290 |
| |-- data/
|
| 291 |
| | |-- lattice_synth.py # DSP disfluency lattice synthesizer
|
|
@@ -294,6 +164,7 @@ speech-model/
|
|
| 294 |
| | |-- precache_models.py # Docker build pre-cache script
|
| 295 |
| |-- cli.py # Unified developer command-line interface
|
| 296 |
|-- webapp.py # Streamlit clinical diagnostic user interface
|
|
|
|
| 297 |
|-- docs/
|
| 298 |
| |-- DATASETS_AND_MODEL_GUIDE.md # Plain-English guide to datasets and model design
|
| 299 |
| |-- DEPLOY_RENDER.md # Step-by-step Render deployment documentation
|
|
@@ -304,12 +175,12 @@ speech-model/
|
|
| 304 |
|-- requirements.txt # Python package dependencies
|
| 305 |
|-- Dockerfile # Production container configuration for Render
|
| 306 |
|-- render.yaml # Render 1-Click deployment blueprint
|
| 307 |
-
|-- README.md #
|
| 308 |
```
|
| 309 |
|
| 310 |
---
|
| 311 |
|
| 312 |
-
##
|
| 313 |
|
| 314 |
The following PowerShell commands allow anyone to clone, set up, and test the entire framework locally on Windows:
|
| 315 |
|
|
@@ -374,6 +245,7 @@ res = engine.diagnose_audio(
|
|
| 374 |
)
|
| 375 |
print('Inference Latency: ', res['latency_ms'], 'ms')
|
| 376 |
print('Clinical Assessment: ', res['decision']['buckets']['overall'].upper())
|
|
|
|
| 377 |
print('Fluency Index: ', res['decision']['fluency_100'], '/ 100')
|
| 378 |
print('Decoded Text: ', res['pronunciation']['asr_hypothesis'])
|
| 379 |
"@
|
|
@@ -392,8 +264,6 @@ Once the web server starts, open your browser and navigate to:
|
|
| 392 |
|
| 393 |
### Alternative: Bash / Linux / macOS Commands
|
| 394 |
|
| 395 |
-
For macOS and Linux environments, use standard bash commands:
|
| 396 |
-
|
| 397 |
```bash
|
| 398 |
# Setup
|
| 399 |
git clone https://github.com/notUbaid/speech-model.git
|
|
|
|
| 9 |
pinned: false
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# Anvaya: Multi-Modal Speech Disfluency & Articulation Screening Prototype
|
| 13 |
|
| 14 |
+
## Abstract & Research Context
|
| 15 |
|
| 16 |
+
Anvaya is an open-source exploratory research framework designed for assistive speech disfluency screening, acoustic phonation analysis, and phonological practice feedback. The system combines fine-tuned self-supervised acoustic representations (Wav2Vec 2.0 with Low-Rank Adaptation), Connectionist Temporal Classification (CTC) Goodness of Pronunciation (GOP), and acoustic correlates of phonation extracted via Parselmouth/Praat.
|
| 17 |
|
| 18 |
+
**Scientific & Clinical Scope Disclaimer**: Anvaya is an educational and screening research prototype. It is **not** an FDA-cleared medical device, nor does it perform direct physical/biomechanical vocal fold imaging. All diagnostic classifications represent acoustic approximations and rule-based heuristic scoring intended for research exploration and self-guided speech practice.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
---
|
| 21 |
|
| 22 |
+
## 1. System Architecture & Subsystem Boundaries
|
| 23 |
|
| 24 |
+
The prototype executes four distinct acoustic feature extractors followed by a rule-weighted fusion scoring layer:
|
| 25 |
|
| 26 |
```
|
| 27 |
===================================================================================
|
|
|
|
| 32 |
v
|
| 33 |
+---------------------------------------------------------------------------------+
|
| 34 |
| ACOUSTIC SIGNAL PRECONDITIONING |
|
| 35 |
+
| - Sample Rate: 16,000 Hz Mono Float32 |
|
| 36 |
+
| - High-Pass Filter: 60 Hz 2nd-Order Butterworth (DC rumble attenuation) |
|
| 37 |
+
| - Dynamic Energy Gate: Silence Guard (RMS thresholding) |
|
| 38 |
+
| - Peak Normalization: Standardized to -1.0 dBFS |
|
| 39 |
+---------------------------------------------------------------------------------+
|
| 40 |
|
|
| 41 |
+-----------------------------+-----------------------------+
|
|
|
|
| 57 |
v v
|
| 58 |
+-----------------------------------+ +-----------------------------------+
|
| 59 |
| EXPERT MODULE 3: | | EXPERT MODULE 4: |
|
| 60 |
+
| Phonetic Flaw Heuristics | | Acoustic Correlates of Phonation |
|
| 61 |
+
| - Rhotic Error Mapping ('r'->'w')| | - Parselmouth / Praat Core |
|
| 62 |
+
| - Sibilant Lisping ('s'->'th') | | - Pitch Period Tracking (F0) |
|
| 63 |
+
| - Word Substitution Rules | | - Local Jitter, Shimmer, HNR |
|
| 64 |
+
| Outputs: Heuristic Flaw Flags | | Outputs: Voice Roughness Metrics |
|
| 65 |
+-----------------------------------+ +-----------------------------------+
|
| 66 |
|
|
| 67 |
v
|
| 68 |
+---------------------------------------------------------------------------------+
|
| 69 |
+
| MULTI-MODAL FUSION & UNCERTAINTY SCORING |
|
| 70 |
+
| - Heuristic Fluency Index (0 to 100): 40% Stutter, 45% Pronunciation, 15% Voice|
|
| 71 |
+
| - Uncertainty & Confidence Scoring: Low / Moderate / High |
|
| 72 |
+
| - Baseline Calibration: Offset Transformation relative to "My Normal" |
|
| 73 |
+
| - Evidence Audit Trail: JSON Telemetry Trace |
|
| 74 |
+---------------------------------------------------------------------------------+
|
| 75 |
```
|
| 76 |
|
| 77 |
---
|
| 78 |
|
| 79 |
+
## 2. Experimental Results & Clinical Screening Metrics
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
+
All metrics reported below were evaluated on a held-out test partition of 1,666 clips from out-of-training speakers to evaluate generalization.
|
| 82 |
|
| 83 |
+
### 2.1 Complete Evaluation Matrix
|
| 84 |
|
| 85 |
+
| Metric Category | Metric Name | Measured Value | Scientific Interpretation & Formula |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
| :--- | :--- | :--- | :--- |
|
| 87 |
+
| **Neural Disfluency** | **Macro-F1 Score** | **71.47%** | Balanced harmonic mean across fluent and disfluent classes |
|
| 88 |
+
| | **Overall Test Accuracy** | **71.49%** | Total correct classifications on held-out test split |
|
| 89 |
+
| | **ROC-AUC** | **0.7669** | Area under the Receiver Operating Characteristic curve |
|
| 90 |
+
| **Clinical Screening**| **Sensitivity (Recall)**| **75.71%** | True Positive Rate: Percentage of actual stutter events identified |
|
| 91 |
+
| | **Specificity (TNR)** | **67.28%** | True Negative Rate: Percentage of fluent segments correctly cleared |
|
| 92 |
+
| | **Precision (PPV)** | **85.05% - 97.58%**| Positive Predictive Value on synthesized & clinical test subsets |
|
| 93 |
+
| | **False Negative Rate** | **24.29%** | $\text{FNR} = 1 - \text{Sensitivity}$: Missed disfluency events |
|
| 94 |
+
| | **False Positive Rate** | **32.72%** | $\text{FPR} = 1 - \text{Specificity}$: Fluent speech flagged as disfluent |
|
| 95 |
+
| **Pronunciation (ASR)**| **Word-Level Accuracy** | **95.80%** | $(1.0 - \text{WER}) \times 100\%$ on standard clean read speech |
|
| 96 |
+
| | **Word Error Rate (WER)** | **4.20%** | Levenshtein word distance over reference text |
|
| 97 |
+
| | **Character Error Rate**| **1.85%** | Sub-word character distance |
|
| 98 |
+
| **Acoustic Correlates**| **Pitch ($F_0$) Corr.** | **$r = 0.992$** | Pearson correlation against Praat ground-truth pitch contours |
|
| 99 |
+
| | **Local Jitter Error** | **$\pm 0.0018$** | Cycle-to-cycle pitch perturbation variance |
|
|
|
|
| 100 |
|
| 101 |
---
|
| 102 |
|
| 103 |
+
## 3. Methodological Limitations & Research Disclosures
|
| 104 |
|
| 105 |
+
### 3.1 Synthetic DSP Lattice Risk Analysis
|
| 106 |
+
To mitigate the extreme class imbalance in clinical datasets (such as UCLASS, where severe blocks are rare), 4,000 synthetic disfluency lattices were generated via digital signal processing (`ml/data/lattice_synth.py`).
|
| 107 |
|
| 108 |
+
**Acknowledgeable Research Risk**:
|
| 109 |
+
- While Hann cross-fading and zero-crossing snapping eliminate audible clicks, a neural network can potentially learn subtle time-frequency boundary artifacts of WSOLA time-stretching or splice transitions rather than genuine laryngeal pathophysiology.
|
| 110 |
+
- Cross-domain evaluation on purely authentic clinical monologues yields a lower Macro-F1 ($57.35\%$) compared to hybrid lattices ($71.47\%$).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
+
### 3.2 Acoustic Correlates vs. Biomechanical Phonation
|
| 113 |
+
The system uses Parselmouth (Praat) to extract **acoustic correlates** of voice quality:
|
| 114 |
+
- Fundamental Frequency ($F_0$)
|
| 115 |
+
- Cycle-to-cycle Period Jitter (local)
|
| 116 |
+
- Amplitude Perturbation Shimmer (local)
|
| 117 |
+
- Harmonics-to-Noise Ratio (HNR in dB)
|
| 118 |
|
| 119 |
+
These measurements reflect acoustic waveforms captured by a microphone; they do **not** constitute direct physical vocal fold contact area measurements (such as Electroglottography / EGG) or endoscopic imaging.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
|
| 121 |
+
### 3.3 Phonological Flaw Heuristics
|
| 122 |
+
The Rhotacism ('r') and Sigmatism ('s') detectors currently operate via:
|
| 123 |
+
1. CTC phonetic sub-word decoding.
|
| 124 |
+
2. Levenshtein substitution mapping against expected target text (e.g. flagging *red* $\to$ *wed*, *sun* $\to$ *thun*).
|
| 125 |
+
3. Spectral energy centroid heuristics.
|
| 126 |
|
| 127 |
+
These provide assistive practice feedback but are not equivalent to formal Speech-Language Pathologist (SLP) phonetic transcriptions.
|
|
|
|
| 128 |
|
| 129 |
+
### 3.4 Heuristic Weighting in Fluency Index
|
| 130 |
+
The composite Fluency Index ($0\text{ to }100$) uses empirically chosen heuristic weights:
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
+
$$\text{Fluency} = 100 \cdot \max\left(0, 1.0 - \left(0.40 \cdot \mathcal{L}_{\text{stutter}} + 0.45 \cdot \mathcal{L}_{\text{pron}} + 0.15 \cdot \mathcal{L}_{\text{acoustics}}\right)\right)$$
|
| 133 |
|
| 134 |
+
These weights are designed for user practice feedback and have not been fitted via longitudinal clinical trials.
|
|
|
|
|
|
|
|
|
|
| 135 |
|
| 136 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
+
## 4. Ablation & Component Contribution Framework
|
| 139 |
|
| 140 |
+
| Model Configuration | Stutter Precision | Stutter Recall | Macro-F1 | Notes / Contribution |
|
| 141 |
+
| :--- | :--- | :--- | :--- | :--- |
|
| 142 |
+
| **Wav2Vec2 Base (Zero-Shot)** | 38.20% | 46.10% | 41.80% | High false-positive rate on natural pauses |
|
| 143 |
+
| **Wav2Vec2 + UCLASS Only** | 44.21% | 41.50% | 57.35% | Severe class imbalance limits block detection |
|
| 144 |
+
| **Wav2Vec2 + LoRA + Focal Loss** | 72.10% | 68.40% | 66.80% | Focal loss focuses on boundary transitions |
|
| 145 |
+
| **Full Hybrid (Real + Synthetic)**| **85.05%** | **75.71%** | **71.47%** | Balanced lattice improves prolongation & block detection |
|
| 146 |
|
| 147 |
---
|
| 148 |
|
| 149 |
+
## 5. Repository File Map
|
|
|
|
|
|
|
| 150 |
|
| 151 |
```
|
| 152 |
speech-model/
|
| 153 |
|-- ml/
|
| 154 |
| |-- model/
|
| 155 |
| | |-- engine.py # Singleton high-speed inference pipeline
|
| 156 |
+
| | |-- infer.py # LoRA model loader and probability extraction
|
| 157 |
| | |-- pron_eval.py # CTC ASR decoding, word alignment, 'r'/'s' flaw rules
|
| 158 |
+
| | |-- fusion.py # Multi-modal fusion, uncertainty, self-calibration
|
| 159 |
| | |-- stutter_trainer.py # Wav2Vec2 + LoRA training script with Focal Loss
|
| 160 |
| |-- data/
|
| 161 |
| | |-- lattice_synth.py # DSP disfluency lattice synthesizer
|
|
|
|
| 164 |
| | |-- precache_models.py # Docker build pre-cache script
|
| 165 |
| |-- cli.py # Unified developer command-line interface
|
| 166 |
|-- webapp.py # Streamlit clinical diagnostic user interface
|
| 167 |
+
|-- app.py # Gradio interface for Hugging Face Spaces (ZeroGPU)
|
| 168 |
|-- docs/
|
| 169 |
| |-- DATASETS_AND_MODEL_GUIDE.md # Plain-English guide to datasets and model design
|
| 170 |
| |-- DEPLOY_RENDER.md # Step-by-step Render deployment documentation
|
|
|
|
| 175 |
|-- requirements.txt # Python package dependencies
|
| 176 |
|-- Dockerfile # Production container configuration for Render
|
| 177 |
|-- render.yaml # Render 1-Click deployment blueprint
|
| 178 |
+
|-- README.md # Transparent research documentation
|
| 179 |
```
|
| 180 |
|
| 181 |
---
|
| 182 |
|
| 183 |
+
## 6. Step-by-Step Local Setup & Execution Guide (PowerShell / Windows)
|
| 184 |
|
| 185 |
The following PowerShell commands allow anyone to clone, set up, and test the entire framework locally on Windows:
|
| 186 |
|
|
|
|
| 245 |
)
|
| 246 |
print('Inference Latency: ', res['latency_ms'], 'ms')
|
| 247 |
print('Clinical Assessment: ', res['decision']['buckets']['overall'].upper())
|
| 248 |
+
print('Confidence Level: ', res['decision'].get('confidence', 'N/A'))
|
| 249 |
print('Fluency Index: ', res['decision']['fluency_100'], '/ 100')
|
| 250 |
print('Decoded Text: ', res['pronunciation']['asr_hypothesis'])
|
| 251 |
"@
|
|
|
|
| 264 |
|
| 265 |
### Alternative: Bash / Linux / macOS Commands
|
| 266 |
|
|
|
|
|
|
|
| 267 |
```bash
|
| 268 |
# Setup
|
| 269 |
git clone https://github.com/notUbaid/speech-model.git
|