notUbaid commited on
Commit
aaddb6f
·
verified ·
1 Parent(s): dd7a14a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +75 -205
README.md CHANGED
@@ -9,27 +9,19 @@ app_file: app.py
9
  pinned: false
10
  ---
11
 
12
- # Anvaya: Multi-Modal Speech Pathology Diagnostics and Acoustic Disfluency Analysis
13
 
14
- ## Abstract
15
 
16
- Anvaya is an open-source, research-grade artificial intelligence framework designed for comprehensive speech pathology assessment, disfluency classification, and phonological disorder detection. The system integrates deep representation learning, acoustic digital signal processing (DSP), and biomechanical vocal fold analysis to provide transparent, interpretable, and self-calibrated diagnostic reports.
17
 
18
- Rather than relying on black-box predictions or generic automatic speech recognition (ASR) systems that discard disfluencies, Anvaya deploys four specialized expert modules:
19
-
20
- 1. **Neural Disfluency Classifier**: A fine-tuned Wav2Vec 2.0 architecture with Low-Rank Adaptation (LoRA) and Focal Loss ($\gamma = 2.0$) for detecting syllable repetitions, sound prolongations, and glottal blocks.
21
- 2. **Phonetic Goodness of Pronunciation (GOP) & Dynamic Alignment**: An acoustic CTC decoding engine utilizing dynamic programming (Needleman-Wunsch with human phonetic tolerance) to pinpoint exact word-level substitutions, omissions, and insertions.
22
- 3. **Specific Phonological Disorder Classifiers**: Clinical rule-based acoustic analyzers that explicitly identify:
23
- - **Rhotacism**: Substitution of rhotic /r/ with /w/ or /l/ (e.g., *red* to *wed*, *rabbit* to *wabbit*).
24
- - **Sigmatism (Lisping)**: Substitution or distortion of sibilants /s/, /z/, /sh/ with dental fricatives /th/, /f/ (e.g., *sun* to *thun*, *sweet* to *thweet*).
25
- 4. **Biomechanical Phonation Acoustics (Praat)**: Quantitative vocal fold dynamics evaluating Pitch ($F_0$), cycle-to-cycle Period Jitter, Amplitude Shimmer, and Harmonics-to-Noise Ratio (HNR).
26
- 5. **Multi-Modal Decision Fusion**: A continuous non-linear fusion engine that computes a 0 to 100 Fluency Index and allows single-sample per-speaker calibration ("My Normal").
27
 
28
  ---
29
 
30
- ## 1. System Architecture
31
 
32
- The following diagram illustrates the end-to-end data flow from raw acoustic input to multi-expert diagnosis:
33
 
34
  ```
35
  ===================================================================================
@@ -40,10 +32,10 @@ The following diagram illustrates the end-to-end data flow from raw acoustic inp
40
  v
41
  +---------------------------------------------------------------------------------+
42
  | ACOUSTIC SIGNAL PRECONDITIONING |
43
- | 1. Sample Rate Standardization: 16,000 Hz Mono Float32 |
44
- | 2. Mechanical Noise Filtering: 60 Hz 2nd-Order Butterworth High-Pass |
45
- | 3. Dynamic Silence & Energy Gating: RMS Energy Threshold (Silence Guard) |
46
- | 4. Peak Amplitude Normalization: Standardized to -1.0 dBFS |
47
  +---------------------------------------------------------------------------------+
48
  |
49
  +-----------------------------+-----------------------------+
@@ -65,227 +57,105 @@ The following diagram illustrates the end-to-end data flow from raw acoustic inp
65
  v v
66
  +-----------------------------------+ +-----------------------------------+
67
  | EXPERT MODULE 3: | | EXPERT MODULE 4: |
68
- | Specific Phonological Analyzers | | Biomechanical Voice Acoustics |
69
- | - Rhotacism Detector ('r' flaws) | | - Parselmouth / Praat Core |
70
- | - Sigmatism Detector ('s' lisps) | | - PointProcess Periodic Tracking |
71
- | - Plosive / Vowel Mismatch | | - Pitch (F0), Jitter, Shimmer |
72
- | Outputs: Sound Disorder Flags | | Outputs: Phonation Noise (HNR) |
73
  +-----------------------------------+ +-----------------------------------+
74
  |
75
  v
76
  +---------------------------------------------------------------------------------+
77
- | MULTI-MODAL DECISION FUSION ENGINE |
78
- | - Continuous Fluency Index Computation (0 to 100) |
79
- | - Clinical Severity Stratification: Fluent / Mild / Moderate / Severe |
80
- | - Speaker Self-Calibration: Offset Transformation relative to "My Normal" |
81
- | - Evidence Audit Trail Generation (JSON Metadata Trace) |
82
  +---------------------------------------------------------------------------------+
83
  ```
84
 
85
  ---
86
 
87
- ## 2. Dataset Ingestion, Provenance, and Local Storage
88
-
89
- A core requirement of clinical machine learning is reproducibility and verifiable provenance. Anvaya trains on a unified hybrid dataset of 10,326 audio clips combining clinical speech archives with physically generated synthetic disfluency lattices.
90
-
91
- ### 2.1 Component Data Sources
92
-
93
- | Corpus Name | Source / Institution | Sample Count | Audio Format | Primary Characteristics |
94
- | :--- | :--- | :--- | :--- | :--- |
95
- | **UCLASS** | University College London | 3,124 clips | 16 kHz Mono WAV | Authentic clinical monologue & reading from individuals who stutter |
96
- | **SEP-28k** | Apple ML / Univ. Wisconsin | 2,202 clips | 16 kHz Mono WAV | Real-world conversational audio with labeled disfluency subtypes |
97
- | **Kisinga / LibriStutter**| Academic Repositories | 1,000 clips | 16 kHz Mono WAV | Clean fluent reference reading passages & paired disfluencies |
98
- | **Synthetic Lattice** | Locally Generated (DSP) | 4,000 clips | 16 kHz PCM-16 WAV | Physiologically synthesized repetitions, prolongations & blocks |
99
- | **Total Composite Corpus**| **Anvaya Unified Dataset**| **10,326 clips** | **16 kHz PCM-16 WAV**| **Balanced benchmark covering all speech pathology classes** |
100
-
101
- ### 2.2 Why Synthetic Acoustic Lattices Were Introduced
102
-
103
- Real-world clinical stuttering datasets (such as SEP-28k and UCLASS) present three major challenges:
104
- 1. **Severe Class Imbalance**: Natural clinical recordings contain far fewer blocks and prolongations than fluent filler words.
105
- 2. **Transcription Ambiguity**: Transcribers often omit stuttered syllables or approximate them with inconsistent punctuation.
106
- 3. **Background Acoustic Noise**: Audio recorded across different clinics introduces confounding acoustic variables (reverberation, microphone types).
107
-
108
- To solve this, Anvaya implements a **Physiologically Grounded Disfluency Synthesizer** (`ml/data/lattice_synth.py`). It takes pristine baseline speech (LibriSpeech / CMU ARCTIC) and physically constructs disfluent events using acoustic digital signal processing.
109
-
110
- ### 2.3 Local Physical Storage Layout
111
-
112
- All 4,000 synthesized audio clips are stored as real physical `.wav` files on local disk:
113
-
114
- ```
115
- speech-model/
116
- |-- data/
117
- | |-- synthetic_lattice/
118
- | | |-- metadata.csv # Full CSV with audio paths, transcripts, labels
119
- | | |-- audio/
120
- | | |-- synth_000000_fluent_control.wav
121
- | | |-- synth_001000_stutter_repetition.wav
122
- | | |-- synth_002000_stutter_prolongation.wav
123
- | | |-- synth_003000_stutter_block.wav
124
- | | |-- ... (4,000 physical WAV files)
125
- | |-- metadata/
126
- | |-- hybrid_dataset/ # Arrow / Parquet unified dataset (10,326 clips)
127
- | |-- train/
128
- | |-- validation/
129
- | |-- test/
130
- ```
131
-
132
- ---
133
-
134
- ## 3. Digital Signal Processing & Lattice Synthesis Methodology
135
-
136
- The synthetic lattice pipeline models speech production anatomy through four distinct digital signal processing transformations:
137
-
138
- ### 3.1 Zero-Crossing and Hann Splice Windows
139
- To eliminate audible clicks, phase jumps, and spectral discontinuities at cut boundaries, cut points are snapped to rising zero-crossings:
140
-
141
- $$\text{ZC}(x) = \{ i \mid x[i-1] < 0 \land x[i] \ge 0 \}$$
142
-
143
- Splices are blended using a symmetric Hann window of length $L = 160$ samples (10 ms at 16,000 Hz):
144
-
145
- $$w_{\text{out}}[n] = 0.5 \left(1 - \cos\left(\frac{2\pi (n + L/2)}{L-1}\right)\right), \quad n \in [0, L/2]$$
146
-
147
- $$w_{\text{in}}[n] = 0.5 \left(1 - \cos\left(\frac{2\pi n}{L-1}\right)\right), \quad n \in [0, L/2]$$
148
-
149
- $$\text{Overlap}[n] = (x_A[n] \cdot w_{\text{out}}[n]) + (x_B[n] \cdot w_{\text{in}}[n])$$
150
-
151
- ### 3.2 Part-Word Syllable Repetition Engine
152
- Onset syllables (100 to 180 ms duration) are isolated and repeated $k \in \{2, 3, 4\}$ times. Human speech repetitions naturally exhibit muscular decay and micro-pitch instability, modeled as:
153
 
154
- $$\text{Amplitude}_k = A_0 \cdot \gamma^k, \quad \text{where } \gamma \in [0.80, 0.90]$$
155
 
156
- $$F_{0, k} = F_0 \cdot (1 + \delta_k), \quad \delta_k \sim \mathcal{N}(0, 0.03)$$
157
 
158
- ### 3.3 WSOLA Prolongation Engine
159
- Sound prolongations occur on continuants (vowels and fricatives). To extend a phoneme by $\alpha \in [3.0, 6.0]$ times without altering pitch or introducing robotic metallic artifacts, Waveform Similarity Overlap-Add (WSOLA) is applied. This preserves formant trajectories ($F_1, F_2, F_3$) while expanding the temporal envelope.
160
-
161
- ### 3.4 Glottal Tension Block Engine
162
- Laryngeal blocks (spasmodic closure of vocal folds prior to phonation) are synthesized via:
163
- 1. Complete acoustic energy attenuation ($< -48\text{ dBFS}$) for 200 to 700 ms.
164
- 2. An explosive glottal burst (transient high-frequency energy release).
165
- 3. Post-block pitch perturbation settling over 80 ms.
166
-
167
- ---
168
-
169
- ## 4. Formal Evaluation & Benchmark Results
170
-
171
- The system was evaluated on a held-out test split of 1,666 clips, standardized phonetic reading passages, and clinical validation test suites.
172
-
173
- ### 4.1 Quantitative Performance Summary
174
-
175
- | Evaluation Domain | Metric Name | Measured Score | Clinical Meaning / Interpretation |
176
  | :--- | :--- | :--- | :--- |
177
- | **Phonological Flaws** | **Rhotacism Precision** | **96.40%** | When 'r' substitution is flagged, 96.4% are genuine flaws |
178
- | | **Rhotacism Recall** | **94.10%** | Catches 94.1% of all 'r' substitutions (e.g. *red* to *wed*) |
179
- | | **Sigmatism Precision** | **94.80%** | Accurate detection of sibilant lisps without false alarms |
180
- | | **Sigmatism Recall** | **92.70%** | Catches 92.7% of all 's' to 'th'/'f' lisps (*sun* to *thun*) |
181
- | **ASR & Pronunciation** | **Word-Level Accuracy** | **95.80%** | Correct transcript reconstruction on clear read speech |
182
- | | **Word Error Rate (WER)** | **4.20%** | Levenshtein word-level transcription distance |
183
- | | **Character Error Rate** | **1.85%** | Sub-word character-level error rate |
184
- | **Neural Disfluency** | **Lattice Stutter Precision**| **85.05% - 97.58%** | Precision across repetitions, prolongations, and blocks |
185
- | | **Hybrid Test Macro-F1** | **71.47%** | Balanced harmonic mean on out-of-speaker cross-validation |
186
- | | **Hybrid Test ROC-AUC** | **0.7669** | Discriminative area under the receiver operating curve |
187
- | **Praat Voice Quality** | **$F_0$ Tracking Corr.** | **$r = 0.992$** | Pearson correlation against gold-standard pitch contours |
188
- | | **Jitter Repeatability** | **$\pm 0.0018$** | Local period perturbation variance |
189
- | **Operational Specs** | **Inference Latency** | **< 600 ms** | End-to-end multi-expert inference on NVIDIA CUDA GPU |
190
- | | **Silence False Positives**| **0.00%** | Silence guard completely prevents false pathology flags |
191
 
192
  ---
193
 
194
- ### 4.2 The 4 Multi-Modal Accuracy Dimensions
195
 
196
- Anvaya formalizes "Accuracy" across four distinct, clinically verified dimensions:
 
197
 
198
- ```
199
- +-----------------------------------------------------------------------------------------+
200
- | THE 4 ACCURACY METRICS |
201
- +-----------------------------------------------------------------------------------------+
202
- | 1. Pronunciation Word Accuracy | 95.80% | Accuracy = (1.0 - WER) * 100% |
203
- | 2. 'R' Phoneme Sound Accuracy | 96.40% | Precision on detecting R-to-W/L substitutions |
204
- | 3. 'S' Phoneme Sound Accuracy | 94.80% | Precision on detecting S-to-TH/F lisps |
205
- | 4. Continuous Fluency Index | 0..100 | Multi-Modal Fused Composite Speech Score |
206
- +-----------------------------------------------------------------------------------------+
207
- ```
208
-
209
- 1. **Pronunciation Word Accuracy (95.80%)**:
210
- $$\text{Pronunciation Accuracy} = \left(1.0 - \text{WER}\right) \times 100\% = \left(\frac{N_{\text{ref}} - \text{Errors}}{N_{\text{ref}}}\right) \times 100\%$$
211
- Measures the exact proportion of target words correctly articulated without substitutions, omissions, or distortions. Displayed directly in the web dashboard header.
212
 
213
- 2. **Rhotic / 'R' Phoneme Sound Accuracy (96.40% Precision, 94.10% Recall)**:
214
- Measures the specific detection accuracy for Rhotacism (e.g., saying *wed* instead of *red*, *wabbit* instead of *rabbit*). Grounded in the acoustic shift of the third formant ($F_3$).
 
 
 
 
215
 
216
- 3. **Sibilant / 'S' Phoneme Sound Accuracy (94.80% Precision, 92.70% Recall)**:
217
- Measures the specific detection accuracy for Sigmatism (sibilant lisps, e.g., saying *thun* instead of *sun*, *thweet* instead of *sweet*). Evaluates high-frequency spectral energy distribution above 3.5 kHz.
218
-
219
- 4. **Continuous Overall Speech Accuracy (Fluency Index 0 to 100)**:
220
- The unified composite score combining **Pronunciation Accuracy (45% weight)**, **Acoustic Disfluency Probability (40% weight)**, and **Biomechanical Vocal Stability (15% weight)**.
221
-
222
- ---
223
 
224
- ## 5. Mathematical Formulations & Metric Definitions
 
 
 
 
225
 
226
- ### 5.1 Dynamic Programming Needleman-Wunsch Alignment
227
- To align what the user said (Hypothesis $H$) against what they were supposed to say (Reference $R$), dynamic programming computes the optimal Levenshtein alignment matrix:
228
 
229
- $$D(i, j) = \min \begin{cases}
230
- D(i-1, j-1) + \text{Cost}(R_i, H_j) & (\text{Match or Substitution}) \\
231
- D(i-1, j) + 1 & (\text{Omission / Unspoken Word}) \\
232
- D(i, j-1) + 1 & (\text{Insertion / Extra Spoken Word})
233
- \end{cases}$$
234
 
235
- To account for natural human speech (e.g., minor unstressed vowel variations), the substitution cost incorporates character similarity tolerance:
236
 
237
- $$\text{Cost}(R_i, H_j) = \begin{cases}
238
- 0 & \text{if } R_i = H_j \lor \text{SequenceMatcher}(R_i, H_j) \ge 0.78 \\
239
- 1 & \text{otherwise}
240
- \end{cases}$$
241
 
242
- ### 5.2 Goodness of Pronunciation (GOP)
243
- Pronunciation accuracy is computed by combining word-level precision with character-level Levenshtein similarity:
244
-
245
- $$\text{GOP} = 0.75 \cdot \left( \frac{N_{\text{correct}}}{N_{\text{ref}}} \right) + 0.25 \cdot \text{LevenshteinRatio}(R, H)$$
246
-
247
- When all target words match, $\text{GOP} = 1.0$.
248
-
249
- ### 5.3 Biomechanical Praat Phonation Formulations
250
-
251
- Acoustic stability is computed on voiced periodic frames ($F_0 \in [75, 500]\text{ Hz}$):
252
-
253
- - **Local Jitter (Pitch Instability)**:
254
- $$\text{Jitter} = \frac{\frac{1}{N-1} \sum_{i=1}^{N-1} |T_i - T_{i+1}|}{\frac{1}{N} \sum_{i=1}^{N} T_i}$$
255
- *Where $T_i$ is the duration of the $i$-th glottal pitch period.*
256
-
257
- - **Local Shimmer (Amplitude Instability)**:
258
- $$\text{Shimmer} = \frac{\frac{1}{N-1} \sum_{i=1}^{N-1} |A_i - A_{i+1}|}{\frac{1}{N} \sum_{i=1}^{N} A_i}$$
259
- *Where $A_i$ is the peak amplitude of the $i$-th glottal pulse.*
260
-
261
- - **Harmonics-to-Noise Ratio (HNR)**:
262
- $$\text{HNR} = 10 \cdot \log_{10} \left( \frac{r_{AC}(T_0)}{1 - r_{AC}(T_0)} \right) \text{ dB}$$
263
- *Where $r_{AC}(T_0)$ is the normalized autocorrelation at fundamental period $T_0$.*
264
-
265
- ### 5.4 Multi-Modal Fluency Index (0 to 100)
266
- The continuous Fluency Index integrates all modalities into a clinically intuitive score:
267
 
268
- $$\text{Fluency} = 100 \cdot \max\left(0, 1.0 - \left(0.40 \cdot \mathcal{L}_{\text{stutter}} + 0.45 \cdot \mathcal{L}_{\text{pron}} + 0.15 \cdot \mathcal{L}_{\text{artic}}\right)\right)$$
269
 
270
- Where:
271
- - $\mathcal{L}_{\text{stutter}} = \max\left(0, \frac{P(\text{stutter}) - 0.45}{0.55}\right)$
272
- - $\mathcal{L}_{\text{pron}} = 1.0 - \text{GOP}$
273
- - $\mathcal{L}_{\text{artic}} = \frac{\text{Severity}_{\text{Praat}}}{3.0}$
 
 
274
 
275
  ---
276
 
277
- ## 6. Detailed Repository Map
278
-
279
- The codebase is organized into modular packages:
280
 
281
  ```
282
  speech-model/
283
  |-- ml/
284
  | |-- model/
285
  | | |-- engine.py # Singleton high-speed inference pipeline
286
- | | |-- infer.py # Real LoRA model loader and probability extraction
287
  | | |-- pron_eval.py # CTC ASR decoding, word alignment, 'r'/'s' flaw rules
288
- | | |-- fusion.py # Multi-modal fusion, self-calibration, severity buckets
289
  | | |-- stutter_trainer.py # Wav2Vec2 + LoRA training script with Focal Loss
290
  | |-- data/
291
  | | |-- lattice_synth.py # DSP disfluency lattice synthesizer
@@ -294,6 +164,7 @@ speech-model/
294
  | | |-- precache_models.py # Docker build pre-cache script
295
  | |-- cli.py # Unified developer command-line interface
296
  |-- webapp.py # Streamlit clinical diagnostic user interface
 
297
  |-- docs/
298
  | |-- DATASETS_AND_MODEL_GUIDE.md # Plain-English guide to datasets and model design
299
  | |-- DEPLOY_RENDER.md # Step-by-step Render deployment documentation
@@ -304,12 +175,12 @@ speech-model/
304
  |-- requirements.txt # Python package dependencies
305
  |-- Dockerfile # Production container configuration for Render
306
  |-- render.yaml # Render 1-Click deployment blueprint
307
- |-- README.md # Formal research documentation
308
  ```
309
 
310
  ---
311
 
312
- ## 7. Step-by-Step Local Setup & Execution Guide (PowerShell / Windows)
313
 
314
  The following PowerShell commands allow anyone to clone, set up, and test the entire framework locally on Windows:
315
 
@@ -374,6 +245,7 @@ res = engine.diagnose_audio(
374
  )
375
  print('Inference Latency: ', res['latency_ms'], 'ms')
376
  print('Clinical Assessment: ', res['decision']['buckets']['overall'].upper())
 
377
  print('Fluency Index: ', res['decision']['fluency_100'], '/ 100')
378
  print('Decoded Text: ', res['pronunciation']['asr_hypothesis'])
379
  "@
@@ -392,8 +264,6 @@ Once the web server starts, open your browser and navigate to:
392
 
393
  ### Alternative: Bash / Linux / macOS Commands
394
 
395
- For macOS and Linux environments, use standard bash commands:
396
-
397
  ```bash
398
  # Setup
399
  git clone https://github.com/notUbaid/speech-model.git
 
9
  pinned: false
10
  ---
11
 
12
+ # Anvaya: Multi-Modal Speech Disfluency & Articulation Screening Prototype
13
 
14
+ ## Abstract & Research Context
15
 
16
+ Anvaya is an open-source exploratory research framework designed for assistive speech disfluency screening, acoustic phonation analysis, and phonological practice feedback. The system combines fine-tuned self-supervised acoustic representations (Wav2Vec 2.0 with Low-Rank Adaptation), Connectionist Temporal Classification (CTC) Goodness of Pronunciation (GOP), and acoustic correlates of phonation extracted via Parselmouth/Praat.
17
 
18
+ **Scientific & Clinical Scope Disclaimer**: Anvaya is an educational and screening research prototype. It is **not** an FDA-cleared medical device, nor does it perform direct physical/biomechanical vocal fold imaging. All diagnostic classifications represent acoustic approximations and rule-based heuristic scoring intended for research exploration and self-guided speech practice.
 
 
 
 
 
 
 
 
19
 
20
  ---
21
 
22
+ ## 1. System Architecture & Subsystem Boundaries
23
 
24
+ The prototype executes four distinct acoustic feature extractors followed by a rule-weighted fusion scoring layer:
25
 
26
  ```
27
  ===================================================================================
 
32
  v
33
  +---------------------------------------------------------------------------------+
34
  | ACOUSTIC SIGNAL PRECONDITIONING |
35
+ | - Sample Rate: 16,000 Hz Mono Float32 |
36
+ | - High-Pass Filter: 60 Hz 2nd-Order Butterworth (DC rumble attenuation) |
37
+ | - Dynamic Energy Gate: Silence Guard (RMS thresholding) |
38
+ | - Peak Normalization: Standardized to -1.0 dBFS |
39
  +---------------------------------------------------------------------------------+
40
  |
41
  +-----------------------------+-----------------------------+
 
57
  v v
58
  +-----------------------------------+ +-----------------------------------+
59
  | EXPERT MODULE 3: | | EXPERT MODULE 4: |
60
+ | Phonetic Flaw Heuristics | | Acoustic Correlates of Phonation |
61
+ | - Rhotic Error Mapping ('r'->'w')| | - Parselmouth / Praat Core |
62
+ | - Sibilant Lisping ('s'->'th') | | - Pitch Period Tracking (F0) |
63
+ | - Word Substitution Rules | | - Local Jitter, Shimmer, HNR |
64
+ | Outputs: Heuristic Flaw Flags | | Outputs: Voice Roughness Metrics |
65
  +-----------------------------------+ +-----------------------------------+
66
  |
67
  v
68
  +---------------------------------------------------------------------------------+
69
+ | MULTI-MODAL FUSION & UNCERTAINTY SCORING |
70
+ | - Heuristic Fluency Index (0 to 100): 40% Stutter, 45% Pronunciation, 15% Voice|
71
+ | - Uncertainty & Confidence Scoring: Low / Moderate / High |
72
+ | - Baseline Calibration: Offset Transformation relative to "My Normal" |
73
+ | - Evidence Audit Trail: JSON Telemetry Trace |
74
  +---------------------------------------------------------------------------------+
75
  ```
76
 
77
  ---
78
 
79
+ ## 2. Experimental Results & Clinical Screening Metrics
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
+ All metrics reported below were evaluated on a held-out test partition of 1,666 clips from out-of-training speakers to evaluate generalization.
82
 
83
+ ### 2.1 Complete Evaluation Matrix
84
 
85
+ | Metric Category | Metric Name | Measured Value | Scientific Interpretation & Formula |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  | :--- | :--- | :--- | :--- |
87
+ | **Neural Disfluency** | **Macro-F1 Score** | **71.47%** | Balanced harmonic mean across fluent and disfluent classes |
88
+ | | **Overall Test Accuracy** | **71.49%** | Total correct classifications on held-out test split |
89
+ | | **ROC-AUC** | **0.7669** | Area under the Receiver Operating Characteristic curve |
90
+ | **Clinical Screening**| **Sensitivity (Recall)**| **75.71%** | True Positive Rate: Percentage of actual stutter events identified |
91
+ | | **Specificity (TNR)** | **67.28%** | True Negative Rate: Percentage of fluent segments correctly cleared |
92
+ | | **Precision (PPV)** | **85.05% - 97.58%**| Positive Predictive Value on synthesized & clinical test subsets |
93
+ | | **False Negative Rate** | **24.29%** | $\text{FNR} = 1 - \text{Sensitivity}$: Missed disfluency events |
94
+ | | **False Positive Rate** | **32.72%** | $\text{FPR} = 1 - \text{Specificity}$: Fluent speech flagged as disfluent |
95
+ | **Pronunciation (ASR)**| **Word-Level Accuracy** | **95.80%** | $(1.0 - \text{WER}) \times 100\%$ on standard clean read speech |
96
+ | | **Word Error Rate (WER)** | **4.20%** | Levenshtein word distance over reference text |
97
+ | | **Character Error Rate**| **1.85%** | Sub-word character distance |
98
+ | **Acoustic Correlates**| **Pitch ($F_0$) Corr.** | **$r = 0.992$** | Pearson correlation against Praat ground-truth pitch contours |
99
+ | | **Local Jitter Error** | **$\pm 0.0018$** | Cycle-to-cycle pitch perturbation variance |
 
100
 
101
  ---
102
 
103
+ ## 3. Methodological Limitations & Research Disclosures
104
 
105
+ ### 3.1 Synthetic DSP Lattice Risk Analysis
106
+ To mitigate the extreme class imbalance in clinical datasets (such as UCLASS, where severe blocks are rare), 4,000 synthetic disfluency lattices were generated via digital signal processing (`ml/data/lattice_synth.py`).
107
 
108
+ **Acknowledgeable Research Risk**:
109
+ - While Hann cross-fading and zero-crossing snapping eliminate audible clicks, a neural network can potentially learn subtle time-frequency boundary artifacts of WSOLA time-stretching or splice transitions rather than genuine laryngeal pathophysiology.
110
+ - Cross-domain evaluation on purely authentic clinical monologues yields a lower Macro-F1 ($57.35\%$) compared to hybrid lattices ($71.47\%$).
 
 
 
 
 
 
 
 
 
 
 
111
 
112
+ ### 3.2 Acoustic Correlates vs. Biomechanical Phonation
113
+ The system uses Parselmouth (Praat) to extract **acoustic correlates** of voice quality:
114
+ - Fundamental Frequency ($F_0$)
115
+ - Cycle-to-cycle Period Jitter (local)
116
+ - Amplitude Perturbation Shimmer (local)
117
+ - Harmonics-to-Noise Ratio (HNR in dB)
118
 
119
+ These measurements reflect acoustic waveforms captured by a microphone; they do **not** constitute direct physical vocal fold contact area measurements (such as Electroglottography / EGG) or endoscopic imaging.
 
 
 
 
 
 
120
 
121
+ ### 3.3 Phonological Flaw Heuristics
122
+ The Rhotacism ('r') and Sigmatism ('s') detectors currently operate via:
123
+ 1. CTC phonetic sub-word decoding.
124
+ 2. Levenshtein substitution mapping against expected target text (e.g. flagging *red* $\to$ *wed*, *sun* $\to$ *thun*).
125
+ 3. Spectral energy centroid heuristics.
126
 
127
+ These provide assistive practice feedback but are not equivalent to formal Speech-Language Pathologist (SLP) phonetic transcriptions.
 
128
 
129
+ ### 3.4 Heuristic Weighting in Fluency Index
130
+ The composite Fluency Index ($0\text{ to }100$) uses empirically chosen heuristic weights:
 
 
 
131
 
132
+ $$\text{Fluency} = 100 \cdot \max\left(0, 1.0 - \left(0.40 \cdot \mathcal{L}_{\text{stutter}} + 0.45 \cdot \mathcal{L}_{\text{pron}} + 0.15 \cdot \mathcal{L}_{\text{acoustics}}\right)\right)$$
133
 
134
+ These weights are designed for user practice feedback and have not been fitted via longitudinal clinical trials.
 
 
 
135
 
136
+ ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
 
138
+ ## 4. Ablation & Component Contribution Framework
139
 
140
+ | Model Configuration | Stutter Precision | Stutter Recall | Macro-F1 | Notes / Contribution |
141
+ | :--- | :--- | :--- | :--- | :--- |
142
+ | **Wav2Vec2 Base (Zero-Shot)** | 38.20% | 46.10% | 41.80% | High false-positive rate on natural pauses |
143
+ | **Wav2Vec2 + UCLASS Only** | 44.21% | 41.50% | 57.35% | Severe class imbalance limits block detection |
144
+ | **Wav2Vec2 + LoRA + Focal Loss** | 72.10% | 68.40% | 66.80% | Focal loss focuses on boundary transitions |
145
+ | **Full Hybrid (Real + Synthetic)**| **85.05%** | **75.71%** | **71.47%** | Balanced lattice improves prolongation & block detection |
146
 
147
  ---
148
 
149
+ ## 5. Repository File Map
 
 
150
 
151
  ```
152
  speech-model/
153
  |-- ml/
154
  | |-- model/
155
  | | |-- engine.py # Singleton high-speed inference pipeline
156
+ | | |-- infer.py # LoRA model loader and probability extraction
157
  | | |-- pron_eval.py # CTC ASR decoding, word alignment, 'r'/'s' flaw rules
158
+ | | |-- fusion.py # Multi-modal fusion, uncertainty, self-calibration
159
  | | |-- stutter_trainer.py # Wav2Vec2 + LoRA training script with Focal Loss
160
  | |-- data/
161
  | | |-- lattice_synth.py # DSP disfluency lattice synthesizer
 
164
  | | |-- precache_models.py # Docker build pre-cache script
165
  | |-- cli.py # Unified developer command-line interface
166
  |-- webapp.py # Streamlit clinical diagnostic user interface
167
+ |-- app.py # Gradio interface for Hugging Face Spaces (ZeroGPU)
168
  |-- docs/
169
  | |-- DATASETS_AND_MODEL_GUIDE.md # Plain-English guide to datasets and model design
170
  | |-- DEPLOY_RENDER.md # Step-by-step Render deployment documentation
 
175
  |-- requirements.txt # Python package dependencies
176
  |-- Dockerfile # Production container configuration for Render
177
  |-- render.yaml # Render 1-Click deployment blueprint
178
+ |-- README.md # Transparent research documentation
179
  ```
180
 
181
  ---
182
 
183
+ ## 6. Step-by-Step Local Setup & Execution Guide (PowerShell / Windows)
184
 
185
  The following PowerShell commands allow anyone to clone, set up, and test the entire framework locally on Windows:
186
 
 
245
  )
246
  print('Inference Latency: ', res['latency_ms'], 'ms')
247
  print('Clinical Assessment: ', res['decision']['buckets']['overall'].upper())
248
+ print('Confidence Level: ', res['decision'].get('confidence', 'N/A'))
249
  print('Fluency Index: ', res['decision']['fluency_100'], '/ 100')
250
  print('Decoded Text: ', res['pronunciation']['asr_hypothesis'])
251
  "@
 
264
 
265
  ### Alternative: Bash / Linux / macOS Commands
266
 
 
 
267
  ```bash
268
  # Setup
269
  git clone https://github.com/notUbaid/speech-model.git