Hugging Face's logo

mazesmazes
/
tiny-audio

Automatic Speech Recognition
Transformers
Safetensors
English
asr_model
asr
speech-recognition
speech-to-text
audio
speech-llm
word-timestamps
speaker-diarization
qwen
granite-speech
lora
custom_code
Eval Results (legacy)
Model card Files
xet
 Community
mazesmazes commited on
Commit
36b481d
·
verified ·
1 Parent(s): 30774f4

Model card: current WER table, eval results (model-index), tags; list AutoModelForSpeechSeq2Seq first in auto_map

Browse files
Files changed (2) hide show
  1. README.md +172 -18
  2. config.json +1 -1
README.md CHANGED
@@ -18,11 +18,164 @@ pipeline_tag: automatic-speech-recognition
18
  tags:
19
  - asr
20
  - speech-recognition
 
21
  - audio
 
 
 
22
  - qwen
23
  - granite-speech
24
  - lora
25
  library_name: transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ---
27
 
28
  # Tiny Audio
@@ -30,8 +183,8 @@ library_name: transformers
30
  **A tiny, hackable, open-source speech LLM.**
31
 
32
  It transcribes English with punctuation, capitalization, word timestamps, and speaker labels: 1.8%
33
- WER on LibriSpeech test-clean and 8.4% averaged over 12 benchmarks. Built and trained with
34
- [Tiny Audio](https://github.com/alexkroman/tiny-audio).
35
 
36
  **[Try the live demo](https://huggingface.co/spaces/mazesmazes/tiny-audio)** ·
37
  **[Train your own](https://github.com/alexkroman/tiny-audio)** ·
@@ -58,25 +211,26 @@ the model writes punctuation, capitalization, and numbers itself.
58
 
59
  ## Benchmarks
60
 
61
- Word error rate (%, lower is better) on 1,000 samples per dataset, scored with the
62
  [Tiny Audio eval harness](https://github.com/alexkroman/tiny-audio/tree/main/scripts/eval)
63
  (`ta eval`) after text normalization.
64
 
65
- | Dataset | WER |
66
- | ---------------------- | -------: |
67
- | LibriSpeech test-clean | 1.80 |
68
- | SPGISpeech | 2.24 |
69
- | LibriSpeech test-other | 3.09 |
70
- | TED-LIUM | 3.74 |
71
- | LoquaciousSet † | 6.10 |
72
- | Common Voice | 6.62 |
73
- | VoxPopuli | 6.96 |
74
- | AMI (IHM) | 8.88 |
75
- | GigaSpeech | 9.07 |
76
- | Earnings22 † | 10.54 |
77
- | People's Speech | 17.69 |
78
- | AMI (SDM) | 23.59 |
79
- | **Mean (12 sets)** | **8.36** |
 
80
 
81
  † Held out: no data from this source was used in training.
82
 
 
18
  tags:
19
  - asr
20
  - speech-recognition
21
+ - speech-to-text
22
  - audio
23
+ - speech-llm
24
+ - word-timestamps
25
+ - speaker-diarization
26
  - qwen
27
  - granite-speech
28
  - lora
29
  library_name: transformers
30
+ metrics:
31
+ - wer
32
+ model-index:
33
+ - name: tiny-audio
34
+ results:
35
+ - task:
36
+ type: automatic-speech-recognition
37
+ name: Automatic Speech Recognition
38
+ dataset:
39
+ name: "LibriSpeech test-clean, subset of up to 1,000 samples"
40
+ type: openslr/librispeech_asr
41
+ config: clean
42
+ split: test
43
+ metrics:
44
+ - type: wer
45
+ value: 1.84
46
+ name: WER (normalized, subset of up to 1,000 samples)
47
+ - task:
48
+ type: automatic-speech-recognition
49
+ name: Automatic Speech Recognition
50
+ dataset:
51
+ name: "LibriSpeech test-other, subset of up to 1,000 samples"
52
+ type: openslr/librispeech_asr
53
+ config: other
54
+ split: test
55
+ metrics:
56
+ - type: wer
57
+ value: 6.38
58
+ name: WER (normalized, subset of up to 1,000 samples)
59
+ - task:
60
+ type: automatic-speech-recognition
61
+ name: Automatic Speech Recognition
62
+ dataset:
63
+ name: "SPGISpeech, subset of up to 1,000 samples"
64
+ type: kensho/spgispeech
65
+ config: test
66
+ split: test
67
+ metrics:
68
+ - type: wer
69
+ value: 2.29
70
+ name: WER (normalized, subset of up to 1,000 samples)
71
+ - task:
72
+ type: automatic-speech-recognition
73
+ name: Automatic Speech Recognition
74
+ dataset:
75
+ name: "TED-LIUM, subset of up to 1,000 samples"
76
+ type: sanchit-gandhi/tedlium-data
77
+ config: default
78
+ split: test
79
+ metrics:
80
+ - type: wer
81
+ value: 3.71
82
+ name: WER (normalized, subset of up to 1,000 samples)
83
+ - task:
84
+ type: automatic-speech-recognition
85
+ name: Automatic Speech Recognition
86
+ dataset:
87
+ name: "LoquaciousSet (held out), subset of up to 1,000 samples"
88
+ type: speechbrain/LoquaciousSet
89
+ config: small
90
+ split: test
91
+ metrics:
92
+ - type: wer
93
+ value: 6.2
94
+ name: WER (normalized, subset of up to 1,000 samples)
95
+ - task:
96
+ type: automatic-speech-recognition
97
+ name: Automatic Speech Recognition
98
+ dataset:
99
+ name: "VoxPopuli, subset of up to 1,000 samples"
100
+ type: facebook/voxpopuli
101
+ config: en
102
+ split: test
103
+ metrics:
104
+ - type: wer
105
+ value: 7.11
106
+ name: WER (normalized, subset of up to 1,000 samples)
107
+ - task:
108
+ type: automatic-speech-recognition
109
+ name: Automatic Speech Recognition
110
+ dataset:
111
+ name: "Common Voice 17, subset of up to 1,000 samples"
112
+ type: fixie-ai/common_voice_17_0
113
+ config: en
114
+ split: test
115
+ metrics:
116
+ - type: wer
117
+ value: 7.18
118
+ name: WER (normalized, subset of up to 1,000 samples)
119
+ - task:
120
+ type: automatic-speech-recognition
121
+ name: Automatic Speech Recognition
122
+ dataset:
123
+ name: "AMI (IHM), subset of up to 1,000 samples"
124
+ type: edinburghcstr/ami
125
+ config: ihm
126
+ split: test
127
+ metrics:
128
+ - type: wer
129
+ value: 8.99
130
+ name: WER (normalized, subset of up to 1,000 samples)
131
+ - task:
132
+ type: automatic-speech-recognition
133
+ name: Automatic Speech Recognition
134
+ dataset:
135
+ name: "GigaSpeech (dev), subset of up to 1,000 samples"
136
+ type: fixie-ai/gigaspeech
137
+ config: dev
138
+ split: dev
139
+ metrics:
140
+ - type: wer
141
+ value: 9.06
142
+ name: WER (normalized, subset of up to 1,000 samples)
143
+ - task:
144
+ type: automatic-speech-recognition
145
+ name: Automatic Speech Recognition
146
+ dataset:
147
+ name: "Earnings22 (held out), subset of up to 1,000 samples"
148
+ type: sanchit-gandhi/earnings22_robust_split
149
+ config: default
150
+ split: test
151
+ metrics:
152
+ - type: wer
153
+ value: 10.58
154
+ name: WER (normalized, subset of up to 1,000 samples)
155
+ - task:
156
+ type: automatic-speech-recognition
157
+ name: Automatic Speech Recognition
158
+ dataset:
159
+ name: "People's Speech, subset of up to 1,000 samples"
160
+ type: fixie-ai/peoples_speech
161
+ config: clean
162
+ split: test
163
+ metrics:
164
+ - type: wer
165
+ value: 17.59
166
+ name: WER (normalized, subset of up to 1,000 samples)
167
+ - task:
168
+ type: automatic-speech-recognition
169
+ name: Automatic Speech Recognition
170
+ dataset:
171
+ name: "AMI (SDM), subset of up to 1,000 samples"
172
+ type: edinburghcstr/ami
173
+ config: sdm
174
+ split: test
175
+ metrics:
176
+ - type: wer
177
+ value: 23.53
178
+ name: WER (normalized, subset of up to 1,000 samples)
179
  ---
180
 
181
  # Tiny Audio
 
183
  **A tiny, hackable, open-source speech LLM.**
184
 
185
  It transcribes English with punctuation, capitalization, word timestamps, and speaker labels: 1.8%
186
+ WER on LibriSpeech test-clean and 7.4% across 12 benchmarks (11,822 samples pooled). Built and
187
+ trained with [Tiny Audio](https://github.com/alexkroman/tiny-audio).
188
 
189
  **[Try the live demo](https://huggingface.co/spaces/mazesmazes/tiny-audio)** ·
190
  **[Train your own](https://github.com/alexkroman/tiny-audio)** ·
 
211
 
212
  ## Benchmarks
213
 
214
+ Word error rate (%, lower is better) on 11,822 samples (up to 1,000 per dataset), scored with the
215
  [Tiny Audio eval harness](https://github.com/alexkroman/tiny-audio/tree/main/scripts/eval)
216
  (`ta eval`) after text normalization.
217
 
218
+ | Dataset | WER |
219
+ | --------------------------- | -------: |
220
+ | LibriSpeech test-clean | 1.84 |
221
+ | SPGISpeech | 2.29 |
222
+ | TED-LIUM | 3.71 |
223
+ | LoquaciousSet † | 6.20 |
224
+ | LibriSpeech test-other | 6.38 |
225
+ | VoxPopuli | 7.11 |
226
+ | Common Voice | 7.18 |
227
+ | AMI (IHM) | 8.99 |
228
+ | GigaSpeech | 9.06 |
229
+ | Earnings22 † | 10.58 |
230
+ | People's Speech | 17.59 |
231
+ | AMI (SDM) | 23.53 |
232
+ | **Mean (12 sets)** | **8.71** |
233
+ | **Pooled (11,822 samples)** | **7.42** |
234
 
235
  † Held out: no data from this source was used in training.
236
 
config.json CHANGED
@@ -66,8 +66,8 @@
66
  "audio_token": "<audio>",
67
  "auto_map": {
68
  "AutoConfig": "asr_config.ASRConfig",
69
- "AutoModel": "asr_modeling.ASRModel",
70
  "AutoModelForSpeechSeq2Seq": "asr_modeling.ASRModel",
 
71
  "AutoProcessor": "asr_processing.ASRProcessor"
72
  },
73
  "custom_pipelines": {
 
66
  "audio_token": "<audio>",
67
  "auto_map": {
68
  "AutoConfig": "asr_config.ASRConfig",
 
69
  "AutoModelForSpeechSeq2Seq": "asr_modeling.ASRModel",
70
+ "AutoModel": "asr_modeling.ASRModel",
71
  "AutoProcessor": "asr_processing.ASRProcessor"
72
  },
73
  "custom_pipelines": {