Fleet-2 / README.md
Lucasgitguy-co's picture
Link Fleet-Clean
51757dd verified
|
Raw History Blame Contribute Delete
6.28 kB
---
license: apache-2.0
base_model: Lucasgitguy-co/Fleet-1
language: [en, de, fr, es, it, nl, pl, ko]
pipeline_tag: automatic-speech-recognition
datasets: [facebook/voxpopuli, Bingsu/zeroth-korean, google/fleurs]
tags: [whistle, cactus, lora, speech-to-text, distillation, pseudo-labels]
---
# Fleet-2
A 30 MB on-device speech-to-text model (Cactus Whistle architecture) for English, German, French, Spanish, Italian, Dutch, Polish and Korean, trained on transcripts written by [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo).
**Collection:** [Speech recognition](https://huggingface.co/collections/Lucasgitguy-co/speech-recognition-6aef2dcbe0ea208136b0ddc9)
**Clean text:** [Fleet-Clean-0.36B](https://huggingface.co/Lucasgitguy-co/Fleet-Clean-0.36B) chains Fleet-2 with a word tagger that removes fillers, repeats and false starts and fixes punctuation ([GGUF](https://huggingface.co/Lucasgitguy-co/Fleet-Clean-0.36B-GGUF)).
## What changed from Fleet-1
- **Teacher:** Whisper-large-v3-turbo transcribed new audio, and Fleet-2 learned from those transcripts. This is sequence-level distillation, as in Distil-Whisper. Whisper and Whistle use different tokenizers, so no logits were matched.
- **New audio:** VoxPopuli (European Parliament speech) for the seven European languages and Zeroth-Korean for Korean. Fleet-1 used FLEURS only.
- **Start point:** chosen by validation score between Sky-Kim's Korean Whistle and Fleet-1. This run started from `Lucasgitguy-co/Fleet-1`.
## Evaluation
Each language has 100 short FLEURS test clips. These are the same clips, filtered the same way, as Fleet-1's table. All models ran in the same native Cactus engine with automatic language detection. Lower is better. Whistle has no Korean.
| Language | Metric | Whistle | Sky-Kim | Fleet-1 | Fleet-2 |
|---|---|---:|---:|---:|---:|
| en | WER | 13.86% | 14.87% | 14.39% | **14.50%** |
| de | WER | 22.45% | 22.71% | 22.18% | **22.02%** |
| fr | WER | 22.15% | 25.73% | 25.18% | **24.93%** |
| es | WER | 11.83% | 11.83% | 11.93% | **11.93%** |
| it | WER | 17.44% | 16.60% | 16.32% | **16.66%** |
| nl | WER | 32.95% | 29.39% | 29.53% | **30.06%** |
| pl | WER | 45.38% | 43.68% | 43.13% | **41.62%** |
| ko | CER | n/a | 33.68% | 52.35% | **42.80%** |
Fleet-2 beats Fleet-1 in 4 of 8 languages (de, fr, pl, ko). Korean CER is no worse than Sky-Kim: no. **Improvement over Fleet-1 was NOT demonstrated. Check each language; some may regress.**
<details>
<summary>Out-of-domain clips (VoxPopuli / Zeroth-Korean test splits)</summary>
| Language | Source | Clips | Whistle | Sky-Kim | Fleet-1 | Fleet-2 |
|---|---|---:|---:|---:|---:|---:|
| en | VoxPopuli | 50 | 10.56% | 12.69% | 10.91% | **11.98%** |
| de | VoxPopuli | 50 | 17.32% | 19.66% | 19.53% | **18.36%** |
| fr | VoxPopuli | 50 | 13.06% | 12.24% | 12.04% | **13.57%** |
| es | VoxPopuli | 50 | 9.99% | 11.89% | 11.70% | **11.04%** |
| it | VoxPopuli | 50 | 34.45% | 40.21% | 38.22% | **33.19%** |
| nl | VoxPopuli | 50 | 23.06% | 23.29% | 23.17% | **22.95%** |
| pl | VoxPopuli | 50 | 22.02% | 23.53% | 22.98% | **23.53%** |
| ko | Zeroth-Korean | 50 | n/a | 25.19% | 15.65% | **13.24%** |
Fleet-2 trained on other clips from these corpora, so this table shows in-corpus generalisation, not a fully independent test.
</details>
These results come from short clips of 12 s or less. They are not full benchmark scores and not significance tests. Accuracy on long, noisy or conversational audio was not measured. Upstream models may have seen FLEURS training data before.
## Training
- **Data:** 4326 clips kept out of 6400 that the teacher labeled: 3014 new VoxPopuli/Zeroth clips and 1312 of Fleet-1's FLEURS training clips. All clips are 12 s or less.
- **Teacher decoding:** language forced, temperature 0, no conditioning on previous text. Runtime: the MLX conversion `mlx-community/whisper-large-v3-turbo`.
- **Filter:** a clip was kept only if all of these held:
- compression ratio ≤ 2.4
- average log-probability ≥ -1.0
- no-speech probability ≤ 0.6
- teacher text within 10% WER of the human transcript (CER for Korean)
The training target is the teacher's text.
- **Excluded from training:** validation and test clips, and any transcript that matches an evaluation transcript.
- **Optimisation:** the start adapter (LoRA plus Korean vocabulary rows) continued with AdamW at 2e-05, warmup/cosine schedule, batch 1, 2 epochs (8652 steps), clip 1.0, W4/A8 quantisation-aware training on CPU (Apple M4).
- **Checkpoint selection:** step 2163, chosen by macro validation error on Fleet-1's FLEURS validation clips (0.2212, against 0.2283 for the start model). Test clips were not used for selection.
- **Release:** published whatever the scores. The comparison above is reported as measured, including any regressions.
`training_manifest.json` records every clip, teacher transcript and pinned revision. `evaluation.json` has every score. `fleet2.py`, `teacher.py` and `run.sh` reproduce the run.
## Usage
```python
import cactus
from huggingface_hub import hf_hub_download
path = hf_hub_download("Lucasgitguy-co/Fleet-2", "whistle.cact")
print(cactus.transcribe("clip.wav", weights=path)["text"])
```
Use 16 kHz mono WAV and `cactus-compute` at revision `ee2ffc67fe0f9537d4928e158440c7d228a63100`. This is a Cactus model, not a Transformers checkpoint.
## License and attribution
- **Model:** Apache License 2.0. The original Whistle LICENSE is kept verbatim. Architecture and pretrained weights: Cactus Compute's Whistle (Mroz et al., 2026). Korean vocabulary and adapter lineage: [Sky-Kim/whistle-finetuned-ko](https://huggingface.co/Sky-Kim/whistle-finetuned-ko) and [Lucasgitguy-co/Fleet-1](https://huggingface.co/Lucasgitguy-co/Fleet-1). See `UPSTREAM_MODEL_CARD.md`.
- **Teacher:** [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo), MIT License. Only its transcripts were used; none of its weights are included.
- **Data:**
- [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli): CC0
- [Zeroth-Korean](https://www.openslr.org/40/): CC BY 4.0
- [FLEURS](https://huggingface.co/datasets/google/fleurs) (Conneau et al., 2022): CC BY 4.0
No audio is included in this repository. See `NOTICE`.