doof-ferb/whisper-tiny-vi

whisper tiny fine-tuned on a very big collection of vietnamese speech datasets

TODO:

• training then publish checkpoint
• evaluate WER on Common Voice & FLEURS & VIVOS
• convert to openai-whisper, whisper.cpp, faster-whisper
• convert to ONNX: to try https://github.com/k2-fsa/sherpa-onnx & https://github.com/zhuzilin/whisper-openvino
• convert to TensorRT: https://github.com/openai/whisper/discussions/169

21k steps, warm-up 5%, batch size 16×2 (kaggle free T4×2)

manually evaluate WER on test set - vietnamese part:

@ float16	CommonVoice v16.1	FLEURS	VIVOS
original whisper-tiny	>100%	88.6%	62.5%
this model	26.6%	37.1%	18.7%

all training + evaluation scripts are on my repo: https://github.com/phineas-pta/fine-tune-whisper-vi

usage example:

import torch
from transformers import pipeline

PIPE = pipeline(task="automatic-speech-recognition", model="doof-ferb/whisper-tiny-vi", device="cuda:0", torch_dtype=torch.float16)
PIPE_KWARGS = {"language": "vi", "task": "transcribe"}

PIPE("audio.mp3", generate_kwargs=PIPE_KWARGS)["text"]