yuekai/icefall_asr_aishell_whisper

Results

Aishell training results (Fine-tuning Pretrained Models)

Whisper

fine-tuning results on Aishell test set on whisper medium, large-v2, large-v3

	test (greedy search, before fine-tuning)	test (beam=10, after fine-tuning)	comment
medium	7.23	3.27	--epoch 10 --avg 4, ddp
large-v2	6.56	2.47	--epoch 10 --avg 6, deepspeed zero stage1
large-v3	6.06	2.84	--epoch 5 --avg 3, deepspeed zero stage1

Command for training is:

./prepare.sh --stage 30 --stop_stage 30

#fine-tuning with deepspeed zero stage 1
torchrun --nproc-per-node 8 ./whisper/train.py \
  --max-duration 200 \
  --use-fp16 1 \
  --exp-dir whisper/exp_large_v2 \
  --model-name large-v2 \
  --deepspeed \
  --deepspeed_config ./whisper/ds_config_zero1.json

# fine-tuning with ddp
torchrun --nproc-per-node 8 ./whisper/train.py \
  --max-duration 200 \
  --use-fp16 1 \
  --exp-dir whisper/exp_medium \
  --base-lr 1e-5 \
  --model-name medium

Command for decoding is:

python3 ./whisper/decode.py \
  --exp-dir whisper/exp_large_v2 \
  --model-name large-v2 \
  --epoch 999 --avg 1 \
  --beam-size 10 --max-duration 50

NOTE: To decode with original whisper models, you should pad the input features into 30 secs. Otherwise it may not ouput EOS token.

Pretrained models, training logs, decoding logs, tensorboard and decoding results are available at https://huggingface.co/yuekai/icefall_asr_aishell_whisper