w2vbert-lingala-waxal-punct

Dedicated Lingala W2V-BERT CTC trained WITH punctuation in the vocabulary, on WAXAL Lingala. Emits . , ? ! directly rather than relying on a downstream restorer.

Best validation results:

• Loss: 0.1436
• Wer (keep): 0.1187
• Cer (keep): 0.0332
• Zindi (keep): 0.9240
• Wer (strip): 0.0572
• Zindi (strip): 0.9611

Training hyperparameters

• learning_rate: 3e-05
• train_batch_size: 16
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: AdamW (betas=(0.9,0.98))
• lr_scheduler_type: linear, warmup_ratio 0.1
• num_epochs: up to 40 (early stopped)

Training results

Epoch	Step	Val Loss	WER keep	CER keep	Zindi keep	WER strip	Zindi strip
0.24	4	6.3582	0.9999	0.9496	0.0253	0.9994	0.0343
0.49	9	2.3374	0.9861	0.6471	0.1834	0.9855	0.1698
0.73	14	0.2802	0.1980	0.0566	0.8727	0.1410	0.9076
0.98	19	0.1956	0.1397	0.0388	0.9107	0.0777	0.9481
1.22	24	0.1849	0.1275	0.0358	0.9184	0.0653	0.9559
1.47	29	0.1739	0.1243	0.0344	0.9207	0.0620	0.9580
1.71	34	0.1604	0.1245	0.0352	0.9201	0.0615	0.9583
1.96	39	0.1524	0.1201	0.0336	0.9231	0.0586	0.9601
2.20	44	0.1457	0.1202	0.0341	0.9229	0.0581	0.9605
2.45	49	0.1508	0.1202	0.0340	0.9229	0.0585	0.9603
2.69	54	0.1407	0.1210	0.0342	0.9224	0.0597	0.9594
2.94	59	0.1436	0.1187	0.0332	0.9240	0.0572	0.9611
3.18	64	0.1382	0.1204	0.0337	0.9229	0.0588	0.9600
3.42	69	0.1409	0.1220	0.0345	0.9217	0.0574	0.9609
3.67	74	0.1449	0.1202	0.0345	0.9227	0.0580	0.9605
3.91	79	0.1341	0.1238	0.0340	0.9211	0.0601	0.9593
4.16	84	0.1435	0.1245	0.0352	0.9202	0.0602	0.9592
4.40	89	0.1510	0.1245	0.0346	0.9204	0.0586	0.9603
4.65	94	0.1638	0.1246	0.0346	0.9204	0.0622	0.9580
4.89	99	0.1559	0.1253	0.0346	0.9201	0.0600	0.9595

Framework versions

• Transformers 5.13.0
• Pytorch 2.12.1
• Datasets 3.6.0