File size: 457 Bytes
4d3248c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | uv run python tools/preprocess_multiproc.py ^
--manifest dataset/thaitts_emilia_dataset/forbpe.jsonl ^
--output-dir omniman2_data ^
--tokenizer checkpoints/thai3normalbpe.model ^
--config checkpoints/config.yaml ^
--gpt-checkpoint checkpoints/gpt.pth ^
--language th ^
--device cuda ^
--batch-size 1 ^
--workers 1 ^
--num-processes 1 ^
--launch-delay 20 ^
--skip-existing ^
--val-ratio 0.01
|