xingzhaohu
/

patch_megatron

Model card Files Files and versions

patch_megatron / toolkits /pretrain_data_preprocessing

Ctrl+K

Ctrl+K

1 contributor

History: 4 commits

xingzhaohu's picture

Add files using upload-large-folder tool

6e911ff verified about 1 year ago

README.md

6.42 kB
Add files using upload-large-folder tool about 1 year ago
clean_raw_text.py

2.53 kB
Add files using upload-large-folder tool about 1 year ago
convert_json_to_list.py

378 Bytes
Add files using upload-large-folder tool about 1 year ago
img.png

148 kB
xet

Add files using upload-large-folder tool about 1 year ago
preprocess_data.py

8.69 kB
Add files using upload-large-folder tool about 1 year ago
preprocess_data_megatron.py

18.1 kB
Add files using upload-large-folder tool about 1 year ago
preprocess_wudao2.py

2.77 kB
Add files using upload-large-folder tool about 1 year ago
qwen_hf_preprocess_datasets.py

3.17 kB
Add files using upload-large-folder tool about 1 year ago
run_make_pretraining_dataset.sh

3.28 kB
Add files using upload-large-folder tool about 1 year ago
run_make_pretraining_dataset_megatron.sh

2.9 kB
Add files using upload-large-folder tool about 1 year ago
run_prepare_dataset.sh

1.25 kB
Add files using upload-large-folder tool about 1 year ago
run_prepare_wudao.sh

1.67 kB
Add files using upload-large-folder tool about 1 year ago