Implement data pipeline module
#2
by xiexinyuan341 - opened
Complete the data module for English-Chinese translation: dataset loading, BPE tokenizer wrapper/training, preprocessing, collator, dynamic batching, README usage guide, and unit tests.
Verification: /opt/anaconda3/envs/easytranslate/bin/python -m pytest tests/test_data.py -q
ok
sdfjliom changed pull request status to merged