bart / data /data-bin /preprocess.log
sonithoiningombam's picture
Add files using upload-large-folder tool
12f413a verified
Raw
History Blame Contribute Delete
2.06 kB
Namespace(aim_repo=None, aim_run_hash=None, align_suffix=None, alignfile=None, all_gather_list_size=16384, amp=False, amp_batch_retries=2, amp_init_scale=128, amp_scale_window=None, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin', dict_only=False, empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, on_cpu_convert_precision=False, only_source=False, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang='mni_Beng', srcdict=None, suppress_crashes=False, target_lang='eng_Latn', task='translation', tensorboard_logdir=None, testpref='../data/test', tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='../data/train', use_plasma_view=False, user_dir=None, validpref='../data/dev', wandb_project=None, workers=1)
[mni_Beng] Dictionary: 185504 types
[mni_Beng] ../data/train.mni_Beng: 139266 sents, 2418773 tokens, 0.0% replaced (by <unk>)
[mni_Beng] Dictionary: 185504 types
[mni_Beng] ../data/dev.mni_Beng: 2404 sents, 41351 tokens, 3.87% replaced (by <unk>)
[mni_Beng] Dictionary: 185504 types
[mni_Beng] ../data/test.mni_Beng: 1183 sents, 20030 tokens, 4.03% replaced (by <unk>)
[eng_Latn] Dictionary: 104488 types
[eng_Latn] ../data/train.eng_Latn: 139266 sents, 2851668 tokens, 0.0% replaced (by <unk>)
[eng_Latn] Dictionary: 104488 types
[eng_Latn] ../data/dev.eng_Latn: 2404 sents, 48999 tokens, 1.72% replaced (by <unk>)
[eng_Latn] Dictionary: 104488 types
[eng_Latn] ../data/test.eng_Latn: 1183 sents, 23873 tokens, 1.63% replaced (by <unk>)
Wrote preprocessed data to data-bin