Quagmire1
/

wiki-cased

Model card Files Files and versions

wiki-cased / tools /mosesdecoder /scripts /tokenizer

68.2 kB

1 contributor

History: 1 commit

Quagmire1's picture

Upload folder using huggingface_hub

41f6dd8 verified about 1 year ago

mosestokenizer
Upload folder using huggingface_hub about 1 year ago
basic-protected-patterns
267 Bytes

Upload folder using huggingface_hub about 1 year ago
deescape-special-chars-PTB.perl
760 Bytes

Upload folder using huggingface_hub about 1 year ago
deescape-special-chars.perl
729 Bytes

Upload folder using huggingface_hub about 1 year ago
delete-long-words.perl
310 Bytes

Upload folder using huggingface_hub about 1 year ago
detokenizer.perl
12.5 kB

Upload folder using huggingface_hub about 1 year ago
escape-special-chars.perl
847 Bytes

Upload folder using huggingface_hub about 1 year ago
lowercase.perl
383 Bytes

Upload folder using huggingface_hub about 1 year ago
normalize-punctuation.perl
1.91 kB

Upload folder using huggingface_hub about 1 year ago
pre-tok-clean.perl
1.43 kB

Upload folder using huggingface_hub about 1 year ago
pre-tokenizer.perl
967 Bytes

Upload folder using huggingface_hub about 1 year ago
pre_tokenize_cleaning.py
2.96 kB

Upload folder using huggingface_hub about 1 year ago
remove-non-printing-char.perl
549 Bytes

Upload folder using huggingface_hub about 1 year ago
replace-unicode-punctuation.perl
872 Bytes

Upload folder using huggingface_hub about 1 year ago
tokenizer.perl
18.4 kB

Upload folder using huggingface_hub about 1 year ago
tokenizer_PTB.perl
12.3 kB

Upload folder using huggingface_hub about 1 year ago