Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Website
Tasks
HuggingChat
Collections
Languages
Organizations
Community
Blog
Posts
Daily Papers
Learn
Discord
Forum
GitHub
Solutions
Team & Enterprise
Hugging Face PRO
Enterprise Support
Inference Providers
Inference Endpoints
Storage Buckets
Log In
Sign Up
Spaces:
nvidia
/
Plan2Align-NV
like
3
Paused
App
Files
Files
Community
1
Fetching metadata from the HF Docker repository...
main
Plan2Align-NV
/
laser
/
tools-external
/
moses-tokenizer
/
tokenizer
32.8 kB
Ctrl+K
Ctrl+K
3 contributors
History:
1 commit
KuangDW
add laser tool
2aebc50
about 1 year ago
basic-protected-patterns
178 Bytes
add laser tool
about 1 year ago
deescape-special-chars.perl
631 Bytes
add laser tool
about 1 year ago
detokenizer.perl
12.5 kB
add laser tool
about 1 year ago
lowercase.perl
285 Bytes
add laser tool
about 1 year ago
normalize-punctuation.perl
1.91 kB
add laser tool
about 1 year ago
remove-non-printing-char.perl
451 Bytes
add laser tool
about 1 year ago
tokenizer.perl
16.8 kB
add laser tool
about 1 year ago
