Hugging Face's logo

agentlans
/
multilingual-tokenizers

Model card Files
xet
 Community
agentlans commited on
Commit
86c3569
·
verified ·
1 Parent(s): b9b9dcd

Upload 255 files

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. huggingface/all/special_tokens_map.json +6 -0
  2. huggingface/all/tokenizer.json +0 -0
  3. huggingface/all/tokenizer_config.json +0 -0
  4. huggingface/ar/special_tokens_map.json +6 -0
  5. huggingface/ar/tokenizer.json +0 -0
  6. huggingface/ar/tokenizer_config.json +0 -0
  7. huggingface/az/special_tokens_map.json +6 -0
  8. huggingface/az/tokenizer.json +0 -0
  9. huggingface/az/tokenizer_config.json +0 -0
  10. huggingface/bg/special_tokens_map.json +6 -0
  11. huggingface/bg/tokenizer.json +0 -0
  12. huggingface/bg/tokenizer_config.json +0 -0
  13. huggingface/bn/special_tokens_map.json +6 -0
  14. huggingface/bn/tokenizer.json +0 -0
  15. huggingface/bn/tokenizer_config.json +0 -0
  16. huggingface/ca/special_tokens_map.json +6 -0
  17. huggingface/ca/tokenizer.json +0 -0
  18. huggingface/ca/tokenizer_config.json +0 -0
  19. huggingface/cs/special_tokens_map.json +6 -0
  20. huggingface/cs/tokenizer.json +0 -0
  21. huggingface/cs/tokenizer_config.json +0 -0
  22. huggingface/da/special_tokens_map.json +6 -0
  23. huggingface/da/tokenizer.json +0 -0
  24. huggingface/da/tokenizer_config.json +0 -0
  25. huggingface/de/special_tokens_map.json +6 -0
  26. huggingface/de/tokenizer.json +0 -0
  27. huggingface/de/tokenizer_config.json +0 -0
  28. huggingface/el/special_tokens_map.json +6 -0
  29. huggingface/el/tokenizer.json +0 -0
  30. huggingface/el/tokenizer_config.json +0 -0
  31. huggingface/en/special_tokens_map.json +6 -0
  32. huggingface/en/tokenizer.json +0 -0
  33. huggingface/en/tokenizer_config.json +0 -0
  34. huggingface/es/special_tokens_map.json +6 -0
  35. huggingface/es/tokenizer.json +0 -0
  36. huggingface/es/tokenizer_config.json +0 -0
  37. huggingface/et/special_tokens_map.json +6 -0
  38. huggingface/et/tokenizer.json +0 -0
  39. huggingface/et/tokenizer_config.json +0 -0
  40. huggingface/fa/special_tokens_map.json +6 -0
  41. huggingface/fa/tokenizer.json +0 -0
  42. huggingface/fa/tokenizer_config.json +0 -0
  43. huggingface/fi/special_tokens_map.json +6 -0
  44. huggingface/fi/tokenizer.json +0 -0
  45. huggingface/fi/tokenizer_config.json +0 -0
  46. huggingface/fr/special_tokens_map.json +6 -0
  47. huggingface/fr/tokenizer.json +0 -0
  48. huggingface/fr/tokenizer_config.json +0 -0
  49. huggingface/he/special_tokens_map.json +6 -0
  50. huggingface/he/tokenizer.json +0 -0
huggingface/all/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/all/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/all/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/ar/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/ar/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/ar/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/az/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/az/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/az/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/bg/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/bg/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/bg/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/bn/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/bn/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/bn/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/ca/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/ca/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/ca/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/cs/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/cs/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/cs/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/da/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/da/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/da/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/de/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/de/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/de/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/el/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/el/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/el/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/en/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/en/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/en/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/es/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/es/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/es/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/et/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/et/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/et/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fa/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/fa/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fa/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fi/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/fi/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fi/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fr/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/fr/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/fr/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
huggingface/he/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "<pad>",
5
+ "unk_token": "<unk>"
6
+ }
huggingface/he/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff