Instructions to use AgentBull/CJK-Tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AgentBull/CJK-Tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AgentBull/CJK-Tokenizer", dtype="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
library_name: transformers
|
| 4 |
---
|
| 5 |
|
|
@@ -68,5 +68,4 @@ If you use this tokenizer in your research or projects, please cite:
|
|
| 68 |
title = {CJKCharTokenizer: A Character-Level CJK Tokenizer for LLaMA},
|
| 69 |
year = {2025},
|
| 70 |
howpublished = {\\url{https://huggingface.co/AgentBull/CJK-Tokenizer}},
|
| 71 |
-
}
|
| 72 |
-
|
|
|
|
| 1 |
---
|
| 2 |
+
license: llama2
|
| 3 |
library_name: transformers
|
| 4 |
---
|
| 5 |
|
|
|
|
| 68 |
title = {CJKCharTokenizer: A Character-Level CJK Tokenizer for LLaMA},
|
| 69 |
year = {2025},
|
| 70 |
howpublished = {\\url{https://huggingface.co/AgentBull/CJK-Tokenizer}},
|
| 71 |
+
}
|
|
|