README.md

---
license: llama2
library_name: transformers
---

# CJK Tokenizer

A BPE tokenizer that extends the LLaMA tokenizer vocabulary with all Unicode CJK characters. This tokenizer is designed for ablation experiments to evaluate the impact of using **character-level** tokenization instead of subword-level tokenization on the semantic understanding of CJK text.

## Features

- **Extended Vocabulary**  
  - Base: [LLaMA tokenizer](https://github.com/meta-llama/llama)  
  - Added: 16,689 unique CJK characters covering the full Unicode CJK blocks
- **Character-Level**  
  - Each CJK character is treated as its own token  
  - Other scripts (Latin, punctuation, etc.) follow the original LLaMA subword scheme

## Installation

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("AgentBull/CJK-Tokenizer")

text = "天地玄黃， 宇宙洪荒。 日月盈昃， 辰宿列張。"
tokens = tokenizer(text)
print(tokens.tokens)
```


Tokenizer Details

Property	Value
Base tokenizer	LLaMA tokenizer
Vocabulary size	16,689
Added tokens	All CJK Unified Ideographs (Unicode)
Special tokens	Inherited from LLaMA (<unk>, <s>, </s>, etc.)

Experimental Purpose

This tokenizer was created for ablation studies in CJK language modeling:
	1.	Hypothesis: Character-level tokenization of CJK scripts may improve or alter model semantic understanding compared to subword tokenization.
	2.	Method:
	•	Train or fine-tune identical LLM architectures using (a) the standard LLaMA tokenizer, and (b) this CJKCharTokenizer.
	•	Compare downstream performance on tasks such as language modeling, classification, and machine translation.
	3.	Analysis: Evaluate metrics like perplexity, downstream task accuracy, and qualitative behavior on CJK-specific phenomena.

Contributing

Contributions, bug reports, and feature requests are welcome! Please open an issue or submit a pull request on GitHub.

License

This project is licensed under the MIT License.

Citation

If you use this tokenizer in your research or projects, please cite:

@misc{CJKCharTokenizer2025,
  author       = {Your Name},
  title        = {CJKCharTokenizer: A Character-Level CJK Tokenizer for LLaMA},
  year         = {2025},
  howpublished = {\\url{https://huggingface.co/AgentBull/CJK-Tokenizer}},
}