Instructions to use Xenova/claude-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Xenova/claude-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Xenova/claude-tokenizer", device_map="auto") - Transformers.js
How to use Xenova/claude-tokenizer with Transformers.js:
// ⚠️ Unknown pipeline tag
- Notebooks
- Google Colab
- Kaggle
Commit History
-
Update tokenizer_config.json (#2)
Verified signature- Update tokenizer_config.json (01ec25d2655f6e21316f8fa99b116ea3125ffa2c) Co-authored-by: Tiezhen WANG <xianbao@users.noreply.huggingface.co>cae6888
-
Update README.md
Verified signature9e41188 -
Create tokenizer_config.json
Verified signaturef7fee9a -
Upload tokenizer.json
Verified signaturefaa0a2d -
initial commit
Verified signaturebb6cb88