Instructions to use hf-internal-testing/udop-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use hf-internal-testing/udop-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("hf-internal-testing/udop-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
-
Update tokenizer_config.json
Verified signature5c0d38f -
Delete added_tokens.json
Verified signature781b1bc -
Delete special_tokens_map.json
Verified signature76ecccd -
Update tokenizer_config.json
Verified signature3588982 -
Upload tokenizer
Verified signature77483f3 -
initial commit
Verified signatured0d8824