Instructions to use chrishayuk/v11-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chrishayuk/v11-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chrishayuk/v11-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 743 Bytes
fa41352 b779c03 fa41352 361c586 fa41352 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | {
"schema": "v-tokenizers-provenance-1",
"tokenizer_sha256": "10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae",
"vocab_bin_sha256": "873f44dea58558abd355454b6a22281ba69ad5afe850d05e48c271536471905b",
"vocab_size": 71260,
"status": "adopted",
"source_repo": "https://github.com/chrishayuk/v-tokenizers",
"source_commit": "3ed5f3cb7aea16b150588f27f24a7f31121b1792",
"corpus": {
"catalog": "chuk-datasets",
"dataset": "v11/tokenizer",
"version_id": "v1",
"content_sha": null,
"resolve": "https://chuk-datasets.fly.dev/v1/datasets"
},
"evaluation": {
"ledger": "chuk-experiments",
"run_id": null,
"note": "Headline numbers belong in the card; the ledger is authoritative."
}
} |