Instructions to use Xenova/dbrx-instruct-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Xenova/dbrx-instruct-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Xenova/dbrx-instruct-tokenizer", dtype="auto") - Transformers.js
How to use Xenova/dbrx-instruct-tokenizer with Transformers.js:
// ⚠️ Unknown pipeline tag
- Notebooks
- Google Colab
- Kaggle
Upload vocab.json
Browse files- vocab.json +3 -2
vocab.json
CHANGED
|
@@ -100260,6 +100260,7 @@
|
|
| 100260 |
"<|fim_middle|>": 100259,
|
| 100261 |
"<|fim_suffix|>": 100260,
|
| 100262 |
"<|endofprompt|>": 100276,
|
| 100263 |
-
"<|
|
| 100264 |
-
"<|
|
|
|
|
| 100265 |
}
|
|
|
|
| 100260 |
"<|fim_middle|>": 100259,
|
| 100261 |
"<|fim_suffix|>": 100260,
|
| 100262 |
"<|endofprompt|>": 100276,
|
| 100263 |
+
"<|pad|>": 100277,
|
| 100264 |
+
"<|im_start|>": 100278,
|
| 100265 |
+
"<|im_end|>": 100279
|
| 100266 |
}
|