LnL-AI
/

dbrx-instruct-tokenizer

Transformers.js

Model card Files Files and versions

Qubitium commited on Apr 3, 2024

Commit

d65ef93

·

verified ·

1 Parent(s): 0aca37c

Update README.md

Files changed (1) hide show

README.md +4 -1

README.md CHANGED Viewed

@@ -5,7 +5,10 @@ tags:
  - tokenizers
 ---
-## Original Author: @xenova https://huggingface.co/Xenova, repo: https://huggingface.co/Xenova/dbrx-instruct-tokenizer
 # DBRX Instruct Tokenizer

  - tokenizers
 ---
+## Why should you use this and not the titotken included in the orignal model?
+Original tokenizer pad vocabulary to correct size with `<extra_N>` tokens but encoder never uses them causing inconsistency and deterimental to training code that may want to use the unused `<extra_N>` tokens.
+modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer
 # DBRX Instruct Tokenizer