kgrabko commited on
Commit
9772540
·
verified ·
1 Parent(s): a1b6cd0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -42,9 +42,9 @@ tags:
42
  pad_token_id: 151643
43
  eos_token_id: 151643
44
  }
45
- - Compatible with Qwen 3 • Optimized for Code . Use resize function without adaptation, see examples below
46
  - It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
47
- - A Qwen 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
48
  <|fim_prefix|>,
49
  <|fim_middle|>,
50
  <f|im_suffix|>
 
42
  pad_token_id: 151643
43
  eos_token_id: 151643
44
  }
45
+ - Compatible with Qwen 2.5 and DeekSeek R1 • Optimized for Code . Use resize function without adaptation, see examples below
46
  - It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
47
+ - A DeekSeek R1-based tokenizer enhanced with FIM markers from Microsoft datasets and other tags for ML (
48
  <|fim_prefix|>,
49
  <|fim_middle|>,
50
  <f|im_suffix|>