Use the Qwen3.5 base pre-tokenizer and decoder (regex [\p{L}\p{M}]+); tokenizer_class TokenizersBackend so transformers loads tokenizer.json verbatim; supersedes 177cdc85 (T1's Qwen2 pre-tokenizer)

dc692bd verified 14 days ago

History Blame Contribute Delete

12.8 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: a31ef70cf9b97de24bf3d7c79b427130e20bb9550d326a01ef903c1be0da915a
Size of remote file:: 12.8 MB
SHA256:: 5b9d57867977c857094bcd169e11505a4997c55d32ac0aec87dbdb58cf151b21

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.