RaymondLi
/

gpt-4-tokenizer

Transformers.js

Model card Files Files and versions

gpt-4-tokenizer

Commit History

set clean_up_tokenization_spaces to false by default

68ffc2f
unverified

RaymondLi commited on Oct 11, 2023

adjust pretokenizer regex

a80ab01
unverified

RaymondLi commited on Oct 11, 2023

Create README.md

c9238fc

Xenova HF Staff commited on Aug 4, 2023

Upload 3 files

9959d33

Xenova HF Staff commited on Aug 4, 2023

Upload 2 files

80bf8d0

Xenova HF Staff commited on Aug 4, 2023

initial commit

c6d34a3

Xenova HF Staff commited on Aug 4, 2023