anhtn
/

wikitext-vnen-tokenized

Model card Files Files and versions

wikitext-vnen-tokenized / tokenizer.json

anhtn's picture

Upload BPE tokenizer (vocab_size=151k)

6e5fa4a verified 2 months ago

history blame contribute delete

11.1 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: d15591df0069000c0fea5e9ee2f15d7e8fed26280251a748ee3a4fc16a74d93f
Size of remote file:: 11.1 MB
SHA256:: 07f34af9440177370ac91454eb524780622cad3db81e894000080c80f78e0221

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.