StentorLabs
/

StenToken

text-tokenization

Model card Files Files and versions

StenToken / Stentoken /EnglishCode /16k /sten_tokenizer.pt

StentorLabs's picture

Publish runtime tokenizer bundles from 4K through 32K

0ea5ca4 verified 3 days ago

History Blame Contribute Delete

1.2 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: 041376a69cfca8996ea8605ba0014abe2b4e3fcfbdd023e3ae817337f69d49e0
Size of remote file:: 1.2 MB
SHA256:: f9e2127933eccb6f860f225f4ebf9136c3d3d7c6b6a64c2e36b6ef0c80f91628

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.