StentorLabs
/

StenToken

text-tokenization

Model card Files Files and versions

StenToken / Stentoken /EnglishCode /32k /sten_tokenizer.pt

StentorLabs's picture

Publish runtime tokenizer bundles from 4K through 32K

0ea5ca4 verified 3 days ago

History Blame Contribute Delete

2.44 MB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: 8a727f820656dba9cb65f805752d1a843e17fcd2d7e1d8976ac34fafabdf074b
Size of remote file:: 2.44 MB
SHA256:: 96b2ee611fa328c7b46f08ee7a1d705eff2530b9eb99f86d1a649b7dff34adab

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.