Models
Datasets
Spaces
Docs
Enterprise
Pricing
Log In
Sign Up

ByteSpanTokenisers
/

tokenizers

Model card Files Files and versions

18.9 MB

3 contributors

History: 322 commits

Zeb

Remove old tokenizers

cd48ea4 8 months ago

bytelevel
Rename bytelevel 8 months ago
fw57M_Entropy_bytespanP0-5_8064
Upload folder using huggingface_hub 8 months ago
fw57M_Entropy_thresholdBL_16000
Upload folder using huggingface_hub 8 months ago
fw57M_Entropy_thresholdBL_32000
Upload folder using huggingface_hub 8 months ago
fw57M_Entropy_thresholdBL_8064
Upload folder using huggingface_hub 8 months ago
fw57M_Surprisal_bytespanP0-5_8064
Upload folder using huggingface_hub 8 months ago
fw57Mmulti_Entropy_bytespanP0-5_16000
Upload folder using huggingface_hub 8 months ago
fw57Mmulti_Entropy_bytespanP0-5_32000
Upload folder using huggingface_hub 8 months ago
fw57Mmulti_Entropy_bytespanP0-5_64000
Upload folder using huggingface_hub 8 months ago
fw57Mmulti_Entropy_bytespanP0-5_8064
Upload folder using huggingface_hub 8 months ago
.gitattributes

3.99 kB

Upload folder using huggingface_hub 8 months ago