Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing

  • Log In
  • Sign Up

ByteSpanTokenisers
/
tokenizers

Model card Files Files and versions
xet
Community
tokenizers
80.4 MB
Ctrl+K
Ctrl+K
  • 3 contributors
History: 315 commits
Zeb
Remove old tokenizers
0ce8a7b 11 months ago
  • bytelevel
    Rename bytelevel 11 months ago
  • fw57M_Entropy_bytespanP0-5_8064
    Upload folder using huggingface_hub 11 months ago
  • fw57M_Entropy_thresholdBL_16000
    Upload folder using huggingface_hub 11 months ago
  • fw57M_Entropy_thresholdBL_32000
    Upload folder using huggingface_hub 11 months ago
  • fw57M_Entropy_thresholdBL_8064
    Upload folder using huggingface_hub 11 months ago
  • fw57M_Surprisal_bytespanP0-5_8064
    Upload folder using huggingface_hub 11 months ago
  • fw57Mmulti_Entropy_bytespanP0-5_16000
    Upload folder using huggingface_hub 11 months ago
  • fw57Mmulti_Entropy_bytespanP0-5_32000
    Upload folder using huggingface_hub 11 months ago
  • fw57Mmulti_Entropy_bytespanP0-5_64000
    Upload folder using huggingface_hub 11 months ago
  • fw57Mmulti_Entropy_bytespanP0-5_8064
    Upload folder using huggingface_hub 11 months ago
  • mutual-information_128000
    Rename and remove old tokenizers 12 months ago
  • mutual-information_16000
    Rename mutual information tokenizers 12 months ago
  • mutual-information_256000
    Rename and remove old tokenizers 12 months ago
  • mutual-information_32000
    Rename mutual information tokenizers 12 months ago
  • mutual-information_64000
    Rename mutual information tokenizers 12 months ago
  • mutual-information_8064
    Rename mutual information tokenizers 12 months ago
  • .gitattributes
    3.99 kB
    Upload folder using huggingface_hub 11 months ago