Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing

  • Log In
  • Sign Up

2ira
/
Byte-lingua-code

Model card Files Files and versions
xet
Community
Byte-lingua-code / superbpe /tokenizers_superbpe /tokenizers /src /tokenizer
212 kB
Ctrl+K
Ctrl+K
  • 1 contributor
History: 1 commit
2ira's picture
2ira
offline_compression_graph_code
72c0672 verified 4 months ago
  • added_vocabulary.rs
    36.3 kB
    offline_compression_graph_code 4 months ago
  • encoding.rs
    31.5 kB
    offline_compression_graph_code 4 months ago
  • mod.rs
    46.3 kB
    offline_compression_graph_code 4 months ago
  • normalizer.rs
    72 kB
    offline_compression_graph_code 4 months ago
  • pattern.rs
    7.63 kB
    offline_compression_graph_code 4 months ago
  • pre_tokenizer.rs
    10.8 kB
    offline_compression_graph_code 4 months ago
  • serialization.rs
    7.35 kB
    offline_compression_graph_code 4 months ago