eljanmahammadli
/

aztext-tokenizer

Model card Files Files and versions

aztext-tokenizer / tokenizer.model

eljanmahammadli's picture

eljanmahammadli

Initial release: SentencePiece BPE 16k tokenizer for AzText

51b6b2e verified 11 days ago

history blame contribute delete

504 kB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet Pointer Details

( Raw pointer file )

Xet hash:: 9840a0014121ea6d32464c2a1a6a21509a9058cd12f5ff5f3e51205c9b16ae36
Size of remote file:: 504 kB
SHA256:: d63d3f55b22ec08c02fce3f257d9987a8e27c9bfe69cff9544dabcc68c2e0446

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.