jfrick
/

scholarly-code-tokenizer

Model card Files Files and versions

scholarly-code-tokenizer / research /structured-code-v1 /structured-curves.png

jfrick's picture

Publish code-focused 32K tokenizer and promote it to the default

0ebdb4f verified 11 days ago

History Blame Contribute Delete

164 kB

Xet Pointer Details

( Raw pointer file )

Xet hash:: dde4fe27a55224a4d81a31762b2bbc02951cfbc8ff7bdf490685f90d8cc2844e
Size of remote file:: 164 kB
SHA256:: dbb69ccc8a486b41fa99ef1acb535d6204a168ca68cd29395ad59e428ec90d62

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.