Hugging Face's logo

Spaces:
sanderland
/
claude-tokenizer
Running on Zero

App Files Community
claude-tokenizer

Commit History

Document the marker notation
f3775d9
Running
verified

sanderland commited on

Separate adjacent byte markers
f9f1aef
verified

sanderland commited on

Blog notation for markers, no toggle
4c720a7
verified

sanderland commited on

Keep word spacing when markers are hidden
173ddf7
verified

sanderland commited on

Mobile-friendly layout
81723d3
verified

sanderland commited on

Number each article once
3e56579
verified

sanderland commited on

Correctness, RTL and performance fixes
4a15d75
verified

sanderland commited on

Seven languages, one sentence each
e69c434
verified

sanderland commited on

UDHR articles 1-5, one language each
68ab539
verified

sanderland commited on

Six UDHR articles as the lead example
00724b5
verified

sanderland commited on

Cleaner table, links, and examples worth reading
4b80632
verified

sanderland commited on

Link the source, the package and the writeup
1ced414
verified

sanderland commited on

Retitle around Claude
aa492c1
verified

sanderland commited on

Tokenizer comparison app
0d3e580
verified

sanderland commited on

initial commit
1f75a17
verified

sanderland commited on