v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab d33a32a verified almaghrabima commited on Apr 22
Drop Fanar-2-Diwan from benchmark (unfair Arabic-maximalist peer) 85df039 verified almaghrabima commited on Apr 21
Add benchmark_results.json: v0.1 benchmark vs 11 tokenizers 73dc8b5 verified almaghrabima commited on Apr 21
almaghrabima commited on 