JuliaFluxGPT / tokenizer.json

Commit History

Fix tokenizer: trim to 2000 vocab to match trained model
db0e784
verified

LisaMegaWatts commited on

Add BPE tokenizer (required for inference)
bd7cff0
verified

LisaMegaWatts commited on