CMA 1M Byte Experiment

1,018,344-parameter byte-level CMA model with a 256-token vocabulary and 4096-byte context. Selected at step 1,000 by the Open SLM average.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/cma-1M-exp"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

WikiText-103: loss 1.5419, BPB 2.2245.

Downloads last month: -

Safetensors

Model size

1.11M params

Tensor type

F32