metadata
library_name: transformers
pipeline_tag: text-generation
tags:
- custom_code
- causal-lm
- cma
- byte-level
CMA 1M Byte Experiment
1,018,344-parameter byte-level CMA model with a 256-token vocabulary and 4096-byte context. Selected at step 10,000 by the Open SLM average.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "User01110/cma-1M-exp"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
WikiText-103: loss 1.2384, BPB 1.7866.