metadata
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- byte-level
- pretraining
Qwen90M Byte
24 vanilla Qwen3 layers, width 640, MLP 1440, 10 query / 2 KV heads, head dimension 64. 90,148,352 tied-embedding parameters. 259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258. Trained from scratch; this is a base model, not a Qwen instruction checkpoint.
Load the model with AutoModelForCausalLM.from_pretrained(repo).
Load its custom byte tokenizer with AutoTokenizer.from_pretrained(repo, trust_remote_code=True).
Training records begin with BOS; prepend tokenizer.bos_token_id to encoded generation prompts.
See training_config.json and coverage.json for the mixture and consumed data statistics.
Model weights are FP32 masters trained under BF16 autocast. Use dtype=torch.bfloat16 for GPU inference.