---
library_name: transformers
pipeline_tag: text-generation
tags: [qwen3, byte-level, pretraining]
---
# Qwen90M Byte
24 vanilla Qwen3 layers, width 640, MLP 1440, 10 query / 2 KV heads, head dimension 64.
90,148,352 tied-embedding parameters. 259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258.
Trained from scratch; this is a base model, not a Qwen instruction checkpoint.

Load the model with `AutoModelForCausalLM.from_pretrained(repo)`.
Load its custom byte tokenizer with `AutoTokenizer.from_pretrained(repo, trust_remote_code=True)`.
Training records begin with BOS; prepend `tokenizer.bos_token_id` to encoded generation prompts.
See training_config.json and coverage.json for the mixture and consumed data statistics.
Model weights are FP32 masters trained under BF16 autocast. Use dtype=torch.bfloat16 for GPU inference.