| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: [qwen3, byte-level, pretraining] | |
| # Qwen90M Byte | |
| 24 vanilla Qwen3 layers, width 640, MLP 1440, 10 query / 2 KV heads, head dimension 64. | |
| 90,148,352 tied-embedding parameters. 259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258. | |
| Trained from scratch; this is a base model, not a Qwen instruction checkpoint. | |
| Load the model with `AutoModelForCausalLM.from_pretrained(repo)`. | |
| Load its custom byte tokenizer with `AutoTokenizer.from_pretrained(repo, trust_remote_code=True)`. | |
| Training records begin with BOS; prepend `tokenizer.bos_token_id` to encoded generation prompts. | |
| See training_config.json and coverage.json for the mixture and consumed data statistics. | |
| Model weights are FP32 masters trained under BF16 autocast. Use dtype=torch.bfloat16 for GPU inference. | |