| ---
|
| language: en
|
| license: mit
|
| tags:
|
| - pytorch
|
| - text-generation
|
| - qwen3
|
| - tinystories
|
| ---
|
|
|
| # Qwen3-0.6B Pre-trained on TinyStories
|
|
|
| This is a Qwen3-0.6B model pre-trained on the TinyStories dataset for 200k iterations.
|
|
|
| ## Model Details
|
|
|
| - **Architecture**: Qwen3-0.6B
|
| - **Training Data**: TinyStories dataset from HuggingFace
|
| - **Training Iterations**: 200,000
|
| - **Parameters**: ~596M unique parameters
|
| - **Tokenizer**: GPT-2 tokenizer (tiktoken)
|
| - **Training Loss**: Available in training history
|
|
|
| ## Quick Start
|
|
|
| ### Download the Model
|
|
|
| ```python
|
| from huggingface_hub import hf_hub_download
|
| import torch
|
|
|
| # Download model weights
|
| model_path = hf_hub_download(
|
| repo_id="vuminhtue/qwen3-200k-tinystories",
|
| filename="Qwen3_200k_model_params.pt"
|
| )
|
|
|
| # Download config
|
| config_path = hf_hub_download(
|
| repo_id="vuminhtue/qwen3-200k-tinystories",
|
| filename="config.json"
|
| )
|
| ```
|
|
|
| ### Load and Use
|
|
|
| ```python
|
| import torch
|
| import tiktoken
|
| from Qwen3_model import Qwen3Model # You need this file from the original code
|
|
|
| # Set up configuration
|
| QWEN3_CONFIG = {
|
| "vocab_size": 151936,
|
| "context_length": 40960,
|
| "emb_dim": 1024,
|
| "n_heads": 16,
|
| "n_layers": 28,
|
| "hidden_dim": 3072,
|
| "head_dim": 128,
|
| "qk_norm": True,
|
| "n_kv_groups": 8,
|
| "rope_base": 1000000.0,
|
| "dtype": torch.bfloat16,
|
| }
|
|
|
| # Load model
|
| model = Qwen3Model(QWEN3_CONFIG)
|
| device = "cuda" if torch.cuda.is_available() else "cpu"
|
| model.load_state_dict(torch.load(model_path, map_location=device))
|
| model = model.to(device)
|
| model.eval()
|
|
|
| # Generate text
|
| tokenizer = tiktoken.get_encoding("gpt2")
|
| # Your generation code here...
|
| ```
|
|
|
| ## Training Details
|
|
|
| - **Optimizer**: AdamW with weight decay (0.1)
|
| - **Learning Rate**: 1e-4 with warmup and cosine decay
|
| - **Batch Size**: 32 with gradient accumulation (32 steps)
|
| - **Context Length**: 128 tokens
|
| - **Mixed Precision**: bfloat16 training
|
|
|
| ## Model Architecture
|
|
|
| - Grouped Query Attention (GQA) with 8 KV groups
|
| - RoPE (Rotary Position Embeddings)
|
| - RMSNorm for normalization
|
| - SiLU activation function
|
| - 28 transformer layers
|
|
|
| ## Performance
|
|
|
| The model was trained on TinyStories, a dataset of simple stories for children. It can generate coherent short stories in a similar style.
|
|
|
| ## Citation
|
|
|
| If you use this model, please cite:
|
|
|
| ```bibtex
|
| @misc{qwen3-tinystories-2025,
|
| author = {Tue Vu},
|
| title = {Qwen3-0.6B Pre-trained on TinyStories},
|
| year = {2025},
|
| publisher = {HuggingFace},
|
| howpublished = {\url{https://huggingface.co/vuminhtue/qwen3-200k-tinystories}},
|
| }
|
| ```
|
|
|
| ## License
|
|
|
| MIT License
|
|
|
| ## Contact
|
|
|
| For questions or issues, please open an issue on the HuggingFace model page.
|
|
|
