Mini-Qwen Custom Model

This is a compact, custom Qwen-inspired autoregressive language model trained from scratch.

Model Highlights

• Grouped-Query Attention (GQA)
• Learnable QK RMSNorm for training stability
• Rotary Position Embeddings (RoPE)
• SwiGLU Feed-Forward layers
• Tied Input/Output word embeddings

Usage

You can load this model directly using standard Hugging Face transformers APIs with trust_remote_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "sarimahsan101/mini-qwen"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

prompt = "Once upon a time, there was a little boy named"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))