Nee

A from-scratch ~0.5B parameter decoder-only transformer.

This checkpoint is randomly initialized -- it has not been pretrained yet.

Note: currently model is not trained

Architecture

• Grouped Query Attention: 24 query heads / 8 KV heads, head_dim 64
• RoPE (theta=10000.0)
• RMSNorm fused with the residual add
• SwiGLU MLP (SiLU activation), intermediate size 4096
• No linear biases anywhere
• Flash attention via torch.nn.functional.scaled_dot_product_attention
• Fused QKV projection
• Tied input embedding / LM head weights
• Vocabulary: 40,000 tokens (byte-level BPE)
• Context length: 2048
• Hidden size 1536, 18 layers
• Total parameters: ~0.51B

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("pihu21057w/nee", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("pihu21057w/nee", trust_remote_code=True)

inputs = tokenizer("Hello, I am", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(out[0]))

Pretraining

Weights here are randomly initialized. Pretraining is intended to be done separately (e.g. with Unsloth Studio).