khtsly/Nalu-2.0-Base

Nalu 2.0 - 0.3B (Base) 🦭

1. Model Introduction

Nalu 2.0 is an open-weight base model for Luau and Roblox code, and the smallest model in the LuauCoder 2.0 family (Moana 2B, Kai 0.6B, Nalu 0.3B). It is a dense decoder with 24 layers and 338M parameters.

This is a base model with no chat template, so it continues raw text rather than following instructions. Prompt it with the start of the code you want.

2. Model Summary

Architecture	Dense decoder
Total Parameters	338M
Non-Embedding Parameters	304M
Number of Layers	24
Attention-Layer Composition	18 RoPE + 6 NoPE
Attention Mechanism	CCVHA + DiffAttn V2 + XSA
Hidden Dimension	1024
Number of Query Heads	16 (8 computed)
Number of KV Heads	2
Number of Output Heads	8
Head Dimension	128
Convolution Taps	2 depthwise + 2 per head
Head Postmix Rank	2
Residual Connections	Block AttnRes (6 blocks of 4 layers)
Feed-Forward Hidden Dimension	3072
Activation Function	SwiGLU
Normalization	RMSNorm, QK-Norm
RoPE Theta	500,000
Vocabulary Size	32768
Embeddings	Tied
Logit Soft Cap	±30
Modality	Text

3. Architecture

4. Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "khtsly/Nalu-2.0-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype="bfloat16",
    attn_implementation="sdpa",  # or "flash_attention_2"
).to("cuda")

prompt = "local function lerp(a, b, t)"
inputs = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=30,
    repetition_penalty=1.05,
)
print(tok.decode(out[0], skip_special_tokens=True))

License

This repository and LuauCoder families model weights are released under the Apache-2.0 License.