license: apache-2.0
language:
- en
base_model: mlx-community/Qwen3-8B-4bit
tags:
- mlx
- lora
- distillation
- code
- luau
- roblox
- text-generation
pipeline_tag: text-generation
vertex_family: codein
vertex_params: 8B
vertex_purpose: Luau code completion
vertex_result: >-
18% real syntax-error rate on base vs 2% tuned, on 100 held-out real-corpus
prompts (luau-analyze)
vertex_status: released
vertex_name: CodeInLuau 1 Small
vertex_base: Qwen3-8B
CodeInLuau 1 Small
A Luau code-completion specialist, distilled from real Roblox source into Qwen3-8B
Part of the CodeIn family: where Aquamarine is a generalist across languages, CodeIn models go the other direction -- each one distilled for exactly one programming language, and nothing else. CodeInLuau is the first: Luau, Roblox's dialect of Lua.
Available formats
Everything below lives in this one repository: pick the format that matches your runtime.
| Format | File / folder | Precision | Size | Works with |
|---|---|---|---|---|
| MLX | model.safetensors (repo root) |
4-bit (group size 64) | 4.61 GB | mlx-lm / MLX on Apple Silicon |
A GGUF build is not provided for this model.
Overview
CodeInLuau 1 Small is fine-tuned via LoRA on Qwen3-8B for raw Luau code completion -- given a prefix (a partial script, a function signature, a comment), it continues the code, not a chat reply. It's trained on real, human-written Luau scraped from live Roblox experiences, not synthetic/generated code.
Training
- Base model:
mlx-community/Qwen3-8B-4bit - Dataset:
Roblox/luau_corpus-- real, human-written Luau, no synthetic generation. 19,608 train / 4,352 validation examples, each a genuine prefix/continuation pair from real Roblox source, trained as plain text completion (no chat template -- this is a raw code-completion task, not instruction-following). - Method: LoRA fine-tuning (rank 8, scale 20, 16 layers), 3,000 iterations, batch size 1, sequence length 2048.
- Val loss: 1.342 at the end of training, down from 1.466 at initialization.
CodeInLuau's originally-spec'd base was Qwen3.5-9B-4bit, but that architecture's hybrid linear-attention layers hit a confirmed, unresolved upstream bug in mlx-lm during LoRA's backward pass (Metal Insufficient Memory, reproduced independently across chip generations and model sizes -- see ml-explore/mlx-lm#1206). Switched to the previous-generation, pure-transformer Qwen3-8B-4bit, which trains cleanly.
Evaluation
A held-out base-vs-tuned comparison used 100 prompts sampled from luau_corpus's official test split (never touched during training), feeding each model only the prompt half and comparing the generated continuation.
An earlier 10-prompt hand-authored eval suggested the base model frequently broke out of code-completion mode into chat-style narration -- that result did not replicate at n=100 on real corpus data (0/100 for both models). What did show a large, reproducible difference was actual code correctness, checked with the real Luau parser (luau-analyze), separating genuine mid-body syntax errors from harmless truncation-at-the-token-limit artifacts:
| fully clean | truncation-only (forgivable) | real syntax error | |
|---|---|---|---|
| Base Qwen3-8B | 23 | 59 | 18 |
| CodeInLuau 1 Small | 76 | 21 | 2 |
Base's failures are qualitatively worse than the count alone suggests -- on several prompts it abandons Luau entirely (hallucinating C-style // comments, or generating SQL instead of Luau), while CodeInLuau's rare failures are small, plausible mistakes (a : where . belongs, a ... used outside a vararg-scoped closure) -- the kind of thing an actual Luau developer might slip on, not a model losing the plot.
Usage
from mlx_lm import load, generate
model, tokenizer = load("VertexAGI/codeinluau-1-small")
prompt = '''local Players = game:GetService("Players")
Players.PlayerAdded:Connect(function(player)
\tlocal leaderstats = Instance.new("Folder")
'''
response = generate(model, tokenizer, prompt=prompt, max_tokens=150)
print(prompt + response)
This is a completion model, not a chat model -- feed it a real code prefix, not a question.
Formats available
MLX only for this release (4-bit, via mlx-lm, for Apple Silicon). No GGUF yet -- the conversion path (dequantize to fp16, then quantize) needs disk headroom this machine didn't have available at release time; may follow later.
Limitations
An 8B-parameter model fine-tuned via LoRA on ~20K examples -- capable at common Roblox scripting patterns (services, events, leaderstats, tweens, debounce, module returns) but not infallible, and it has no awareness of a specific game's actual object hierarchy or custom APIs. Treat completions as a strong starting point to review, not code to ship unread.
License
Apache 2.0, inherited from the Qwen3 base model.