Model card: add/refresh the Available formats table (generated from the repo's files)
6220256 verified | license: apache-2.0 | |
| language: | |
| - en | |
| base_model: mlx-community/Qwen3-8B-4bit | |
| tags: | |
| - mlx | |
| - lora | |
| - distillation | |
| - code | |
| - luau | |
| - roblox | |
| - text-generation | |
| pipeline_tag: text-generation | |
| vertex_family: codein | |
| vertex_params: "8B" | |
| vertex_purpose: "Luau code completion" | |
| vertex_result: "18% real syntax-error rate on base vs 2% tuned, on 100 held-out real-corpus prompts (luau-analyze)" | |
| vertex_status: released | |
| vertex_name: "CodeInLuau 1 Small" | |
| vertex_base: "Qwen3-8B" | |
| # CodeInLuau 1 Small | |
| **A Luau code-completion specialist, distilled from real Roblox source into Qwen3-8B** | |
| Part of the **CodeIn** family: where Aquamarine is a generalist across languages, CodeIn models go the other direction -- each one distilled for exactly one programming language, and nothing else. CodeInLuau is the first: Luau, Roblox's dialect of Lua. | |
| <!-- formats:start --> | |
| ## Available formats | |
| Everything below lives in this one repository: pick the format that matches your runtime. | |
| | Format | File / folder | Precision | Size | Works with | | |
| |---|---|---|---|---| | |
| | MLX | `model.safetensors` (repo root) | 4-bit (group size 64) | 4.61 GB | mlx-lm / MLX on Apple Silicon | | |
| A GGUF build is not provided for this model. | |
| <!-- formats:end --> | |
| ## Overview | |
| CodeInLuau 1 Small is fine-tuned via LoRA on **Qwen3-8B** for raw Luau code completion -- given a prefix (a partial script, a function signature, a comment), it continues the code, not a chat reply. It's trained on real, human-written Luau scraped from live Roblox experiences, not synthetic/generated code. | |
| ## Training | |
| - **Base model:** `mlx-community/Qwen3-8B-4bit` | |
| - **Dataset:** [`Roblox/luau_corpus`](https://huggingface.co/datasets/Roblox/luau_corpus) -- **real, human-written Luau, no synthetic generation.** 19,608 train / 4,352 validation examples, each a genuine prefix/continuation pair from real Roblox source, trained as plain text completion (no chat template -- this is a raw code-completion task, not instruction-following). | |
| - **Method:** LoRA fine-tuning (rank 8, scale 20, 16 layers), 3,000 iterations, batch size 1, sequence length 2048. | |
| - **Val loss:** 1.342 at the end of training, down from 1.466 at initialization. | |
| CodeInLuau's originally-spec'd base was `Qwen3.5-9B-4bit`, but that architecture's hybrid linear-attention layers hit a confirmed, unresolved upstream bug in `mlx-lm` during LoRA's backward pass (Metal `Insufficient Memory`, reproduced independently across chip generations and model sizes -- see [ml-explore/mlx-lm#1206](https://github.com/ml-explore/mlx-lm/issues/1206)). Switched to the previous-generation, pure-transformer `Qwen3-8B-4bit`, which trains cleanly. | |
| ## Evaluation | |
| A held-out base-vs-tuned comparison used 100 prompts sampled from `luau_corpus`'s **official test split** (never touched during training), feeding each model only the prompt half and comparing the generated continuation. | |
| An earlier 10-prompt hand-authored eval suggested the base model frequently broke out of code-completion mode into chat-style narration -- that result **did not replicate** at n=100 on real corpus data (0/100 for both models). What *did* show a large, reproducible difference was actual code correctness, checked with the real Luau parser (`luau-analyze`), separating genuine mid-body syntax errors from harmless truncation-at-the-token-limit artifacts: | |
| | | fully clean | truncation-only (forgivable) | real syntax error | | |
| |---|---|---|---| | |
| | **Base Qwen3-8B** | 23 | 59 | **18** | | |
| | **CodeInLuau 1 Small** | 76 | 21 | **2** | | |
| Base's failures are qualitatively worse than the count alone suggests -- on several prompts it abandons Luau entirely (hallucinating C-style `//` comments, or generating SQL instead of Luau), while CodeInLuau's rare failures are small, plausible mistakes (a `:` where `.` belongs, a `...` used outside a vararg-scoped closure) -- the kind of thing an actual Luau developer might slip on, not a model losing the plot. | |
| ## Usage | |
| ```python | |
| from mlx_lm import load, generate | |
| model, tokenizer = load("VertexAGI/codeinluau-1-small") | |
| prompt = '''local Players = game:GetService("Players") | |
| Players.PlayerAdded:Connect(function(player) | |
| \tlocal leaderstats = Instance.new("Folder") | |
| ''' | |
| response = generate(model, tokenizer, prompt=prompt, max_tokens=150) | |
| print(prompt + response) | |
| ``` | |
| This is a completion model, not a chat model -- feed it a real code prefix, not a question. | |
| ## Formats available | |
| MLX only for this release (4-bit, via `mlx-lm`, for Apple Silicon). No GGUF yet -- the conversion path (dequantize to fp16, then quantize) needs disk headroom this machine didn't have available at release time; may follow later. | |
| ## Limitations | |
| An 8B-parameter model fine-tuned via LoRA on ~20K examples -- capable at common Roblox scripting patterns (services, events, leaderstats, tweens, debounce, module returns) but not infallible, and it has no awareness of a specific game's actual object hierarchy or custom APIs. Treat completions as a strong starting point to review, not code to ship unread. | |
| ## License | |
| Apache 2.0, inherited from the Qwen3 base model. | |