Sapling-505M-Midtrain-MLX-4bit

An 4-bit MLX build of Sapling-505M-Midtrain: the annealed mid-training checkpoint of a 505M-parameter Qwen3-architecture model trained from scratch (5B pretraining + 1B tokens of math, chat and replay), for Apple Silicon Macs (304 MB).

Not a chat model. It continues text, and it has seen ChatML, but it does not end its turns. Use it as a base for your own fine-tuning, or as a text and math completer. For chat, see Sapling-505M-SFT.

Noticeably lossy at this model size. Perplexity rises about 6–7%, and short factual answers change more often. For example, it answered "Sydney" where the original says "Canberra". Prefer the 8-bit version (545 MB) unless memory is tight.

Usage

pip install mlx-lm

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("VinayRevanuru/Sapling-505M-Midtrain-MLX-4bit")
print(generate(model, tok, prompt="Problem: What is 15% of 240?\nSolution:", max_tokens=128,
               sampler=make_sampler(temp=0.7, top_p=0.9)))

This model does not stop on its own. Always set max_tokens.

Quality

Variant	Size	WikiText-2 PPL	vs FP32
FP32 original (transformers)	2.0 GB	22.86	—
MLX 4-bit (this repo)	304 MB	24.44	+6.9%
MLX 8-bit	545 MB	22.88	+0.1%

Perplexity was measured on the first 64 × 1,024-token windows of the WikiText-2 test set (raw), scoring the second half of each window, as llama-perplexity does. The same token windows and the same method were used for transformers, MLX and llama.cpp, so the numbers are directly comparable. The GGUF BF16 file reproduces the FP32 model's perplexity to within 0.01%, which confirms the conversion and the tokenizer are exact.

How this was made

mlx_lm.convert -q --q-bits 4 --q-group-size 64 (mlx-lm 0.32.2, affine quantization, 5.0 effective bits per weight) from the original FP32 safetensors. Architecture: Qwen3 dense, 30 layers, width 1,024, 16 query and 8 KV heads, tied embeddings, 2,048-token context. License: Apache-2.0, the same as the original.

Other formats: GGUF (llama.cpp / Ollama), original FP32.