pratikgx/TwIL-8B-MLX-4bit

TwIL-8B MLX 4-bit

A 4-bit MLX build of webAI-Official/TwIL-8B (balanced variant, revision 58d49f0d), for running on Apple Silicon with mlx-lm.

Non-commercial use only. This model is licensed under the webAI Non-Commercial License ver. 1.0 (full text in LICENSE.md, unmodified). It may be used for non-commercial research and educational purposes only. The base model it was post-trained from, ibm-granite/granite-4.2-8b, is Apache 2.0 (IBM); that license is included as apache-2.0-LICENSE.txt, as in the source repo.

Unofficial. This is an unofficial community conversion. It is not made, endorsed by, or affiliated with webAI. All credit for the model goes to webAI, and to IBM for the Granite base. For the original weights, model card, evals and GGUF builds, see the source model.

Build details

• Quantization: 4-bit affine, group size 64, applied to every quantizable layer (including o_proj, down_proj and lm_head). 4.50 bits per weight, about 4.95 GB (single model.safetensors, 4,945,853,960 bytes).
• Note: stock mlx_lm.convert -q on Granite keeps some layers at 8-bit. This build forces them all to 4-bit.
• Converted with mlx 0.32.4 and mlx-lm 0.32.3.dev12+g4cfc64784 (Python 3.12.15).
• Config tweak: transformers 5 writes rope_parameters, and mlx-lm's Granite loader needs a top-level rope_theta. The conversion used a copy of the source config.json with rope_theta: 10000000 added. No weights were changed.
• Tokenizer, chat template and generation config come from the source repo. The chat template opens a <think> block by default (reasoning model).

Exact convert command (run from the local model folder, where hf-bf16-mlxcfg is the source download with the config tweak above):

hf download webAI-Official/TwIL-8B --revision 58d49f0d9909e62f9c2fb09e99fc5c76f3f2d630 --exclude "verbose/*" --exclude "*.gguf" --exclude "*.png" --local-dir twil-8b/hf-bf16
python -c "from mlx_lm.convert import convert; convert('hf-bf16-mlxcfg', mlx_path='mlx-4bit-plain', quantize=True, q_group_size=64, q_bits=4, quant_predicate=lambda p, m: True)"

Usage

pip install mlx-lm
mlx_lm.generate --model pratikgx/TwIL-8B-MLX-4bit --prompt "What is the sum of the first 50 odd numbers?" --max-tokens 2048

webAI's card suggests temp 1.0 / top_p 0.95 (these are in generation_config.json). The model thinks before answering, so give it room with --max-tokens.

Measured speed (Apple M5 Pro Mac mini, 64GB)

mlx-lm, temp 0, thinking on, 256 generated tokens, median of 3 timed runs after 1 warmup.

Context	Prompt tok/s	Gen tok/s	Peak memory (GB)
512	1,297.9	53.76	5.57
4,096	1,338.3	48.84	6.38
16,384	1,155.2	35.96	8.24
32,768	904.7	26.57	11.01
65,536	631.8	17.33	16.58
131,072	380.5	9.16	27.85

The full 128K context fits easily on a 64GB Mac. The 128K run had a 16.5% spread across runs and a background inference app was running during the benchmarks, so treat the long-context numbers as a little noisy.

Quality checks on this build (small local suites, greedy, thinking on, not comparable to webAI's published evals): math 20/20, code unit tests 10/10, instruction following 10/10, needles at 32K/64K 8/8, harder math 8/8, harder code 5/5, tool calling 5/5. Same scores as the bf16 build.

Full results, commands, logs and caveats: pratikgx/twil-8b-m5-pro-mac-mini-bench.

License

webAI Non-Commercial License ver. 1.0, see LICENSE.md. Non-commercial research and educational use only. Base model Granite 4.2 8B is Apache 2.0, see apache-2.0-LICENSE.txt.