bumblebuttpow/Luciole-23B-Instruct-1.1-4bit-mlx

Luciole-23B-Instruct-1.1 — MLX 4-bit

MLX (Apple Silicon) 4-bit conversion of OpenLLM-France/Luciole-23B-Instruct-1.1 (LINAGORA / OpenLLM-France, NVIDIA Nemotron architecture, 128K context, Apache 2.0).

Provenance — read this

This quant was converted from the Q4_K_M GGUF, not from the original bf16 weights: GGUF K-quants were dequantized to fp16 and re-quantized to MLX 4-bit affine (group size 64) with a streaming tensor-by-tensor converter. It is therefore a double-quantized artifact — quality is a hair below a quant made from the bf16 originals. If you need maximum fidelity, quantize from the original safetensors instead. For everyday use on 24–36GB Macs this version is fine in practice.

Non-quantized tensors (embeddings, LayerNorms) are fp16. The Nemotron LayerNorm-1P convention is preserved (stored gamma = true gamma − 1; mlx-lm's nemotron.py adds the 1 back at runtime).

Use

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("bumblebuttpow/Luciole-23B-Instruct-1.1-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Or serve an OpenAI-compatible API:

mlx_lm.server --model bumblebuttpow/Luciole-23B-Instruct-1.1-4bit-mlx --port 8080

~13 GB on disk. The authors recommend temp 0.7 / top_p 0.9.

Model details

Arch	nemotron (dense, relu² MLP, partial rotary 0.5, LayerNorm-1P)
Layers	40, hidden 6144, 48 Q / 8 KV heads (head_dim 128)
Vocab / ctx	128000 / 131072
Quant	4-bit affine, group size 64 (embeddings + norms fp16)
Base	OpenLLM-France/Luciole-23B-Instruct-1.1

Conversion script: streaming GGUF→MLX, peak RAM ~1 tensor (contact: see commit history / repo discussions). Verified coherent end-to-end on Apple Silicon (M4 Pro + M1 pipeline-parallel over Thunderbolt) before upload.