runtime-setup.md

MLX runtime setup

Apple silicon. The measurement used Python3.12, mlx0.32.0, mlx-lm0.31.3, mlx-vlm0.6.5 and transformers5.14.1. This is a patched research runtime, not stock Hugging Face Transformers inference. The model is a draft, not a standalone chat model.

uv venv --python 3.12 .venv-humanizer
source .venv-humanizer/bin/activate
uv pip install mlx==0.32.0 mlx-lm==0.31.3 mlx-vlm==0.6.5 transformers==5.14.1 huggingface-hub==1.19.0 safetensors==0.8.0
hf download jayyun98/Humanizer-12B-Draftfit-DFlash mlx-runtime.patch run_mlx.py --local-dir ./humanizer-draft
git clone https://github.com/ARahim3/mlx-dspark.git ./mlx-dspark-humanizer
git -C ./mlx-dspark-humanizer checkout 414309238e0f0f2a97315bcf0157e9f20fffebbc
git -C ./mlx-dspark-humanizer apply ../humanizer-draft/mlx-runtime.patch
uv pip install --no-deps -e ./mlx-dspark-humanizer
python ./humanizer-draft/run_mlx.py --text "The meeting has been moved to Friday." --check

The dependency versions preserve the measured Gemma4 runtime, even though the runtime project's newer minimum covers unrelated targets. The measurement and runner were tested in the existing environment; this fresh-install recipe has not been rerun in a clean environment.

The runner downloads the pinned mixed4/8bit Humanizer target (~9GB), then the draft. It uses completion formatting, FP32 floating tensors,4bit draft quantization, greedy generation, cap3 and no lookup drafting. --check also runs target-only and fails on any token mismatch. The default64 output-token cap is a short benchmark limit; increase --max-tokens for longer rewrites. Longer outputs and sampling were not part of this benchmark.

vLLM and SGLang

The exported checkpoint has not been verified with either CUDA serving engine. No launch command is provided as a working claim. Use the tested MLX runtime above.