Publish Humanizer family draft and verified MLX comparison

7931fb2 verified 11 days ago

1.85 kB

# MLX runtime setup

Apple silicon. The measurement used Python3.12, mlx0.32.0, mlx-lm0.31.3, mlx-vlm0.6.5 and transformers5.14.1. This is a patched research runtime, not stock Hugging Face Transformers inference. The model is a draft, not a standalone chat model.

```bash
uv venv --python 3.12 .venv-humanizer
source .venv-humanizer/bin/activate
uv pip install mlx==0.32.0 mlx-lm==0.31.3 mlx-vlm==0.6.5 transformers==5.14.1 huggingface-hub==1.19.0 safetensors==0.8.0
hf download jayyun98/Humanizer-12B-Draftfit-DFlash2 mlx-runtime.patch run_mlx.py --local-dir ./humanizer-draft
git clone https://github.com/ARahim3/mlx-dspark.git ./mlx-dspark-humanizer
git -C ./mlx-dspark-humanizer checkout 414309238e0f0f2a97315bcf0157e9f20fffebbc
git -C ./mlx-dspark-humanizer apply ../humanizer-draft/mlx-runtime.patch
uv pip install --no-deps -e ./mlx-dspark-humanizer
python ./humanizer-draft/run_mlx.py --text "The meeting has been moved to Friday." --check
```

The dependency versions preserve the measured Gemma4 runtime, even though the runtime project's newer minimum covers unrelated targets. The measurement and runner were tested in the existing environment; this fresh-install recipe has not been rerun in a clean environment.

The runner downloads the pinned mixed4/8bit Humanizer target (~9GB), then the draft. It uses completion formatting, FP32 floating tensors,4bit draft quantization, greedy generation, cap3 and no lookup drafting. `--check` also runs target-only and fails on any token mismatch. The default64 output-token cap is a short benchmark limit; increase `--max-tokens` for longer rewrites. Longer outputs and sampling were not part of this benchmark.

## vLLM and SGLang

The exported checkpoint has not been verified with either CUDA serving engine. No launch command is provided as a working claim. Use the tested MLX runtime above.

	# MLX runtime setup

	Apple silicon. The measurement used Python3.12, mlx0.32.0, mlx-lm0.31.3, mlx-vlm0.6.5 and transformers5.14.1. This is a patched research runtime, not stock Hugging Face Transformers inference. The model is a draft, not a standalone chat model.

	```bash
	uv venv --python 3.12 .venv-humanizer
	source .venv-humanizer/bin/activate
	uv pip install mlx==0.32.0 mlx-lm==0.31.3 mlx-vlm==0.6.5 transformers==5.14.1 huggingface-hub==1.19.0 safetensors==0.8.0
	hf download jayyun98/Humanizer-12B-Draftfit-DFlash2 mlx-runtime.patch run_mlx.py --local-dir ./humanizer-draft
	git clone https://github.com/ARahim3/mlx-dspark.git ./mlx-dspark-humanizer
	git -C ./mlx-dspark-humanizer checkout 414309238e0f0f2a97315bcf0157e9f20fffebbc
	git -C ./mlx-dspark-humanizer apply ../humanizer-draft/mlx-runtime.patch
	uv pip install --no-deps -e ./mlx-dspark-humanizer
	python ./humanizer-draft/run_mlx.py --text "The meeting has been moved to Friday." --check
	```

	The dependency versions preserve the measured Gemma4 runtime, even though the runtime project's newer minimum covers unrelated targets. The measurement and runner were tested in the existing environment; this fresh-install recipe has not been rerun in a clean environment.

	The runner downloads the pinned mixed4/8bit Humanizer target (~9GB), then the draft. It uses completion formatting, FP32 floating tensors,4bit draft quantization, greedy generation, cap3 and no lookup drafting. `--check` also runs target-only and fails on any token mismatch. The default64 output-token cap is a short benchmark limit; increase `--max-tokens` for longer rewrites. Longer outputs and sampling were not part of this benchmark.

	## vLLM and SGLang

	The exported checkpoint has not been verified with either CUDA serving engine. No launch command is provided as a working claim. Use the tested MLX runtime above.