hy-embodied-mlx

A from-scratch Apple MLX implementation of the hunyuan_vl_mot architecture (tencent/HY-Embodied-0.5, a 3.79B mixture-of-transformers embodied VLM), plus weight conversion, a quantization ladder, and a pointing-accuracy eval harness.

The measured 4/5/6/8-bit and bf16 MLX weights are published on Hugging Face (links in the results table below, all variants in this collection); the 3-bit probe is measured but unpublished. Verification record: docs/phase0-manifest.md (architecture), docs/parity.md (MLX-vs-reference parity), docs/quant.md (sizes, throughput), docs/results.md (quality ladder).

Why

No quantization of HY-Embodied-0.5 exists in any format, and no public runtime executes its architecture outside the pinned reference transformers commit. This repo provides the runtime plus bf16 and 4/5/6/8-bit MLX weights with measured spatial-grounding quality per tier, and GGUF conversions once a runnable inference path exists for them. Numbers only get published if a script in this repo reproduces them.

Results

Where2Place pointing accuracy (official soft-mask protocol, greedy, fixed prompt template across tiers; Wilson 95% intervals, n=100 — statistical addendum with discordant pairs and McNemar tests in docs/results.md):

variant	weights	decode tok/s	no-think	think
bf16	7.05 GiB	66.8	0.696 [0.600, 0.778]	0.690 [0.593, 0.772]
8-bit	4.14 GiB	106.9	0.702 [0.607, 0.783]	0.658 [0.560, 0.743]
6-bit	3.36 GiB	124.3	0.690 [0.593, 0.772]	0.694 [0.598, 0.776]
5-bit	2.98 GiB	136.9	0.700 [0.605, 0.781]	0.677 [0.580, 0.761]
4-bit	2.59 GiB	157.6	0.662 [0.565, 0.747]	0.631 [0.534, 0.720]
3-bit (experimental)	2.20 GiB	171.2	0.614 [0.516, 0.704]	0.691 [0.595, 0.773]

Text-only probe, 50 scripted prompts (Wilson 95% intervals, n=50):

variant	no-think	think
bf16	0.700 [0.562, 0.809]	0.900 [0.786, 0.957]
8-bit	0.700 [0.562, 0.809]	0.880 [0.762, 0.944]
6-bit	0.700 [0.562, 0.809]	0.860 [0.738, 0.930]
5-bit	0.660 [0.522, 0.776]	0.840 [0.715, 0.917]
4-bit	0.660 [0.522, 0.776]	0.860 [0.738, 0.930]
3-bit (experimental)	0.400 [0.276, 0.538]	0.780 [0.648, 0.872]

Through 5-bit, nothing separates any variant from bf16 — all intervals overlap and every adjacent-tier McNemar test is non-significant. The first clear break is at 3-bit and it appears in the text path: no-think probe accuracy collapses from 0.70 to 0.40 with non-overlapping intervals, while the matching pointing drop stays within intervals and reaches only uncorrected nominal significance. On this evidence the low-bit failure is text-first; grounding degradation is directionally consistent but not separately established. Tencent's published 68.0 (thinking) comes from an unpublished harness and is cited for range only, not compared against.

Reproduce any cell:

python -m hy_embodied_mlx.convert --out mlx-bf16
python -m hy_embodied_mlx.quantize --model mlx-bf16 --out mlx-4bit --bits 4
python evals/where2place.py --model mlx-4bit --mode nothink --out out.csv

Oracle

The reference implementation requires flash-attn and cannot run on macOS as shipped. oracle/ contains a pure-torch flash-attn shim that makes it run on Apple Silicon (MPS, bf16), a state-dict manifest dumper, and the golden fixture generator used for parity testing. Setup:

python3.12 -m venv venv && source venv/bin/activate
pip install torch timm safetensors huggingface_hub pillow accelerate \
  "git+https://github.com/huggingface/transformers@9293856c419762ebf98fbe2bd9440f9ce7069f1a"
PYTHONPATH=oracle python oracle/generate_fixtures.py tests/fixtures mps

Licensing

Code in this repository is Apache-2.0. The model weights are Tencent's, under the Tencent HY Community License Agreement — not an open-source license; among other restrictions it excludes the territory of the EU, UK, and South Korea, and its Section 5 use restrictions pass through to derivatives. Any quantized weights produced by this tooling will carry the license copy, the required NOTICE text, and a modified-files statement. This project is not affiliated with, sponsored, or endorsed by Tencent.