hermitdave/Hy-Memory-30B-A3B-MLX-8bit

Hy-Memory-30B-A3B — MLX 8-bit

8-bit MLX conversion of tencent/Hy-Memory-30B-A3B, the memory-writer model from Tencent Hunyuan.

Hy-Memory reads a conversation and decides what is worth remembering — names, preferences, constraints, recurring facts — returning them as JSON for an external memory framework (Mem0, MemOS, Hy-Memory). It is a 30B MoE with ~3B active per token.

8.0 bits per weight · 30 GB · effectively lossless

Which repo to use

hy_v3 is not in MLX-LM upstream, and the two runtimes that can run it bring different adapters with different expert key names — so a checkpoint loads in exactly one of them.

	This repo	-omlx variant
Expert keys	mlp.experts.*	mlp.switch_mlp.*
Adapter used	this repo's hy_v3.py	oMLX's vendored hy_v3
mlx_lm (generate, server, LoRA)	✅	❌
oMLX	❌	✅

Use this repo for MLX-LM. Use the -omlx repo for oMLX. Picking the wrong one fails loudly and immediately — each runtime rejects the other's layout with the same Received 423 parameters not in model: error.

What this is for

This is the quality reference. At 8-bit the model is as close to the bf16 original as makes no practical difference, so it is the right choice when you want maximum extraction fidelity and have the memory to spare — and the right baseline for measuring a smaller quant against.

If you want the same behaviour in 57% of the disk and RAM, use hermitdave/Hy-Memory-30B-A3B-MLX-Mixed4-6 instead: it measured +0.94% perplexity against this model and scored identically (100%) on our extraction eval.

Benchmarks

Measured on an M3 Max (64 GB), wikitext-2, 8 × 256-token deterministic windows:

	8-bit	Mixed 4/6
Size	30 GB	17 GB
Perplexity	14.783	14.923
Extraction recall (15 cases / 70 facts)	100%	100%
Load time	7.0 s	3.3 s
Decode	~51 tok/s	~65 tok/s

Quickstart

pip install -U mlx-lm

from mlx_lm import load, stream_generate

model, tokenizer = load("hermitdave/Hy-Memory-30B-A3B-MLX-8bit")

messages = [
    {
        "role": "system",
        "content": (
            "Extract explicit user facts and preferences from the conversation. "
            "Write each memory as a concise, self-contained statement. "
            "Do not infer or invent information. "
            'Return only a JSON object with a "memories" list of strings.'
        ),
    },
    {
        "role": "user",
        "content": (
            "My name is Alex. I live in Seattle and work as a software engineer. "
            "I mainly use Python and prefer concise answers with code examples. "
            "I'm vegetarian, so please keep that in mind when suggesting meals."
        ),
    },
]

prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=False)
for r in stream_generate(model, tokenizer, prompt=prompt, max_tokens=512):
    print(r.text, end="", flush=True)

Or serve it OpenAI-compatible:

mlx_lm.server --model hermitdave/Hy-Memory-30B-A3B-MLX-8bit --port 8080

Recommended: temperature=0.0 for extraction.

Quantization

Uniform 8-bit, group size 64, affine mode. Norms and the router's expert_bias stay in their original precision; every projection and embedding is quantized.

Converted layer-by-layer from the bf16 source — a 56 GB bf16 checkpoint does not fit alongside a quantized copy on a 64 GB machine, so a streaming converter is used rather than mlx_lm convert.

Notes

• Custom architecture. hy_v3 (HYV3ForCausalLM) is not in MLX-LM, so hy_v3.py ships with this repo. MLX-LM finds it through the "model_file": "hy_v3.py" key in config.json — the only hook MLX-LM offers for a custom model class. Note that MLX-LM's --trust-remote-code flag is tokenizer-only and does not load a custom model class; without model_file, MLX-LM raises ValueError: Model type hy_v3 not supported. and never opens hy_v3.py at all.
• Two config corrections were needed: upstream moe_intermediate_size: 1536 disagrees with the checkpoint's own expert weights (all 768-wide), and rope_theta moved into a nested rope_parameters block. Both are fixed here.
• This model is a writer, not a retriever or answerer. Pair it with a general-purpose LLM; it does not replace one.

Attribution

• Base model: tencent/Hy-Memory-30B-A3B by Tencent Hunyuan, Apache-2.0.
• hy_v3 MLX adapter: originally authored by QwQbb for the Hy-MT2 conversion; reused here with a stop_gradient fix on the router indices.
• Conversion and benchmarking: Hermes Agent using mlx-lm 0.31.3.
• Paper: arXiv:2610.22196

Community conversion, not an official Tencent release.

Citation

@misc{hy-memory-30b-a3b,
  title  = {Hy-Memory-30B-A3B},
  author = {Tencent Hunyuan},
  year   = {2026},
  eprint = {2610.22196},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/pdf/2610.22196},
}

License

Apache-2.0, same as upstream.