engineering-group

/

EngGPT2-16B-A3B

like 17

Follow

Engineering Group 32

Hi,

first of all, thank you for releasing EngGPT2-16B-A3B.

I have been experimenting with the model on Apple Silicon and created a set of community MLX conversions for local inference with mlx-lm.

The converted models are available here:

• Unquantized MLX version:
  https://huggingface.co/robertobissanti/EngGPT2-16B-A3B-MLX

• 4-bit MLX version:
  https://huggingface.co/robertobissanti/EngGPT2-16B-A3B-MLX-4bit

• 8-bit MLX version:
  https://huggingface.co/robertobissanti/EngGPT2-16B-A3B-MLX-8bit

Since enggpt_moe is currently a custom architecture not supported by upstream mlx-lm, I also created a patched fork with preliminary support for this architecture:

https://github.com/robertobissanti/mlx-lm-enggpt

The implementation adds support for the main architectural features I found in the model:

• model_type = enggpt_moe
• head_dim = 128
• Q/K RMSNorm inside attention
• GQA with 32 attention heads and 4 KV heads
• MoE layers with 64 experts
• top-8 routing
• SwiGLU experts
• routing logic closer to the original PyTorch implementation: softmax over all experts, top-k selection, and optional top-k probability renormalization

I tested the conversions locally on a Mac Studio M1 Ultra with 64 GB unified memory.

Approximate local results:

The 4-bit version appears to be the most practical one for local Apple Silicon inference.

A typical command is:

python -m mlx_lm generate \
  --model ./EngGPT2-16B-A3B-MLX-4bit \
  --prompt "Explain briefly what a Mixture of Experts model is." \
  --trust-remote-code \
  --chat-template-config '{"enable_thinking": false}' \
  --temp 0.1 \
  --max-tokens 160

The model can also be served through the mlx-lm OpenAI-compatible local server:

python -m mlx_lm server \
  --model ./EngGPT2-16B-A3B-MLX-4bit \
  --host 127.0.0.1 \
  --port 8080 \
  --trust-remote-code \
  --chat-template-args '{"enable_thinking": false}' \
  --temp 0.1 \
  --top-p 1.0 \
  --max-tokens 512

I would be very happy if you could take a look and let me know whether the conversion and the architecture mapping look correct from your side.

In particular, feedback would be very useful on:

• whether the MoE routing logic matches your intended implementation;
• whether Q/K normalization and head_dim = 128 are handled correctly;
• whether there are any special inference details that should be documented;
• whether the model cards should include additional license or usage notes.

This is an experimental community conversion, and I have clearly documented that it currently requires a patched mlx-lm fork until upstream support for enggpt_moe is available.

Thanks again for the model and for any feedback you may be willing to provide.

Roberto Bissanti