SenseNova-Loop-9B — Unofficial MLX 8-bit

简体中文说明

This is an unofficial, text-only MLX conversion of sensenova/SenseNova-Loop-9B, quantized to 8-bit weights. It is not an official SenseNova release.

Important runtime requirement

SenseNova-Loop-9B uses a custom looped, dual-stream mHC architecture. Stock mlx-lm does not support this architecture. These weights require an MLX/oMLX runtime with a compatible SenseNova_loop implementation.

The customized implementation used in our tests is included as mlx_sensenova_loop.py for integration and reference. Merely downloading this repository into a stock mlx-lm installation is not sufficient.

The conversion contains text-generation weights only; vision inputs are not supported by this package.

Quantization

  • Tool: mlx_lm.convert
  • Method: MLX affine weight quantization
  • Bits: 8
  • Group size: 64
  • Source: BF16 Hugging Face weights
  • Custom runtime flags used in testing: SENSENOVA_MHC_METAL=1 and SENSENOVA_MHC_MERGE_METAL=1

Recommended generation settings

The source repository's default generation configuration is temperature=1.0, top_p=0.95, with sampling enabled. In our Chinese long-form tests, that setting could produce irrelevant foreign words in all tested precisions, including unquantized BF16.

For formal Chinese long-form generation, we recommend:

{
  "temperature": 0,
  "top_p": 1
}

This recommendation targets language artifacts; it does not guarantee factual correctness. Reasoning may also be in English while the final answer is entirely Chinese.

Speed on the test machine

Test environment: Apple Silicon Mac with 128 GB unified memory, oMLX 0.7.3, customized mlx-lm, MLX 0.32.4. Each result is the median of three OpenAI-compatible requests generating 64 tokens. Results depend on hardware, prompt, runtime, and workload.

Variant Generation speed
BF16 baseline 8.50 token/s
8-bit 13.51 token/s
4-bit 20.90 token/s

For the 4-bit variant, enabling compiled mHC router subgraphs increased speed from 16.44 to 20.90 token/s (about 27.1% faster). The compilation optimization passed 32-token output and logit-equivalence checks. That equivalence applies to compilation, not to quantization.

Quantization quality checks

Evaluation covered 614 token positions across Chinese, English, Python, and math text. Lower perplexity is better; top-1 agreement is measured against BF16.

Variant Perplexity Top-1 agreement with BF16
BF16 2.7127 reference
8-bit 2.6938 99.02%
4-bit 2.7875 93.81%

Small differences in perplexity can arise from evaluation variation; the table should not be interpreted as proof that 8-bit improves the base model.

Chinese long-form language tests

Official default sampling (temperature=1.0, top_p=0.95)

Three fixed seeds were run for each precision. All nine essays finished naturally. Every matched foreign word below was manually reviewed and was inappropriate in context.

Variant Foreign words across 3 essays Foreign words per 1,000 Chinese characters
BF16 7 1.21
8-bit 23 3.32
4-bit 18 2.76

This small sample shows that the artifact exists in the base model too. Quantization may aggravate it, but the results do not show a monotonic relationship with quantization depth.

Deterministic decoding (temperature=0, top_p=1)

The 4-bit variant was tested on five different approximately 2,000-character Chinese tasks: two book reflections, one argumentative essay, one literary essay, and one popular-science article.

  • 5/5 articles finished naturally.
  • Total output: 12,109 characters, including 10,034 Chinese characters.
  • No meaningless foreign words, scripts, or Unicode replacement characters were found.
  • Contextually appropriate terms such as ChatGPT, AI, and REM were not counted as errors.

Five tasks do not prove that artifacts are impossible under all prompts.

Optional MTP speculative draft

This repository includes model-mtp.safetensors, an optional MLX BF16 conversion of the official MTP draft weights. The main model remains 8-bit; the draft is deliberately kept in BF16 because it is a small file and quantizing it did not address the measured bottleneck.

MTP is disabled by default and requires SENSENOVA_MTP=1 with the customized runtime. The included mlx_mtp_speculative.py provides the tested single-request runner. The current oMLX generic MTP toggle does not recognize this custom architecture.

Use depth 2. Deeper drafts did not increase useful acceptance and became slower; depths above 3 are not recommended.

SENSENOVA_MTP=1 python mlx_mtp_speculative.py \
  --model /path/to/SenseNova-Loop-9B-MLX-8bit \
  --prompt "Your prompt" --depth 2

Add --baseline to run the same prompt without speculative decoding.

MTP speed was measured against the 4-bit target, where one code task improved by about 9.6% while another was slower. The draft behavior may differ with the 8-bit target and has not been separately benchmarked here. Treat MTP as an optional, workload-dependent experiment, not a general speed setting.

Limitations

  • Quantization is not lossless, especially at 4-bit.
  • Deterministic decoding does not prevent factual errors, fabricated quotations, or invented personal experiences. Review factual content before use.
  • The language tests targeted foreign-token artifacts, not overall writing quality or factual accuracy.
  • A customized runtime is required; stock mlx-lm is insufficient.

Download

hf download NeosQ/SenseNova-Loop-9B-MLX-8bit \
  --local-dir SenseNova-Loop-9B-MLX-8bit

License and attribution

Apache-2.0. Please also follow the base model's license and terms. All quantization and testing described here are unofficial community work.

Downloads last month
82
Safetensors
Model size
9B params
Tensor type
U32
·
F32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NeosQ/SenseNova-Loop-9B-MLX-8bit

Quantized
(49)
this model