--- license: apache-2.0 language: - en - zh library_name: mlx pipeline_tag: text-generation base_model: sensenova/SenseNova-Loop-9B tags: - mlx - SenseNova - quantized - text-generation - custom-runtime --- # SenseNova-Loop-9B — Unofficial MLX 4-bit [简体中文说明](README.zh-CN.md) This is an **unofficial, text-only MLX conversion** of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B), quantized to 4-bit weights. It is not an official SenseNova release. ## Important runtime requirement SenseNova-Loop-9B uses a custom looped, dual-stream mHC architecture. **Stock mlx-lm does not support this architecture.** These weights require an MLX/oMLX runtime with a compatible `SenseNova_loop` implementation. The customized implementation used in our tests is included as `mlx_sensenova_loop.py` for integration and reference. Merely downloading this repository into a stock mlx-lm installation is not sufficient. The conversion contains text-generation weights only; vision inputs are not supported by this package. ## Quantization - Tool: `mlx_lm.convert` - Method: MLX affine weight quantization - Bits: **4** - Group size: **64** - Effective size reported by the converter: **4.511 bits per weight**, including quantization metadata - Source: BF16 Hugging Face weights - Custom runtime flags used in testing: `SENSENOVA_MHC_METAL=1` and `SENSENOVA_MHC_MERGE_METAL=1` ## Recommended generation settings The source repository's default generation configuration is `temperature=1.0`, `top_p=0.95`, with sampling enabled. In our Chinese long-form tests, that setting could produce irrelevant foreign words in **all** tested precisions, including unquantized BF16. For formal Chinese long-form generation, we recommend: ```json { "temperature": 0, "top_p": 1 } ``` This recommendation targets language artifacts; it does not guarantee factual correctness. Reasoning may also be in English while the final answer is entirely Chinese. ## Speed on the test machine Test environment: Apple Silicon Mac with 128 GB unified memory, oMLX 0.7.3, customized mlx-lm, MLX 0.32.4. Each result is the median of three OpenAI-compatible requests generating 64 tokens. Results depend on hardware, prompt, runtime, and workload. | Variant | Generation speed | | --- | ---: | | BF16 baseline | 8.50 token/s | | 8-bit | 13.51 token/s | | **4-bit** | **20.90 token/s** | For this 4-bit variant, enabling compiled mHC router subgraphs increased speed from 16.44 to 20.90 token/s (about **27.1%** faster). The compilation optimization passed 32-token output and logit-equivalence checks. That equivalence applies to compilation, not to quantization. ## Quantization quality checks Evaluation covered 614 token positions across Chinese, English, Python, and math text. Lower perplexity is better; top-1 agreement is measured against BF16. | Variant | Perplexity | Top-1 agreement with BF16 | | --- | ---: | ---: | | BF16 | 2.7127 | reference | | 8-bit | 2.6938 | 99.02% | | **4-bit** | **2.7875** | **93.81%** | Small differences in perplexity can arise from evaluation variation; the table should not be interpreted as proof that 8-bit improves the base model. ## Chinese long-form language tests ### Official default sampling (`temperature=1.0`, `top_p=0.95`) Three fixed seeds were run for each precision. All nine essays finished naturally. Every matched foreign word below was manually reviewed and was inappropriate in context. | Variant | Foreign words across 3 essays | Foreign words per 1,000 Chinese characters | | --- | ---: | ---: | | BF16 | 7 | 1.21 | | 8-bit | 23 | 3.32 | | **4-bit** | **18** | **2.76** | This small sample shows that the artifact exists in the base model too. Quantization may aggravate it, but the results do **not** show a monotonic relationship with quantization depth. ### Deterministic decoding (`temperature=0`, `top_p=1`) This 4-bit variant was tested on five different approximately 2,000-character Chinese tasks: two book reflections, one argumentative essay, one literary essay, and one popular-science article. - 5/5 articles finished naturally. - Total output: 12,109 characters, including 10,034 Chinese characters. - No meaningless foreign words, scripts, or Unicode replacement characters were found. - Contextually appropriate terms such as `ChatGPT`, `AI`, and `REM` were not counted as errors. Five tasks do not prove that artifacts are impossible under all prompts. ## Optional MTP speculative draft This repository includes `model-mtp.safetensors`, an optional MLX BF16 conversion of the official MTP draft weights. The main model remains 4-bit; the draft is deliberately kept in BF16 because it is a small file and quantizing it did not address the measured bottleneck. MTP is **disabled by default** and requires `SENSENOVA_MTP=1` with the customized runtime. The included `mlx_mtp_speculative.py` provides the tested single-request runner. The current oMLX generic MTP toggle does not recognize this custom architecture. Use depth 2. Deeper drafts did not increase useful acceptance and became slower; depths above 3 are not recommended. ```bash SENSENOVA_MTP=1 python mlx_mtp_speculative.py \ --model /path/to/SenseNova-Loop-9B-MLX-4bit \ --prompt "Your prompt" --depth 2 ``` Add `--baseline` to run the same prompt without speculative decoding. Three-run medians for greedy code generation with a 4-bit target: | Task | Normal decoding | MTP depth 2 | Output match | | --- | ---: | ---: | --- | | LRU cache implementation | 36.00 token/s | **39.43 token/s (about 9.6% faster)** | exact in 3/3 runs | | Merge intervals | 33.34 token/s | 31.88 token/s (4.4% slower) | exact in 3/3 runs | Long Chinese prose and several other tasks were slower with MTP. This is an optional, workload-dependent code-generation experiment, not a general speed setting. ## Limitations - Quantization is not lossless, especially at 4-bit. - Deterministic decoding does not prevent factual errors, fabricated quotations, or invented personal experiences. Review factual content before use. - The language tests targeted foreign-token artifacts, not overall writing quality or factual accuracy. - A customized runtime is required; stock mlx-lm is insufficient. ## Download ```bash hf download NeosQ/SenseNova-Loop-9B-MLX-4bit \ --local-dir SenseNova-Loop-9B-MLX-4bit ``` ## License and attribution Apache-2.0. Please also follow the base model's license and terms. All quantization and testing described here are unofficial community work.