Qwimi-3.6-27B-Coder-MTP-MLX-8bit

A 8-bit MLX quantization of trjxter/Qwimi-3.6-27B-Coder-MTP-BF16, a coding-focused SFT of Qwen 3.6 27B.

The model was trained in one mixed SFT run across coding, Qwen-native XML-style tool calling, and agentic software-engineering trajectories. This repository is intended for local inference on Apple silicon using mlx-lm.

This is an MLX repository, not GGUF. It is not intended for llama.cpp, CUDA inference, or a GGUF-only Ollama workflow.

Quant position: highest-fidelity quantized MLX release.
Primary tradeoff: smallest quantization loss of the Q4/Q6/Q8 set.
Recommendation: Best when quality is the priority and the model fits comfortably.

---

Contents

1. Source and quantization
2. Which MLX quant should I use?
3. Unified-memory guidance
4. Installation
5. How to run
6. Tool calling
7. Training summary
8. Benchmarks
9. Known limitations
10. Verification and reproducibility

---

1. Source and quantization

• Merged source: trjxter/Qwimi-3.6-27B-Coder-MTP-BF16
• Base architecture: Qwen 3.6 27B, approximately 27.8B parameters
• Architecture family: dense hybrid transformer with standard attention and GatedDeltaNet layers
• Specialization: text-only coding SFT; vision tower frozen
• Format: MLX model weights for Apple silicon
• Nominal weight precision: 8-bit
• Expected conversion flag: --q-bits 8
• Validated SFT sequence length: 16,384 tokens

The uploaded repository's config.json is the source of truth for quantization mode, group size, and tensor metadata. This README intentionally does not invent a group size that may differ from the actual conversion.

The source name includes MTP. Do not assume that conversion to MLX automatically preserves or activates the same MTP runtime path used by a specific llama.cpp build.

---

2. Which MLX quant should I use?

Choose Q8 when you want the closest of these three quantized releases to BF16 behavior and your Mac has enough unified memory.

Prefer Q6 for a better size/quality balance or Q4 when memory and context headroom dominate.

MLX release	Relative memory use	Expected fidelity	Best fit
Q4	Lowest	Good, with the most quantization loss	Constrained memory and more context headroom
Q6	Medium	Strong balance	General recommendation
Q8	Highest	Highest of the three	Maximum quality on larger-memory Macs

Exact quality and memory behavior should be measured with the uploaded artifacts and your own workload.

---

3. Unified-memory guidance

• Practical target: 64 GB unified memory or more.
• 48 GB Macs: may work with controlled context and limited competing memory use.
• 96 GB+ Macs: provide generous headroom for long prompts, cache, IDEs, and agents.

These are rough planning targets, not guarantees. Actual memory use depends on the exact file size, group size, prompt length, generation length, cache configuration, macOS version, and other applications using unified memory.

---

4. Installation

MLX requires Apple silicon.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip mlx-lm

Qwen 3.6 / qwen3_5 support is relatively new, so update mlx-lm before diagnosing model, cache, server, or tool-parser problems.

---

5. How to run

One-shot generation

mlx_lm.generate \
  --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  --prompt "Write a Python function that reverses a linked list and include tests."

Interactive chat

mlx_lm.chat --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit

Python API

from mlx_lm import generate, load

MODEL_ID = "trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit"
model, tokenizer = load(MODEL_ID)

messages = [
    {
        "role": "user",
        "content": "Implement an LRU cache in Python and explain the complexity.",
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=1024,
    verbose=True,
)

print(response)

Local OpenAI-style server

mlx_lm.server \
  --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  --host 127.0.0.1 \
  --port 8080

mlx_lm.server is most appropriate as a local development endpoint, not a hardened public production service.

Context guidance

The base architecture may expose a larger native context, but Qwimi was trained and validated at 16,384 tokens. Longer-context behavior should be tested separately.

---

6. Tool calling

Qwimi was trained on Qwen 3.6's native XML-style tool-call format rather than the older plain JSON-object style.

<tool_call>
<function=search_web>
<parameter=query>latest MLX documentation</parameter>
</function>
</tool_call>

Recommendations:

1. Apply the repository's tokenizer chat template.
2. Test the exact mlx-lm version you plan to deploy.
3. Validate multi-turn tool calls, inserted tool results, and stop behavior.
4. Do not assume that working text generation guarantees correct server-side tool parsing.
5. Keep direct XML validation or a fallback parser for agent workflows.

---

7. Training summary

• Base: unsloth/Qwen3.6-27B
• Model size: approximately 27.8B parameters
• Method: 4-bit QLoRA during SFT, rank 64, alpha 64
• Trainable parameters: 466,911,232, approximately 1.68%
• Training style: one mixed SFT run, not three sequential stages
• Total training rows: 22,359
• Training tokens: approximately 72.5M
• Maximum sequence length: 16,384
• Effective batch size: 16
• Epochs: 1
• Optimizer steps: 1,398
• Hardware: 1× A100-SXM4-80GB
• Vision tower: frozen
• Final eval loss: coding 0.4761; tool-calling 0.0208; agentic 0.2892
• All three eval curves improved monotonically through the run with no observed overfitting signal.

Domain	Train rows	Eval rows	Token share
Coding	16,083	1,778	82.5%
Tool-calling	5,360	595	4.8%
Agentic	917	102	12.7%

trjxter/Kimi-K2.7-CodingTraces-9000x was used in full: 9,014 rows.

8. Benchmarks

Qwimi was compared against the base Qwen 3.6 27B model using equivalent Q6 GGUF builds with MTP enabled and the same evaluation harness.

Important: these results were produced by the Q6 GGUF / llama.cpp builds. They are included as evidence about the underlying Qwimi model, not as direct benchmarks of this MLX quant. MLX Q4, Q6, and Q8 should be benchmarked separately before making backend- or precision-specific claims.

8.1 Fully auto-scored 300-task custom benchmark

Category	Tasks
Coding	150
Tool-calling	75
Agentic workflows	75
Total	300

Model	Tasks	Pass rate	Average wall time	Total wall time
Base Qwen 3.6 27B Q6 MTP	300	81.33%	53.56 s	267.79 min
Qwimi 3.6 27B Coder Q6 MTP	300	82.67%	27.09 s	135.45 min

Overall change	Result
Absolute pass-rate gain	+1.33 percentage points
Relative pass-rate gain	+1.64%
Average wall-time reduction	49.4%
Effective task-level speedup	~1.98×
Total evaluation time saved	132.34 minutes

Category results

Category	Base pass rate	Qwimi pass rate	Accuracy delta	Base avg time	Qwimi avg time	Wall-time reduction
Custom coding	81.33%	85.33%	+4.00 pts	59.33 s	21.16 s	64.3%
Custom tool-calling	88.00%	89.33%	+1.33 pts	31.77 s	27.13 s	14.6%
Custom agentic	74.67%	70.67%	-4.00 pts	63.80 s	38.92 s	39.0%

Pairwise outcomes

Pairwise result	Tasks	Share
Qwimi passed and base failed	23	7.67%
Base passed and Qwimi failed	19	6.33%
Both passed	225	75.00%
Both failed	33	11.00%
Total	300	100%

Among the 42 decisive tasks where only one model passed, Qwimi won 23/42 = 54.8%.

Category	Qwimi-only wins	Base-only wins	Both pass	Both fail
Coding	11	5	117	17
Tool-calling	4	3	63	5
Agentic	8	11	45	11
Total	23	19	225	33

300-task verdict: a slight overall quality win for Qwimi, driven by stronger coding and structured tool-use performance, with a large reduction in end-to-end task time. Agentic accuracy remains the clearest weakness.

8.2 Mixed 200-task benchmark

A separate 200-task run produced 400 raw answer rows. Categories with incomplete scorer or export integration were omitted rather than counted as zero.

Model	Auto-scored rows	Passed	Pass rate	Average wall time
Base Qwen 3.6 27B Q6 MTP	151	46	30.46%	92.86 s
Qwimi 3.6 27B Coder Q6 MTP	166	51	30.72%	71.30 s

Overall change	Result
Absolute pass-rate gain	+0.26 percentage points
Relative pass-rate gain	+0.85%
Average wall-time reduction	23.2%

On the 146 tasks with a valid paired auto-score:

Pairwise result	Tasks	Share
Qwimi better	7	4.79%
Base better	2	1.37%
Same result	137	93.84%
Total paired tasks	146	100%

Among the nine decisive paired tasks, Qwimi won 7/9 = 77.8%.

Category	Base pass rate	Qwimi pass rate	Delta
HumanEval	80.00%	94.29%	+14.29 pts
Personal XML tool tasks	50.00%	100.00%	+50.00 pts
MBPP	5.77%	1.92%	-3.85 pts
Tool-calling checks	100%	100%	Tie
Architecture checks	100%	100%	Tie
Debugging checks	100%	100%	Tie

8.3 Interpreting the latency results

The measured advantage is task-level wall time, not proof of faster kernels or higher per-token throughput.

Task time ≈ prefill time + generated tokens × decode latency per token

SFT can reduce task time by producing shorter, more direct, better-formatted outputs and reaching the stop condition sooner. A controlled comparison of prompt throughput, generation throughput, and generated-token counts is required to claim a per-token speed advantage.

8.4 Benchmark limitations

• The published comparison used Q6 GGUF under llama.cpp, not MLX.
• MLX Q4, Q6, and Q8 were not benchmarked against one another.
• Wall time varies with Mac hardware, mlx-lm version, context length, cache behavior, generation length, and sampling settings.
• The custom suite is useful internal evidence, not a replacement for broad public benchmarks.
• Agentic auto-scoring cannot capture every aspect of repository-level engineering quality.

9. Known limitations

• Text-only specialization: the vision tower was frozen during SFT and was not improved.
• Validated context: training and evaluation used a maximum sequence length of 16,384 tokens. Behavior beyond that length is unvalidated.
• Agentic coverage: agentic data was much smaller than the coding portion, and the custom benchmark showed a four-point agentic accuracy regression against the base model.
• Quantization loss: this MLX conversion adds quantization error on top of BF16.
• Apple Silicon only: MLX is designed for Apple silicon and is not a CUDA/Windows runtime.
• Bleeding-edge architecture: Qwen 3.6 uses the qwen3_5 hybrid architecture with GatedDeltaNet layers. Use a current mlx-lm build and verify generation locally.
• MTP caveat: the name retains MTP from the source model, but this does not guarantee that MLX preserves or actively uses the same MTP acceleration path as the GGUF benchmark.
• Tool-calling caveat: XML-style tool output and server-side parsing can be version-sensitive. Validate multi-turn tools before relying on them in an agent workflow.
• Server caveat: mlx_lm.server is intended primarily as a local endpoint and is not a hardened production service.

---

10. Verification and reproducibility

Record the exact:

• mlx-lm version;
• source BF16 commit SHA;
• conversion command;
• quantization mode, bit width, and group size;
• output repository commit SHA;
• uploaded size and weight-shard checksums;
• smoke-test prompt and structural result.

Representative conversion command:

mlx_lm.convert \
  --hf-path trjxter/Qwimi-3.6-27B-Coder-MTP-BF16 \
  --mlx-path ./Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  -q \
  --q-bits 8

Add the actual --q-group-size, quantization mode, and upload arguments used during conversion instead of assuming defaults.

Suggested smoke tests

1. Plain coding generation.
2. Exact <think>...</think> formatting.
3. A single XML-style tool call.
4. A multi-turn tool call with a tool result.
5. A 12K-16K-token prompt to check memory and cache behavior.
6. Repeated generation to detect truncation or cache regressions.

---

License

Apache 2.0, inherited from the base and merged source model.