Three-bit MLP vector quantization on the Apple Neural Engine
By Anemll Open Source Project · anemll.com
Code and models: GitHub · Runnable model bundle · Quantized conversion export
Jump to: Method · Accelerated inference · Experimental setup · Results · Code and data
Abstract
ANEMLL Forge release 0.2 adapts Qwen3.8-27B to the Apple Neural Engine using three-bit vector quantization for every MLP matrix. Each matrix stores a small table of learned weight pairs, an index for each pair, and per-output-channel scales. Fixed rotations, GPTQ and 600 steps of quantization-aware training help preserve the original model's predictions.
On M6, the compiled model reaches 0.0518 KL-512 to the BF16 reference, compared with 0.1838 for release 0.1, and averages 37.5 tokens/s across 24 chat and coding prompts with DFlash2. Packed weights occupy 9.74 GiB, including correction factors and excluding embeddings. Separate attention optimizations use FP8 softmax intermediates on M6 and FP16 on M5. These results measure prediction fidelity and workload-specific performance; standardized task accuracy and release-level energy evaluation remain future work.
Introduction
A low-bit model is useful when its compressed representation survives compilation, preserves useful behavior, and runs well on the actual device. Those three requirements shaped this project.
We started with the Qwen Team's Qwen3.8-27B and adapted its weights and runtime for the ANE through Core AI. Release 0.1 split the MLP between two-bit vector tables in 38 layers and four-bit scalar tables in 26. Release 0.2 uses three-bit vector tables throughout the MLP and extends online rotations to the token mixers.
The main lessons are practical:
- Pairing weights makes three-bit storage possible. One six-bit index selects two FP16 centroid components, with separate scales restoring each output row's magnitude.
- Compiler behavior belongs in the format decision. Weight sharing, duplicated programs and temporary tensors can outweigh the nominal bit count.
- Validate the compiled model. A good PyTorch result needs to survive conversion, attention precision changes and speculative execution.
The sections below explain how those choices fit together, what we measured, and where the evidence still has gaps.
Related work
Our recipe combines established ideas. GPTQ compensates for rounding error using calibration inputs; GPTVQ, AQLM and VPTQ explore richer vector or additive representations. Weighted centroid fitting connects to SqueezeLLM and DKM.
QuIP, QuIP# and QuaRot show how transformations can make low-bit approximation easier. SpinQuant studies learned rotations. Our release uses fixed Hadamard rotations; we do not report a completed learned-rotation experiment. SmoothQuant motivates a separate input-rescaling ablation discussed in the results.
The floating-point correction branches draw on low-rank compensation methods such as ZeroQuant-V2 and LoftQ. Training continuous quantization parameters against a teacher relates to AQLM, PV-Tuning, EfficientQAT and LLM-QAT. Apple's foundation-model work also uses palettization and accuracy-recovery adapters; its ANE transformer guide informs the convolution-based layout here.
The runtime builds on Gated DeltaNet and speculative decoding with DFlash. SageAttention2 is related work on low-precision attention on GPUs. Our attention work concerns the ANE's compiled INT8/FP8 dataflow and the different M5/M6 paths. Links to these works and the implementation are collected under References.
Hardware format constraints
Formats such as MXFP4 and NVFP4 use small blocks with their own scales. That flexibility helps low-bit values follow local changes in weight magnitude.
The ANE execution path we tested does not provide MXFP4/NVFP4-style block-scaled execution. Core ML exposing per-block quantization for GPU execution does not establish ANE support. Grouped scalar lookup tables can run on ANE, but our grouped vector-table probes fell back to CPU.
Our working vector format uses one shared codebook per matrix, plus one scale per output channel—per-channel scaling, or PCS. Rotations and QAT help recover fidelity within that constraint. Vector tables run on the ANE on both M5 Max and M6, with substantially lower LUT latency on M6 in the sweep reported below. These observations concern the tested shapes and software paths, rather than every possible ANE implementation.
Method
Model and vector representation
Qwen3.8-27B has 64 layers: 48 Gated DeltaNet layers and 16 full-attention layers. Hidden width is 5,120 and MLP intermediate width is 17,408. Each MLP contains gate, up and down projections, giving 192 MLP matrices.
Gate and up produce the intermediate activation; down maps it back to the model's hidden width. Every one of these matrices uses 64 two-component FP16 centroids. A centroid is a representative pair of values learned from the weights.
One six-bit index selects a pair. The pair covers two adjacent output rows at the same input column. Each row has its own FP16 scale, shared across every input column in that row.
For example:
- Index 7 selects the centroid (0.5, −1.0).
- The two output rows have scales 2.0 and 0.25.
- The reconstructed weights are (1.0, −0.25).
Six index bits for two weights gives three bits per weight. The shared codebook adds 256 bytes per matrix, and the scales add two bytes per output row. Compiler copies and runtime buffers are additional.
This describes weight storage, not native INT3 arithmetic. The intermediate safetensors export stores one byte per pair; conversion packs the indices into six bits per pair. The Core AI builder imports the fitted tables and indices instead of fitting replacement codebooks.
Per-channel scaling across model blocks
Per-channel scaling means one scale for each output row. It is retained across all 192 MLP matrices, the selected token-mixer LUT projections, the attention K/V projection weights and the output head.
The mixer allocation remains mixed: selected DeltaNet and attention projections use two-bit vector tables in layers 0–23 and four-bit scalar tables in layers 24–63. Full-attention K and V projections separately use symmetric INT8 weights, each with an FP16 output-channel scale. The output head uses a four-bit scalar table and channel scales.
The FP16 low-rank correction branches are added alongside the scaled main projections. Embeddings, norms, the small DeltaNet projections and depthwise-convolution weights remain floating point. Their representation is different from the LUT/INT8 weight formats, and weight scales are distinct from the changing cache scales used during attention.
Online rotations
We use fixed, signed Hadamard rotations in blocks of 1,024. Applying the matching transformation to inputs and weights preserves the original linear operation before quantization. Spreading difficult directions across a block can then make the rounded weights more accurate.
Gate and up share an input rotation. Down receives a separate rotation after the nonlinear gating operation. Release 0.2 extends the online transforms to the large token-mixer input and output projections; the residual stream, embeddings and output head retain their original basis.
These rotations are implemented as grouped convolutions and add some runtime and memory cost. Their placement and seeds must agree between fitting, conversion and evaluation. Training learns the quantization parameters, not the rotations.
GPTQ workflow
Three operations work together:
- PCS makes rows comparable. Divide each output row by its RMS-derived scale before fitting the table.
- Centroid fitting chooses the table. Activation-weighted k-means gives more influence to input directions with greater calibration energy, then rounds the centroid components to FP16.
- GPTQ chooses indices and compensates for error. After rounding an input column, adjust the columns that have not yet been quantized to account for the introduced error.
The MLP workflow is:
- Capture example activations and measure input sensitivity.
- Compute row scales and fit a shared table to normalized weight pairs.
- Select centroid indices for the current column in normalized space.
- Restore the row scales before calculating the error passed to later columns.
- Continue through the matrix, keeping its table and scales fixed during this GPTQ pass.
PCS acts across output rows; GPTQ compensates across input columns. Pairing weights along output rows keeps each pair inside one input column. The 128-column processing blocks organize the computation; they do not introduce block scales. For rows with unequal scales, choosing the nearest normalized centroid is not identical to minimizing unscaled weight error.
Calibration is sequential. The down projection sees activations from the already-quantized gate and up projections, and later layers see quantized earlier layers. The release pass used 128 calibration rows of 1,024 tokens: 44 WikiText rows, 36 BF16 chat rows and 48 coding-agent rows.
For selected mixer projections, rank-64 FP16 factors approximate part of the remaining weight error. GPTQ fits the main weights around that correction, and calibration continues through the combined result. The factors add 0.30 GiB. We omit them from the MLP, where local three-bit tests found only a 2.8% error reduction from adding rank-64 corrections.
The LUT helpers and sequential quantizer contain the implementation.
Quantization-aware training
After GPTQ, 600 steps of QAT distill the BF16 model's next-token distribution. LUT indices and INT8 codes stay frozen. Training updates the centroid values, output-channel scales, mixer correction factors and the output head's table and scales. INT8 K/V weight scales train too. Continuous parameters are rounded to FP16 on export.
The training target uses the teacher's top 256 tokens plus a bucket for the rest of the vocabulary. Historical teacher log-probabilities were cached in FP16, with loss computed in FP32; this introduces a small numerical bias from cache rounding. The public code offers FP32 teacher caching for new experiments.
The generated corpus came from 527 prompt entries covering code, tool use, multilingual text, math and rare tokens. It produced 637 training rows. Adding the other calibration sources gave 757 training rows, with separate monitoring rows and a 128-prompt selection trace. The selected checkpoint was step 600. Repeated prompt strings in the source list mean that splitting generated sequences does not guarantee unique prompts across splits.
Materialization combines the trained parameters with the frozen codes into a complete export. The training recipe provides the full data and optimizer settings.
Accelerated inference on the ANE
Weight fitting and runtime attention precision are separate. GPTQ fits the saved projection weights. At inference, the graph also reduces precision for changing attention scores, softmax intermediates and cached values.
The default V8 cache keeps K in FP16 and V in INT8. Within each attention layer, every cached token and KV head has one FP16 value scale, shared across that head's value components. An optional KV8 export stores both K and V in INT8. The export option named “both” instead provides selectable FP16 and V8 layouts; it does not mean both K and V are quantized.
INT8 projection kernels and W8A8 compute
Both K and V projections use INT8 weights with FP16 activations. The compiler retains the INT8 kernels without an extra FP16 weight copy. The recorded M6 inspection reduced compiled weights from 50 to 40 MB per attention layer. A compatibility option expands the weights to FP16 instead.
Quantizing activations too—W8A8—can enable faster INT8 × INT8 arithmetic. A separate M6 convolution probe measured 38.4 TOPS with per-channel scales versus 18.6 for FP16, about 2.06×. Shared scales reached 52.8 TOPS. These are isolated arithmetic measurements, not language-model throughput or a matched comparison across M4, M5 and M6.
The release does not insert that activation quantization into the K/V projections. Its selected attention core uses FP16 × FP16 for query–key products and INT8 × FP8 for cached-value weighting. Supported quantize/dequantize boundaries fuse into producer and consumer tasks, helping preserve the compact representation between operations.
FP8 softmax intermediates on M6
The M6 cached-history path works like this:
- Multiply FP16 queries and keys, then store the masked scores in INT8.
- Subtract the tile maximum and store the exponentials in FP8 E4M3.
- Sum those exponentials for normalization, with an FP16 sum output.
- Fold in the value-cache scales and store the resulting softmax weights in FP8.
- Multiply those weights by INT8 cached values, combine tile results and the current floating-point block, then perform the final normalization.
The two FP8 stages belong to one softmax path. The weights remain unnormalized until the final division. The sum branches off before the value-cache scales enter the value-weighting path.
| Stage or tensor | M6 | M5 |
|---|---|---|
| Query–key multiply | FP16 × FP16 | FP16 × FP16 |
| Scores after quantization and mask | INT8 | INT8 |
| Exponentials for the softmax sum | FP8 | FP16 |
| Maximum and sum outputs | FP16 | FP16 |
| Scaled softmax weights for cached V | FP8 | FP16 |
| History keys / values | FP16 / INT8 | FP16 / INT8 |
| Attention output | FP16 | FP16 |
The main benefit is lower traffic for large intermediate tensors. The measured FP8-weight × INT8-value product had the same multiply-add cycle count as its FP16-weight control. FP8 also preserved more small softmax weights than the tested uniform UINT8 alternative after folding in value scales.
“FP8 softmax” describes this intermediate dataflow. The compiled program has no dedicated hardware softmax instruction, and an FP16 reduction output does not establish the precision of its internal accumulator.
Range control and the M5 path
FP8 needed device-level range checks. An earlier scaling choice produced incorrect M6 results for a captured head when a scaled weight reached about 200. The selected 1/64 scale limits the tested FP8 intermediates to at most 64 and matched the host simulation in that investigation. This is an observed implementation constraint, not E4M3's theoretical range.
The tested M5 ANE compiler rejects the FP8 variant. M5 retains INT8 scores and transposed FP16 keys, with FP16 softmax intermediates. Release 0.1 did not use FP8 attention.
In an earlier same-weight experiment, the combined INT8-score and corrected FP8 changes increased target-only prefill from 292.5 to 310.8 tokens/s at 8K and 199.6 to 223.9 at 64K. These measurements preceded transposed keys and measure the combined graph change, not FP8 alone or a release-to-release gain. The attention notes and compiled-program analysis record the details.
Blocked Gated DeltaNet verification
DFlash2 proposes seven tokens after an anchor, and the target verifies all eight rows together. Within Gated DeltaNet, later rows depend on earlier ones.
We express the small triangular dependency solve as a short sequence of parallel products instead of seven serial row updates. Broadcast multiplication and reduction keep these products on the ANE. The enabled path also uses native depthwise convolution and combines state products during prefill.
In a benchmark of three Gated DeltaNet cores, the combined optimizations reduced eight-row verification from 1.18 to 0.64–0.69 ms and 64-row prefill from 6.82 to 3.42–3.49 ms. These timings exclude input/output projections and the rest of the model. The algebraic rewrite is exact, though floating-point execution can differ. Single-token recurrence has no block solve and does not receive this particular benefit.
Accepted state and drafter execution
Verifier updates stay pending until the accepted length is known. The next target call commits only that prefix; rejected proposals do not advance the committed recurrent state or visible K/V history. Persistent buffers and reusable Swift binding plans reduce allocation and dispatch work.
Eight rows were a useful tradeoff in the measured workload. Shorter verifier calls saved little time while allowing fewer draft tokens to pass. Combining measured call times with acceptance statistics predicted 21–36% lower throughput for three- or four-row variants. That is a model of throughput, not an end-to-end benchmark of those variants, and eight evaluated rows do not guarantee eight emitted tokens.
The drafter combines new target features with its next draft block in one call and maintains a 2,048-slot history ring. During prefill, the runtime only ingests the final feature window needed by the drafter, including the next block's allowance. The target still processes every prompt token; the drafter's window does not truncate the target context.
The drafter is rebuilt with the target's QAT-updated output head. The speculative decoding guide explains pairing and state management.
Experimental setup
Hardware and artifacts
Compiled fidelity and runtime measurements use M6 and M5 Max. PyTorch quantization comparisons use M3 Ultra. Fitting used a Colab G4 with one 96 GB NVIDIA RTX PRO 6000 Blackwell. Inference requires macOS 27 and a compatible Xcode 27/Core AI environment.
The implementation uses PyTorch and Hugging Face Transformers. Initial centroid fitting and low-rank SVD run on CPU; Colab calibration, GPTQ factorization/error compensation and QAT use CUDA. GPTQ also supports Apple GPU execution through MPS, with FP64 factorization on CPU. End-to-end QAT on MPS has not been validated.
The final fitting run took about four GPU-hours: 54 minutes for GPTQ, three minutes for the teacher cache, and 3.1 hours for QAT. This excludes corpus generation and roughly 28 additional G4-hours plus four A100-hours of recipe-selection experiments.
The paper identifies revisions of the upstream checkpoint, quantized export, inference bundle and source. Those fix artifact identity; replaying a measurement also needs its data, environment and run settings. See the environment guide.
Evaluation protocols
The compiled evaluation uses 64 recorded conversation sequences and 40,023 next-token positions. Both models see the same recorded preceding tokens at each position—teacher forcing—so different generated histories do not affect the comparison.
KL-512 compares the BF16 teacher's top 512 tokens plus one bucket for the remaining vocabulary. Lower is closer to BF16. Top-1 agreement asks whether both models choose the same highest-probability token. This evaluation differs from the top-256 target used during QAT.
The cross-format comparison uses PyTorch for ANEMLL and llama.cpp for GGUF, excludes a shared 45-token system prompt, and scores development and held-out chat traces separately. WikiText uses 16 sequences of 1,024 tokens. The 16K score evaluates the final 2,048 tokens of 16 windows, each 16,384 tokens long. These results should be compared within their table, not directly against compiled KL-512.
Runtime results use DFlash2. The varied-prompt result averages 24 chat and coding prompts; fixed-prompt measurements are reported separately because generated text and draft acceptance affect speed. Compiled target memory counts chunks and the output head, excluding the drafter and other allocations.
Vector lookup table microbenchmark
The hardware-format sweep is a separate Core ML experiment, rather than the Core AI language-model benchmark. It compares chains of eight and sixteen 4,096-channel convolutions, with FP16 activations. Subtracting their median times and dividing by eight estimates the per-layer cost while reducing fixed call overhead.
The harness warms up the models and alternates chain lengths across repeated measurements. Placement checks assign the convolutions to the ANE, and outputs are checked against each quantized-weight reference. These checks validate the synthetic workload, not language-model accuracy. The M5 Max and M6 ran different OS builds, which limits claims about silicon alone.
Results
Compiled distributional fidelity
Across the same 64 conversation sequences, release 0.2 tracks the BF16 reference more closely:
| Compiled M6 measurement | Release 0.1 | Release 0.2 |
|---|---|---|
| Mean KL-512 ↓ | 0.1838 | 0.0518 |
| Median KL-512 ↓ | 0.030 | 0.009 |
| 99th-percentile KL-512 ↓ | 2.28 | 0.50 |
| Top-1 agreement ↑ | 86.0% | 91.9% |
Release 0.2 measured 0.0517 KL on M5 Max and 0.0515 in same-weight PyTorch evaluation. Each value compares that model against BF16; their differences are not direct ANE-versus-PyTorch KL measurements.
Comparison with other quantizations
The separate cross-format experiment compares fidelity and storage with two GGUF families:
| Format | Dev KL ↓ | Held-out chat KL ↓ | WikiText KL ↓ | 16K-window KL ↓ |
|---|---|---|---|---|
| ANEMLL 0.2 | 0.0536 | 0.0565 | 0.0707 | 0.762 |
| Unsloth UD-IQ3_XXS | 0.0619 | 0.0651 | 0.0506 | 1.108 |
| Draw Things IQ3_XXS | 0.0789 | 0.0833 | 0.1246 | 1.204 |
Packed storage, excluding embeddings, is 9.74 GiB / 3.27 bits per base weight for ANEMLL including its correction factors, 9.46 GiB / 3.17 bits for Unsloth, and 8.94 GiB / 3.00 bits for Draw Things.
Both IQ3 formats use local block quantization unavailable in our tested ANE path. They are useful fidelity/storage references under different hardware constraints, rather than directly deployable ANE alternatives.
ANEMLL has 13.2% lower held-out chat KL and 31.2% lower measured 16K-window KL than Unsloth, using about 3% more storage. Unsloth has lower WikiText KL. The comparison is not exactly size-matched, and the gains do not establish improvements in task accuracy. See the quantization guide for the experiment record.
Memory and throughput
The base packed weights take 9.44 GiB, excluding embeddings. FP16 correction factors add 0.30 GiB, giving 9.74 GiB total, or 3.27 bits per base weight over 25.6 billion base weights.
Compiled M6 target chunks plus the head occupy 12.97 GB, down from 13.32 GB for release 0.1. This is a different representation from packed weights. GiB uses binary units and GB uses decimal units; neither number is the full process memory or safetensors download size.
With DFlash2, M6 averages 37.5 tokens/s across 24 chat and coding prompts, versus 37.6 for the earlier mixed-format comparison. M5 Max averages 16.6 versus 17.2 tokens/s against its separately fitted mixed-format comparison.
On a separate fixed 16K prompt, M6 slows from 60.3 to 53.3 tokens/s as draft acceptance falls. The varied-prompt average does not predict every workload. At 64K with the drafter, the recorded 32 GB M6 setup needs about 2 GB of swap headroom.
M5 Max and M6 vector lookup table sweep
The synthetic sweep found substantially lower LUT latency on M6. Representative rows show the format effect:
| Weight representation | M5 Max, ms/layer | M6, ms/layer | M6 speedup |
|---|---|---|---|
| Dense FP16 | 0.211 | 0.218 | 0.97× |
| Dense INT8 | 0.109 | 0.103 | 1.06× |
| Scalar LUT 16, INT8 codebook | 0.108 | 0.045 | 2.42× |
| Vector 2 × 64, INT8 codebook | 0.110 | 0.049 | 2.23× |
| Vector 2 × 16, INT8 codebook | 0.112 | 0.028 | 3.96× |
| Scalar LUT 16, FP16 codebook | 0.112 | 0.055 | 2.02× |
| Vector 2 × 16, FP16 codebook | 0.109 | 0.025 | 4.34× |
INT8 or FP16 identifies the probe's weight/codebook type; activations are FP16. The released three-bit MLP uses FP16 centroids. These are per-layer synthetic timings, not token-generation speeds, and speedup ratios were calculated before rounding the displayed times.
On M5 Max, most tested LUT formats have similar latency. M6 benefits more from the compact representations, though vector quantization is not uniquely responsible: its scalar two-bit and vector 2 × 16 formats both take about 0.028 ms per layer. In a separate parallel-branch weight-streaming test, vector 2 × 16 is 2.32× faster on M6, and wider vectors do not improve its time further.
The M5 Max report and M6 report include the full sweep and placement checks.
Quantization and rotation ablations
Why three bits throughout the MLP? Across 24 matrices sampled from eight layers, three-bit tables close roughly 80–83% of the output-error gap between two-bit and four-bit tables. In a matched 150-step QAT comparison at the same total MLP index budget, three bits in every layer gives 23.0% lower development KL and 22.7% lower selection KL than two bits in half the layers and four bits in the other half.
What does the compiler change? Relative to two-bit tables, a format-only M6 test adds 0.145 ms per layer for three-bit vectors versus 0.365 ms for four-bit scalar tables. Some scalar down-projection weights are duplicated across functions, while vector weights are shared. Removing those copies helps explain the smaller compiled footprint despite more index bits and added rotations.
Do mixer rotations help? In matched 600-step runs with two seeds on a proxy allocation, fixed online rotations lower held-out KL by 6.4% and sealed-trace KL by 7.6%. That proxy upgrades 15 late MLP layers from the first release's two-bit format to four bits. The 16K-window result is inconclusive: a 2.2% increase in KL, with an interval that crosses zero. These transforms add roughly 1.2% decode time on M6, 2–3% on M5 Max, and about 0.4 GB of compiled target memory.
What did we leave out? A QuaRot-style residual-stream experiment required folding normalization gains into the weights. That worsened gate/up quantization error by 8% overall and 21–25% on the proxy's two-bit layers, outweighing the rotation benefit. Separate SmoothQuant-style input rescaling was neutral to worse, apart from small output-projection gains. We retained the online transforms and output-channel weight scales. These results do not establish that learned rotations would fail; no completed learned-rotation experiment is reported.
Limitations
The strongest quality evidence is distributional fidelity on fixed contexts. QAT optimizes a related KL objective, so lower KL is an expected outcome of training. It does not establish better coding, reasoning or instruction-following accuracy. WikiText also shows that the advantage does not hold across every evaluated distribution.
The released weights use one training seed. The rotation study uses two seeds on a proxy allocation, and the matched three-bit study uses a shorter 150-step budget. Those experiments do not measure the variation across independent full release-training runs.
Packing, weight sharing, placement and timing depend on shapes, OS and compiler behavior. Throughput additionally depends on prompts, generated text and draft acceptance. Microbenchmark gains should not be read as whole-model speedups, and the 64K swap requirement shows why packed weight size alone cannot predict serving memory.
Private calibration sessions, nonpublic evaluation traces and incomplete historical run records limit exact reproduction. Exact GGUF file revisions and backend build identifiers are also needed to replay those comparisons. Standard task benchmarks, independent training runs and matched energy measurements of the released model remain work to do.
Code and data availability
The public repository is ANEMLL Forge. File links in this article point to the reviewed source revision, so the implementation behind each explanation stays identifiable.
- Run the release: anemll/anemll-forge-qwen3.8-27B provides the matching Core AI target, DFlash2 drafter and supporting model assets. Follow the download and quick-test guide and environment requirements.
- Inspect conversion inputs: anemll/anemll-quantized-qwen3.8-27b-for-CoreAI contains indices, tables, scales, rotation metadata and correction factors. It is a conversion input, not a directly loadable Transformers checkpoint.
- Train a new variant: the training guide covers GPTQ, corpus generation, frozen-index QAT, materialization and source provenance.
- Inspect the evidence: the quantization guide contains ablations and release measurements; the sweep reports link archived timing data and placement checks; the attention notes link compiled-program investigations.
The training source was recovered from a committed research snapshot. The historical record does not identify the exact complete checkout executed by the Colab job. The public port also adds numerical safeguards, logging fixes and optional FP32 teacher caching. Small CPU checks cover these updates; the reported 27B training and benchmarks have not been rerun with them.
Some original calibration sessions and cached evaluation traces are not distributed. Replacement data defines a new experiment. Running the prepared release and reproducing its training history are therefore different goals. Bundle checks and a smoke test verify installation and execution, rather than model capability.
Conclusion
Release 0.2 combines three-bit MLP vector tables, fixed online rotations and frozen-index QAT in a representation that runs on the tested ANE path. The compiled model follows BF16 more closely, with mean KL-512 falling from 0.1838 to 0.0518 and top-1 agreement rising from 86.0% to 91.9%. Compiled target memory falls from 13.32 to 12.97 GB, while the varied-prompt M6 result is 37.5 tokens/s with workload-dependent slowdowns elsewhere.
The practical lesson is to choose formats using both model fidelity and measured compiler behavior, then validate the deployed graph. We welcome independent task evaluations, new calibration recipes and measurements on more Apple hardware through GitHub.
Credit for the original model belongs to the Qwen Team. ANEMLL contributes the quantization, conversion and runtime work.
References
The following links retain the paper's references, shortened to titles and artifact descriptions for browsing.
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- GPTVQ: The Blessing of Dimensionality for LLM Quantization
- AQLM: Extreme Compression of Large Language Models via Additive Quantization
- VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SpinQuant: LLM Quantization with Learned Rotations
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- SqueezeLLM: Dense-and-Sparse Quantization
- DKM: Differentiable K-Means Clustering Layer for Neural Network Compression
- ZeroQuant-V2: Post-training Quantization and Low-Rank Compensation
- LoftQ: LoRA-Fine-Tuning-Aware Quantization
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
- EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
- LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
- Distilling the Knowledge in a Neural Network
- Accuracy is Not All You Need
- Apple Intelligence Foundation Language Models
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- Fast Inference from Transformers via Speculative Decoding
- DFlash: Block Diffusion for Flash Speculative Decoding · DFlash code
- Apple: Deploying Transformers on the Apple Neural Engine
- Qwen3.8-27B source checkpoint
- ANEMLL Forge reviewed source revision
- ANEMLL Forge quantization guide and release results
- Centroid fitting, encoding and GPTQ helpers · Sequential quantizer
- Core AI model builder and exact-table conversion
- Release 0.2 quantization allocation
- Compiled Core AI KL-512 evaluator
- Environment setup and provenance
- Model download, integrity checks and inference guide
- Quantized Core AI export · Export instructions
- Prepared target and drafter bundle
- llama.cpp inference framework · IQ3 format definitions
- Unsloth Qwen3.8-27B GGUF quantizations
- Draw Things Qwen3.8-27B GGUF quantizations
- OCP Microscaling Formats specification
- NVIDIA: Introducing NVFP4 for Efficient and Accurate Low-Precision Inference
- Apple: Quantization Performance · Apple silicon deployment session
- Vector LUT investigation and observed ANE placement constraints
- M6 vector-LUT sweep · M5 Max sweep
- Archived M6 timing data · Archived M5 Max timing data
- Synthetic LUT model generator · Core ML timing harness
- Release 0.2 training workflow and validation · Source provenance
- Frozen-index QAT and teacher caching
- QAT corpus generation · Checked release prompt list
- QAT overlay materialization
- ANE HWX and MLIR analysis
- M6 compute acceleration experiments
- Release 0.2 validation on M5 · Softmax probes · Captured-attention checks
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
- Verifier block length experiments
- Speculative server and accepted-prefix state management · Core AI target runtime
- Core AI DFlash2 builder · DFlash2 runtime
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length
- KV-cache formats and per-token, per-head quantization

