| --- |
| base_model: oceanicity/Qwen3-4B-Instruct-2507 |
| library_name: coreml |
| tags: |
| - text-generation |
| - coreml |
| - apple-silicon |
| - 8-bit |
| - quantized |
| - qwen |
| --- |
| |
| # Qwen3-4B-Instruct - CoreML (8-Bit Quantised) |
|
|
| This is an 8-bit quantised CoreML conversion of the [oceanicity/Qwen3-4B-Instruct-2507](https://huggingface.co/oceanicity/Qwen3-4B-Instruct-2507) model. It has been heavily optimised for fast, efficient, and low-memory inference on Apple Silicon using the Apple Neural Engine (ANE). |
|
|
| Conversion and quantisation work was performed using a customised version of [0seba's coremlmodels tool](https://github.com/0seba/coremlmodels). |
|
|
| ## Model Details |
| - **Architecture:** Qwen3 (4 Billion Parameters) |
| - **Precision:** 8-bit Weights (Linear Symmetric, Per-Channel) / 16-bit Activations |
| - **Context Length:** 8,192 Tokens (KV Cache) |
| - **Format:** CoreML `.mlpackage` chunks |
|
|
| ## Optimisations Applied |
| - **Linear-to-Conv2d Patching:** Transformer linear layers were patched into 1x1 convolutions to better align with the Neural Engine backend. |
| - **RMSNorm Fusion:** Layer normalisation layers were fused using CoreMLTools graph passes to prevent FP16 overflow. |
| - **Chunking:** The model was split into 8 chunks to safely bypass the Neural Engine's hardware memory limits per segment. |
| - **Vocabulary Chunking:** The massive LM head was exported as a standalone chunked model to bypass the ~16,384 dimension limit on Apple Silicon. |
| - **Pre-computed Position Embeddings:** RoPE embeddings were computed statically during tracing to avoid precision loss and runtime math overhead. |
|
|
| ## Files Included |
| - `chunk_0.mlpackage` through `chunk_7.mlpackage`: The core transformer layers. |
| - `lm_head.mlpackage`: The chunked vocabulary output head. |
| - `embeddings.npy`: The standalone token embedding weights. |
|
|
| ## Usage |
| This model is ready to be used in CoreML inference pipelines that support multi-chunked stateful transformers. Ensure that your inference engine stitches the chunks together sequentially and routes the KV cache states appropriately. |
|
|
