Dolphin 3.0 Llama 3.2 3B — Core ML
Built with Llama.
This repository contains Core ML conversions of
dphn/Dolphin3.0-Llama3.2-3B,
plus the tokenizer, generation configuration, reproducible export code, and
Python and Swift generation examples needed to use the recommended model.
Multipart LUT6 export with persistent state
An additional conversion of the same pinned Dolphin 3.0 Llama 3.2 3B checkpoint
is available under ane-lut6-c2048/. It contains
nine Core ML packages: embedding, seven four-layer transformer blocks,
and the final normalization/vocabulary head. Each block keeps its own KV
state. The projections use 1×1 convolutions and the runtime requests
.cpuAndNeuralEngine for an architecture intended for the Apple Neural Engine.
| Multipart release setting | Value |
|---|---|
| Minimum OS | iOS 18 / macOS 15 |
| Context | 2,048 consumed tokens |
| Query and prefill | One token per invocation |
| Weights | Scalar LUT6, 64 FP16 palette entries per group of 16 output channels |
| Nine packages together | 2,718,527,235 bytes |
| KV state allocation | 224 MiB, excluding weights and runtime buffers |
| Runtime and instructions | ane-lut6-c2048/toolkit/swift |
This multipart LUT6 release has not been compiled or executed with the Apple
SDK/Core ML runtime. Its ANE placement and runtime performance remain
unverified. A .cpuAndNeuralEngine configuration permits CPU fallback and
does not prove execution on the ANE. The supplied evidence verifies all
205 reconstructed compressed matrices, package hashes and interfaces, and
84 Python tests. Three fixed CPU next-token probes agree with the FP16
reference; they do not establish general answer quality or device parity.
See the LUT6 conversion report.
Download the complete multipart layout with an authenticated or public Hub CLI:
hf download ales27pm/Dolphin3.0-CoreML \
--revision main \
--include 'ane-lut6-c2048/**' \
--local-dir Dolphin3.0-CoreML
From Dolphin3.0-CoreML/ane-lut6-c2048/, follow its README; for example, its
Apple validation command begins with cd toolkit/swift and reads
../../models. All nine packages, the manifest and both RoPE tables are
required. The source, runtime and evidence are also maintained in
ales27pm/CoreML-DOLPHIN3.0.
The sections below describe the existing monolithic INT4 and legacy
artifacts. Their Python/Swift examples, dynamic input interface and Apple
validation reports apply to those separate packages. For the multipart LUT6
release, use ane-lut6-c2048/toolkit/swift and its own unverified Apple-device
validation status; the monolithic examples do not load the nine-part pipeline.
Recommended artifact
| Artifact | Weights | Core ML state | Context | Prefill chunk | Minimum OS |
|---|---|---|---|---|---|
Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage |
INT4, per-block linear | keyCache, valueCache |
2,048 tokens | 512 tokens | iOS 18 / macOS 15 |
The stateful package is the supported interactive-generation artifact. It
keeps the transformer key/value cache inside MLState, so decode submits only
the new token instead of recomputing the full prefix.
The upstream model advertises a 131,072-token context. This Core ML conversion deliberately caps state at 2,048 tokens to bound on-device memory. The prompt prefill chunk is capped at 512 tokens. A caller must reject or deliberately truncate input before either limit is exceeded.
Quick start: Python on Apple silicon
Requirements: macOS 15 or newer, Xcode Command Line Tools, and
uv. The Core ML runtime is available only on
Apple platforms; the export script itself can run on Linux.
uvx --from 'huggingface_hub[cli]' hf download ales27pm/Dolphin3.0-CoreML \
--include 'Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage/**' \
--include 'config.json' \
--include 'generation_config.json' \
--include 'special_tokens_map.json' \
--include 'tokenizer.json' \
--include 'tokenizer_config.json' \
--include 'scripts/generate.py' \
--include 'examples/swift/DolphinCoreMLCLI/**' \
--local-dir Dolphin3.0-CoreML
cd Dolphin3.0-CoreML
uv run scripts/generate.py \
"Explain why the sky is blue in two sentences." \
--model Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
--tokenizer . \
--max-new-tokens 64
scripts/generate.py applies the bundled Dolphin chat template, creates a new
Core ML state for the conversation, performs a causal prefill, decodes one
token at a time, verifies that logits are finite, and stops on token IDs
128256, 128001, 128008, or 128009.
Quick start: iPhone
The iOS 18 SwiftUI test app downloads and verifies the pinned v2.0.0 model
and tokenizer, compiles the model into the app bundle, prevents overlapping
generations, and shows load and generation timings from the physical device.
cd examples/ios/DolphinCoreMLTestApp
zsh prepare-model.sh
open DolphinCoreMLTestApp.xcodeproj
In Xcode, choose your development Team, connect and select an iOS 18 or newer iPhone, then choose Product > Run. Open Settings, tap Load Model, and wait for Model ready. Return to Chat and send a short request. Use the Activity tab to distinguish model turns, planned actions, approvals, and verified tool receipts. See the iPhone assistant guide for tools, continuity, memory limits, metrics, and cancellation notes.
Quick start: Swift
The buildable example uses a revision-pinned copy of Hugging Face
swift-transformers for tokenization and calls Core ML directly for correct
state and causal-mask handling.
cd examples/swift/DolphinCoreMLCLI
swift run dolphin-coreml \
../../../Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
"Write one friendly sentence about Montreal." \
--tokenizer-folder ../../..
The example targets macOS 15 and iOS 18. It uses greedy decoding so its output is deterministic for a fixed model and prompt. The source is suitable as a small reference integration; production apps should add cancellation, memory pressure handling, sampling controls, and their own safety policy.
Model interface
| Name | Kind | Type | Shape |
|---|---|---|---|
inputIds |
Input | Int32 | [1, 1...512] |
causalMask |
Input | Float16 | [1, 1, 1...512, 1...2048] |
keyCache |
State | Float16 | [28, 1, 8, 2048, 128] |
valueCache |
State | Float16 | [28, 1, 8, 2048, 128] |
logits |
Output | Float16 | [1, query_length, 128258] |
The causalMask final dimension is the absolute end position of the submitted
chunk. For a prompt of N tokens, pass [1, 1, N, N]. For the following
single token, pass [1, 1, 1, N + 1] using the same MLState. Create a fresh
state to start a new conversation.
Core ML stateful models and the attention operations used here require iOS 18 or macOS 15. The model is not compatible with the older iOS 15/macOS 12 claims that appeared in the previous card.
Reproducible export
The conversion pins the source model and tokenizer to:
dphn/Dolphin3.0-Llama3.2-3B@392a6f57223e7ccfe6ef4ebdb2ff101a42d57364
Pinned conversion dependencies are in requirements.txt and in the PEP 723
header of scripts/export_stateful_coreml.py:
uv run scripts/export_stateful_coreml.py \
--model-id dphn/Dolphin3.0-Llama3.2-3B \
--revision 392a6f57223e7ccfe6ef4ebdb2ff101a42d57364 \
--max-context-length 2048 \
--max-query-length 512 \
--quantize int4 \
--output Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage
The exporter first compares cached single-token decode against a full-prefix PyTorch calculation, traces the stateful model, converts it with Core ML Tools 8.0 for iOS 18, applies symmetric per-block INT4 weight quantization, embeds the source/runtime contract in model metadata, and emits a per-file hash report.
Bundled tokenizer/config file hashes:
| File | SHA-256 |
|---|---|
config.json |
e21ff53ea39726f972362beba869807216775d5e308bc2f531784846c06a0249 |
generation_config.json |
e627b5a8b2dc371f90388947ada64fa6e71de0f991c04c835f0c0bc97e305a4f |
special_tokens_map.json |
2df2c4620bb1a9eb877bc7c90c7fa04608bda9fa7c0cf2cdcc0a17b849649683 |
tokenizer.json |
e40b93124a3e29f62d5f4ff41be56cb2af34ecacf9239acd9da53a98860380b5 |
tokenizer_config.json |
51ad9580aba8d00016efda43357185a0d8ff9884584dcc82ab58ca552afd14e1 |
The recommended package totals 1,808,547,525 bytes. Its release inventory is:
| Package path | Bytes | SHA-256 |
|---|---|---|
Data/com.apple.CoreML/model.mlmodel |
809,496 | a34a00a253c98153cf3b231105493edddde532086e224443e40b255b0f10a924 |
Data/com.apple.CoreML/weights/weight.bin |
1,807,737,412 | 6240edc377b1a0158812454c4bb6e3053d8e8a75a7eedb751b9740fffdfd3e15 |
Manifest.json |
617 | 5b8ac347a822f02ba3a6d9ccff60dd723f2649424c8e88570961f12b1c59afb6 |
coreml_artifacts.json is the machine-readable inventory for the recommended
package and all three legacy packages.
Validation
Run the repository, model-schema, hash, Swift interface, and Apple compiler checks with:
uv run scripts/validate_release.py \
Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
--compile
swift build --package-path examples/swift/DolphinCoreMLCLI
validation/ contains the machine-readable export and release reports. The
release report independently matches every package file against the export
report and artifact manifest, checks specification version 9, both state
shapes, dynamic input ranges, FP16 logits, and 197 blockwise-quantized weight
operations, then passes coremlcompiler metadata, Swift generation, and macOS
15 compilation. A compiler pass proves package and deployment-target
compatibility; it does not substitute for generation on the target device.
Before shipping in an app, measure cold/warm load, peak resident memory,
prefill latency, tokens/second, thermal behaviour, cancellation, and
memory-pressure recovery on each supported device class.
validation/tiny-stateful-runtime-smoke.json records an actual Core ML
prefill/decode test of the cache and dynamic-position contract with a synthetic
two-layer model. It is deliberately not presented as 3B model-quality or
target-device performance evidence.
Legacy artifacts
The original FP16, INT8, and INT4-LUT packages remain available for reproducibility, but are not the recommended chat runtime:
| Artifact | Context per call | Core ML state | Status |
|---|---|---|---|
Dolphin3.0-Llama3.2-3B-fp16.mlpackage |
256 | None | Legacy, stateless |
Dolphin3.0-Llama3.2-3B-int8.mlpackage |
256 | None | Legacy, stateless |
Dolphin3.0-Llama3.2-3B-int4-lut.mlpackage |
256 | None | Legacy, stateless |
All three legacy packages require iOS 18/macOS 15 and expose input_ids,
attention_mask, cache_position, and logits. They require full-prefix
recomputation for every generated token. They are retained as conversion
baselines, not advertised as interactive 131K-context models.
Intended use and limitations
The upstream model is a Dolphin instruction-tuned Llama 3.2 3B model. It can hallucinate, generate biased or unsafe content, and produce incorrect factual claims. It should not make unsupervised medical, legal, financial, safety-critical, or high-impact decisions. Dolphin is intentionally less guardrailed than many assistant models, so downstream applications are responsible for appropriate policy enforcement and evaluation.
INT4 compression can change output quality. Compare the stateful INT4 model
against the pinned PyTorch source on representative application prompts before
deployment. This repository does not claim parity beyond the checks recorded
under validation/.
License and attribution
This conversion inherits the Llama 3.2 Community License. See LICENSE,
USE_POLICY.md, and NOTICE. Redistribution and use must comply with those
terms, including the required attribution. Review the upstream Dolphin model
card for any additional dataset or fine-tune considerations.
Relevant primary references:
- Downloads last month
- 717