--- license: gemma library_name: coreml base_model: google/gemma-4-E2B-it tags: - coreml - apple-silicon - ane - on-device - gemma-4 - gemma-3n - multimodal - vision - audio pipeline_tag: image-text-to-text --- ## Use it from Swift ### Add the package `Package.swift`: ```swift .package(url: "https://github.com/john-rocky/CoreML-LLM", branch: "main"), // In your target: .product(name: "CoreMLLLM", package: "CoreML-LLM"), ``` Platforms: iOS 18+ / macOS 15+. ### Download + chat (one call) ```swift import CoreMLLLM let llm = try await CoreMLLLM.load(repo: "mlboydaisuke/gemma-4-E2B-coreml") let stream = try await llm.generate( [CoreMLLLM.Message(role: .user, content: "Hello!")], maxTokens: 256 ) for await chunk in stream { print(chunk, terminator: "") } ``` ### Image / video / audio ```swift // Image let stream = try await llm.generate( [CoreMLLLM.Message(role: .user, content: "Describe this image")], image: cgImage) // Video (frames + audio extracted internally) let stream = try await llm.generate( [CoreMLLLM.Message(role: .user, content: "What happens in this clip?")], videoURL: localFileURL) ``` Audio-only and other variants are exposed via the same `generate(_:)` overloads — see the Swift file [`CoreMLLLM.swift`](https://github.com/john-rocky/CoreML-LLM/blob/main/Sources/CoreMLLLM/CoreMLLLM.swift) for the full surface. # Gemma 4 E2B — Core ML (ANE multimodal) Core ML port of [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) (the 2B-effective Gemma 4 / Gemma 3n decoder), optimized for Apple Neural Engine. Text + image + audio + short video, INT4 weights. > **Branches:** `main` is the long-running 4-chunk text+vision+audio bundle. The default ship target for [`CoreMLLLMChat`](https://github.com/john-rocky/CoreML-LLM) v1.6+ is the **`n1024`** branch (3-chunk merged decoder, slightly faster prefill). Both ship the same architecture — only the chunk topology and tokenizer artifacts differ. Pick whichever matches the Swift runtime you're using; if in doubt, use `n1024`. ## Files (root, `n1024` branch — recommended) ``` chunk1.mlmodelc/ # L0–7 — INT4 palettized chunk2_3way.mlmodelc/ # L8–24 — merged middle (3-chunk decoder) chunk3_3way.mlmodelc/ # L25–34 + lm_head — multifunction prefill_chunk{1..4}.mlmodelc/ # T=N prefill bodies (mlmodelc, weights shared # with decode chunks via hardlink) vision.mlmodelc/ # SigLIP encoder, 322 MB vision_video.mlmodelc/ # video frame encoder (64 tok/frame) audio.mlmodelc/ # 282 MB Whisper-style audio encoder embed_tokens_q8.bin 402 MB — INT8 token embeddings (262144 × 1536) embed_tokens_scales.bin 512 KB embed_tokens_per_layer_q8.bin 2.19 GB — INT8 PLE embed_tokens_per_layer_scales.bin 512 KB per_layer_projection.bin 26 MB per_layer_norm_weight.bin 1 KB cos_{full,sliding}.npy 8 MB / 4 MB — precomputed RoPE cos sin_{full,sliding}.npy 8 MB / 4 MB — precomputed RoPE sin mel_filterbank.bin 129 KB — for audio path embed_proj_weight.npy 4.5 MB — vision/audio → text embed projection output_proj_{weight,bias}.npy 3 MB / 3 KB — audio output projection model_config.json 434 B — runtime config (hidden=1536, layers=35, …) audio_config.json 402 B — audio path config hf_model/{tokenizer.json, tokenizer_config.json, config.json} ``` The `main` branch additionally carries the older 4-chunk topology (`chunk2.mlmodelc` + `chunk3.mlmodelc` + `chunk4.mlmodelc`) and several legacy variant directories (`sdpa/`, `sdpa-8k/`, `swa/`, `stateless/`, `stateless-ctx2048/`, `lite/`, `lite-chunks/`, `mf/`, `w8a8-8k/`, `model.mlmodelc`, `model.mlpackage`). These are research builds — only the `chunk*.mlmodelc` (or `chunk{1,2_3way,3_3way}.mlmodelc`) family is the shipping path. ## Why so many sidecars Gemma 4 / 3n uses a per-layer embedding (PLE) bank that dwarfs the token embedding. Loading PLE through Core ML would dequant the whole 2.19 GB into the CPU heap. Instead, the raw INT8 + scale files are mmap'd in Swift and only the rows actually touched are dequantized on the fly. The chunks themselves stay ANE-resident. `cos`/`sin` `.npy` are pre-baked so the Swift side doesn't ship a RoPE builder. ## Tokenizer Already in `hf_model/`. Or pull from upstream: ```python from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("google/gemma-4-E2B-it") ``` ## Standalone usage (Python / Mac) ```python from huggingface_hub import snapshot_download import coremltools as ct, json local = snapshot_download( "mlboydaisuke/gemma-4-E2B-coreml", revision="n1024", allow_patterns=[ "chunk1.mlmodelc/*", "chunk2_3way.mlmodelc/*", "chunk3_3way.mlmodelc/*", "prefill_chunk*.mlmodelc/*", "embed_tokens*.bin", "per_layer_*.bin", "cos_*.npy", "sin_*.npy", "model_config.json", "hf_model/*", ], ) cfg = json.load(open(f"{local}/model_config.json")) chunks = [ ct.models.MLModel(f"{local}/chunk1.mlmodelc"), ct.models.MLModel(f"{local}/chunk2_3way.mlmodelc"), ct.models.MLModel(f"{local}/chunk3_3way.mlmodelc"), ] ``` For a working end-to-end loop (PLE dequant, vision/audio injection, KV alias plumbing), see [`Sources/CoreMLLLM/ChunkedEngine.swift`](https://github.com/john-rocky/CoreML-LLM/blob/main/Sources/CoreMLLLM/ChunkedEngine.swift) — the canonical reference. ## Vision / Audio - `vision.mlmodelc` expects `pixel_values (1, 3, 256, 256)` fp16, outputs `(1, 256, 1536)` text-aligned tokens. - `audio.mlmodelc` expects mel-spectrogram features (use `mel_filterbank.bin` for the front-end), outputs an audio token stream injected into the same text decoder. - `vision_video.mlmodelc` packs 64 tokens per frame for short video. ## iOS / Mac app Pick **Gemma 4 E2B** in [`CoreMLLLMChat`](https://github.com/john-rocky/CoreML-LLM/tree/main/Examples/CoreMLLLMChat) — it auto-downloads this repo (the picker fetches the `n1024` branch by default) and runs it via `ChunkedEngine`. ## Architecture | | value | |---|---:| | `num_hidden_layers` | 35 | | `hidden_size` | 1536 | | `num_key_value_heads` | 1 | | `intermediate_size` | 6144 | | `num_kv_shared_layers` | 20 | | KV producers (sliding/full) | L13 / L14 | | sliding window | 512 | | context length (shipping) | 1024 (n1024) / 2048 (main) | | vocab | 262144 | ## License Inherits the [Gemma terms of use](https://ai.google.dev/gemma/terms).