README.md

---
license: gemma
language:
- en
tags:
- function-calling
- tool-calling
- gemma3
- coreml
- ane
- on-device
base_model: google/functiongemma-270m-it
library_name: coreml
pipeline_tag: text-generation
---

# FunctionGemma 270M — CoreML (fp16)

A Core ML export of [google/functiongemma-270m-it](https://huggingface.co/google/functiongemma-270m-it),
optimized for the Apple Neural Engine on iOS 18 / macOS 15. The 18-layer
transformer is reshaped into Apple's **BC1S layout** (`(B, C, 1, T)`
channel-last with 1×1 `Conv2d` projections and per-head split attention)
and the K/V cache lives in **`MLState`** slots, so token-by-token decode
sends no tensor I/O back to the host.

This is the **fp16 reference** build — full Float16 weights, no
quantization. For the same accuracy with ~½ the disk and faster decode,
see the
[Palettize-8 variant](https://huggingface.co/aufklarer/FunctionGemma-270M-CoreML-Palettize8).

## Model

| | |
|---|---|
| Parameters | 270M |
| Architecture | Gemma 3 (18 layers, 4 query heads, 1 KV head, head_dim 256, hidden 640, MLP 2048) |
| Quantization | none (Float16 weights, fp16 compute) |
| Format | Core ML `.mlmodelc` (ML Program) |
| Cache layout | BC1S `MLState`, fixed cache length 128 |
| Shapes | T_q ∈ {1, 128} via `EnumeratedShapes` |
| File size | 513 MB model + 33 MB tokenizer ≈ 546 MB total |
| Min target | **iOS 18 / macOS 15** |
| Compute units | `cpuAndNeuralEngine` (required — CPU-only emulation diverges) |

## Files

| File | Size | Description |
|---|---|---|
| `FunctionGemmaANEUnifiedStateful.mlmodelc/` | 513 MB | Compiled Core ML model. Load with `MLModel(contentsOf:)`. |
| `config.json` | ~2 KB | Architecture metadata (state names, input/output names, deployment target). |
| `chat_template.jinja` | ~1 KB | Jinja chat template used by `tokenizer.apply_chat_template`. |
| `tokenizer.json` | ~33 MB | Hugging Face `tokenizers` fast SentencePiece model. |
| `tokenizer_config.json` | ~1 KB | Tokenizer settings. |

## Performance

Measured on Apple M-series Mac via `cpuAndNeuralEngine`, on the canonical
"Convert 23 USD to EUR" tool-call prompt (91-token prompt → 31-token
function call), warmed.

| | Value |
|---|---|
| Prefill (128 tokens) | 8.5 ms |
| Decode | 5.65 ms/token (**177 tok/s**) |
| End-to-end (32 tokens) | ~185 ms |
| Swift peak RSS (warm) | **~37 MB** private + ~510 MB mmap'd from disk (evictable) |
| Compute-plan device | 96 %+ of ops prefer `neuralEngine` |
| Output | Byte-identical to PyTorch fp32 reference on the canonical prompt |

## Usage

### Swift (iOS 18 / macOS 15)

```swift
import CoreML

let url = URL(fileURLWithPath: "FunctionGemmaANEUnifiedStateful.mlmodelc")
let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine
let model = try MLModel(contentsOf: url, configuration: config)
let state = model.makeState()

// Build prefill inputs (input_ids, cos/sin tables, attention mask,
// write_mask=ones, logits_mask one-hot at the last prompt position),
// then for decode call repeatedly with T_q=1 inputs and a one-hot
// write_mask at the current cache slot.
let output = try await model.prediction(from: prefillInputs, using: state)
let logits = output.featureValue(for: "logits")!.multiArrayValue!
```

The full prefill + decode driver is published as part of the
[speech-swift](https://github.com/soniqo/speech-swift) SDK.

### Python (coremltools, macOS only)

```python
import coremltools as ct
import numpy as np

model = ct.models.MLModel(
    "FunctionGemmaANEUnifiedStateful.mlpackage",
    compute_units=ct.ComputeUnit.CPU_AND_NE,
)
state = model.make_state()
out = model.predict(prefill_inputs, state=state)
next_id = int(out["logits"][0].argmax())
```

## Source

Upstream model: **[google/functiongemma-270m-it](https://huggingface.co/google/functiongemma-270m-it)**
— Gemma 3 270M instruction-tuned for structured function calls.

## Links

- [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK
- [soniqo.audio](https://soniqo.audio) — website
- [blog](https://soniqo.audio/blog)