mlboydaisuke/Nemotron-3-Nano-4B-CoreAI

Nemotron-3-Nano-4B — Core AI

Decode-only (S=1) Core AI bundles for NVIDIA's Mamba2 + attention + MLP hybrid (42 blocks: 21 Mamba2 / 17 MLP / 4 GQA NoPE attention), int8 weights with an absmax int8 head. No custom Metal kernel — at S=1 the selective scan is a single recurrence step, so the graph is loop-free.

variant	asset	for
gpu-pipelined/	nemotron_3_nano_4b_decode_int8hu.aimodel	Mac (JIT specialization)
ios-h18p/	nemotron_3_nano_4b_decode_int8hu.h18p.aimodelc	iPhone (AOT — a 4B graph cannot specialize on-device)

Use it

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Nemotron-3-Nano 4B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model nemotron-3-nano-4b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "nemotron-3-nano-4b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: none needed
• Entitlements: iOS needs com.apple.developer.kernel.increased-memory-limit — a 4.3 GB bundle is past the default jetsam limit
• First run downloads the model — 4.6 GB (Mac) / 4.6 GB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

Measured: 16.0 tok/s decode on an iPhone 17 Pro (cooled, AOT h18p, bandwidth-saturated) and 85.2 tok/s on an M4 Max GPU. Greedy output is token-identical to the fp32 transformers rollout on the probe prompts.

Requires COREAI_CHUNK_THRESHOLD=1 (S=1 bundle) and an engine that carries two extra fixed-shape states (the Mamba conv columns + SSM state) alongside the KV cache.

Port + recipe: coreai-model-zoo / nemotron-3-nano