On-device LLM on Android (LiteRT-LM)

By default the agent runs its LLM through Ollama on desktop/Editor. On Android, it can instead run the LLM fully on-device via the LiteRT-LM runtime, behind the same IAgentTransport interface. The transport is chosen at compile time:

// AgentBuilder.PrepareSetup()
#if UNITY_ANDROID && !UNITY_EDITOR
    IAgentTransport transport = new AndroidLlmTransport(dispatcher, "NPU", Log);
#else
    IAgentTransport transport = new OllamaTransport(Endpoint, Model, options.AgentName, Log);
#endif

Components

AndroidLlmTransport.cs  (C#, IAgentTransport)            Runtime/AgentCore/Runtime/
   ├─ LiteRtModelProvisioner.cs   places the .litertlm model in persistentDataPath/LiteRtModels/
   └─ AndroidJavaObject → LlmBridge (JNI)                Runtime/Plugins/Android/llm-release.aar
                          └─ LiteRT-LM Engine            built from the in-package AndroidBridge~/ source

AndroidLlmTransport marshals all JNI calls onto the Unity main thread via IUnityMainThreadDispatcher and wraps the bridge's async callbacks in Tasks.
The LiteRT-LM Engine is large (2–3 GB resident) and is treated as a per-process singleton.
A fresh conversation is created per request; the C# agent layer owns history and the prompt it sends (see Tool calling and context budgeting), avoiding unbounded native KV growth.

Tool calling and context budgeting

The transport uses the bridge's native tool template with a manual, callback-driven loop (RunNativeToolLoopAsync) — not a prompt-based two-stage protocol. There is no select_tool step and no schema-withholding:

Native OpenAPI tools, manual loop. AndroidLlmTransport builds a JSON array of full OpenAPI tool specs — [{name, description, parameters: {schema}}], the parameter schemas are not withheld — and passes it to the bridge's tool-aware generate(...) (see JNI contract). The bridge feeds the model's own tool template, the model emits a structured tool call, and the bridge fires onToolCall(callId, name, argsJson). C# runs the tool and replies via provideToolResult(callId, result); the whole loop runs inside one bridge.generate() call. LiteRT-LM's fully-automatic automaticToolCalling path is deliberately not used — it deadlocks after tool-result injection (callback_thread_pool DEADLINE_EXCEEDED, zero second-pass decode).
Single-engine defer-and-resume (vision / RAG). There is one process-wide LiteRT-LM engine, and it cannot run a nested generate() while one is in flight ("busy generating"). So a tool that itself needs the LLM mid-turn cannot make a nested call: instead it defers. A vision tool stashes the captured camera frame; the SearchKnowledge RAG tool (which needs its own LLM call to translate the query to English, plus any graph-mode keyword extraction) stashes its arguments and answers the bridge with a sentinel so the turn ends. RunNativeToolLoopAsync then resumes after the outer turn completes, answering the stashed vision frame with one multimodal generate, or running RAG retrieval
- answer, once the engine is free. The model still decides when to look or search; only the call is deferred.
Knowledge queries always search in English. The bundled knowledge base is authored in English, so SearchKnowledge translates a non-English query to English before retrieving (ResolveSearchQueryAsync in SearchKnowledgeTool, gated by IsLikelyNonEnglish so an already-English query skips the round trip). The translation is its own generate() call — on Android it runs at resume time, when the engine is free (the defer-and-resume above is what makes this nested-free), wired via BuiltinToolDependencies.TranslateToEnglish (set up by the host app). Before the search runs the tool emits a one-line note — 🌐 <원문> → <english> — through OnToolNote, which AlexaVoiceView renders as a persistent block in the result view so the translation step is visible (the same intermediate-state surface the rest of capture-and-resume uses). If no translator is wired the search falls back to the original query — the multilingual-e5 embedder still matches cross-lingually.
Context budgeting. The Android context budget is 3072 tokens (set by the host app), deliberately lower than LiteRT-LM's MAX_NUM_TOKENS = 4096 session cap (LlmBridge.kt). The lower budget makes memory compaction (CompactionTriggerPercent = 95) fire before the prompt + tool result can overflow the native window, which is what would otherwise make decoding break down after a few turns.

JNI contract

The AAR must expose exactly this (the C# side depends on the names):

initialize(String modelPath, String backend, InitCallback)
generate(String prompt, byte[] jpeg /* nullable, JPEG bytes */, StreamCallback)                       // text/multimodal, no tools
generate(String systemPrompt, String prompt, byte[] jpeg, String toolsJson, StreamCallback)           // tool-aware overload
provideToolResult(String callId, String result)
interface InitCallback   { onReady(); onError(String) }
interface StreamCallback { onChunk(String); onToolCall(String callId, String name, String argsJson); onComplete(String); onError(String) }

generate has two overloads; the 5-arg, toolsJson-carrying one is used when tools are registered. The StreamCallback has four methods — note onToolCall, which the bridge fires for each tool call; C# answers via provideToolResult. Image input is JPEG bytes from C#; the bridge decodes and re-encodes to PNG for LiteRT-LM. Streaming deltas are emitted via onChunk.

Backends

AndroidLlmTransport takes a backend string. Supported values map to LiteRT-LM backends:

Backend	Notes
`GPU`	OpenCL; silently falls back to CPU where OpenCL is unavailable. Most portable.
`CPU`	Always available, slowest.
`NPU`	Qualcomm Hexagon via QNN. Requires a matching NPU model export and a supported SoC/HTP arch. Falls back to CPU if init fails.

Choosing a backend. The repository ships with NPU as the default (set in AgentBuilder), targeting the Galaxy Z Fold 7 (Qualcomm SM8750 / Hexagon v79) NPU with an NPU-exported model — verified working (~25 tok/s, text + multimodal, streaming). NPU support is hardware-specific: it requires an NPU-exported .litertlm model and a matching Qualcomm SoC. If your target lacks a supported NPU, switch the backend to GPU (which falls back to CPU) in AgentBuilder and use the plain GPU/CPU .litertlm export instead of the NPU one.

Model provisioning (development)

For development, side-load the model and let the provisioner copy it into app-private storage on first run:

adb push gemma-4-E2B-it_qualcomm_sm8750.litertlm /data/local/tmp/

LiteRtModelProvisioner copies /data/local/tmp/gemma-4-E2B-it_qualcomm_sm8750.litertlm (LiteRtModelProvisioner.ModelFileName, the SM8750/Hexagon-v79 NPU export) into persistentDataPath/LiteRtModels/ on first launch and hands that path to the engine. If the file is absent it can download from Hugging Face (repo litert-community/gemma-4-E2B-it-litert-lm, revision main) using a runtime-supplied token (the HfToken field on the provisioner — never commit a token).

The model file name and backend are set in code today (LiteRtModelProvisioner.ModelFileName, AgentBuilder); make them configurable for your device/model.

Building the AAR

The prebuilt llm-release.aar ships inside the framework package (Runtime/Plugins/Android/), so there is no build step to run the sample — just resolve the transitive Maven deps via Assets ▸ External Dependency Manager ▸ Android Resolver ▸ Resolve.

The Kotlin/JNI source ships inside this package under AndroidBridge~/ (hidden from the importer by the trailing ~); there is no separate bridge repo. To rebuild the AAR, run ./gradlew assembleRelease (JDK 17) from AndroidBridge~/, then replace Runtime/Plugins/Android/llm-release.aar with the output. See AndroidBridge~/README.md for the build spec, JNI contract, and third-party notices.

Player Settings: IL2CPP, ARM64, minSdk 31.

Verifying on device

First launch: model-copy log, then an "engine ready" log.
Text prompt → streamed on-device reply (test in airplane mode to prove it is local).
Image prompt → the model describes the current camera frame (multimodal path).