# On-device LLM on Android (LiteRT-LM)

By default the agent runs its LLM through [Ollama](https://ollama.com) on desktop/Editor. On Android,
it can instead run the LLM **fully on-device** via the [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)
runtime, behind the same `IAgentTransport` interface. The transport is chosen at compile time:

```csharp
// AgentBuilder.PrepareSetup()
#if UNITY_ANDROID && !UNITY_EDITOR
    IAgentTransport transport = new AndroidLlmTransport(dispatcher, "NPU", Log);
#else
    IAgentTransport transport = new OllamaTransport(Endpoint, Model, options.AgentName, Log);
#endif
```

## Components

```
AndroidLlmTransport.cs  (C#, IAgentTransport)            Runtime/AgentCore/Runtime/
   ├─ LiteRtModelProvisioner.cs   places the .litertlm model in persistentDataPath/LiteRtModels/
   └─ AndroidJavaObject → LlmBridge (JNI)                Runtime/Plugins/Android/llm-release.aar
                          └─ LiteRT-LM Engine            built from the in-package AndroidBridge~/ source
```

- `AndroidLlmTransport` marshals all JNI calls onto the Unity main thread via
  `IUnityMainThreadDispatcher` and wraps the bridge's async callbacks in `Task`s.
- The LiteRT-LM `Engine` is large (2–3 GB resident) and is treated as a per-process singleton.
- A fresh conversation is created per request; the C# agent layer owns history and the prompt it
  sends (see [Tool calling and context budgeting](#tool-calling-and-context-budgeting)), avoiding
  unbounded native KV growth.

## Tool calling and context budgeting

The transport uses **the bridge's native tool template with a manual, callback-driven loop**
(`RunNativeToolLoopAsync`) — not a prompt-based two-stage protocol. There is no `select_tool` step and
no schema-withholding:

- **Native OpenAPI tools, manual loop.** `AndroidLlmTransport` builds a JSON array of **full** OpenAPI
  tool specs — `[{name, description, parameters: {schema}}]`, the parameter schemas are **not** withheld
  — and passes it to the bridge's tool-aware `generate(...)` (see [JNI contract](#jni-contract)). The
  bridge feeds the model's own tool template, the model emits a structured tool call, and the bridge
  fires `onToolCall(callId, name, argsJson)`. C# runs the tool and replies via
  `provideToolResult(callId, result)`; the whole loop runs inside one `bridge.generate()` call.
  LiteRT-LM's fully-automatic `automaticToolCalling` path is deliberately **not** used — it deadlocks
  after tool-result injection (`callback_thread_pool DEADLINE_EXCEEDED`, zero second-pass decode).
- **Single-engine defer-and-resume (vision / RAG).** There is one process-wide LiteRT-LM engine, and it
  cannot run a nested `generate()` while one is in flight (`"busy generating"`). So a tool that itself
  needs the LLM mid-turn cannot make a nested call: instead it **defers**. A vision tool stashes the
  captured camera frame; the `SearchKnowledge` RAG tool (which needs its own LLM call to translate the
  query to English, plus any graph-mode keyword extraction) stashes its arguments and answers the bridge
  with a sentinel so the turn ends. `RunNativeToolLoopAsync` then **resumes** after the outer turn
  completes, answering the stashed vision frame with one multimodal `generate`, or running RAG retrieval
  + answer, once the engine is free. The model still decides when to look or search; only the call is
  deferred.
- **Knowledge queries always search in English.** The bundled knowledge base is authored in English, so
  `SearchKnowledge` translates a non-English query to English before retrieving (`ResolveSearchQueryAsync`
  in `SearchKnowledgeTool`, gated by `IsLikelyNonEnglish` so an already-English query skips the round
  trip). The translation is its own `generate()` call — on Android it runs at **resume** time, when the
  engine is free (the defer-and-resume above is what makes this nested-free), wired via
  `BuiltinToolDependencies.TranslateToEnglish` (set up by the host app). Before the search runs the tool
  emits a one-line note — `🌐 <원문> → <english>` — through `OnToolNote`, which `AlexaVoiceView` renders
  as a persistent block in the result view so the translation step is visible (the same intermediate-state
  surface the rest of capture-and-resume uses). If no translator is wired the search falls back to the
  original query — the multilingual-e5 embedder still matches cross-lingually.
- **Context budgeting.** The Android context budget is **3072 tokens** (set by the host app),
  deliberately lower than LiteRT-LM's `MAX_NUM_TOKENS = 4096` session cap (`LlmBridge.kt`). The lower
  budget makes memory compaction (`CompactionTriggerPercent = 95`) fire **before** the prompt + tool
  result can overflow the native window, which is what would otherwise make decoding break down after a
  few turns.

## JNI contract

The AAR must expose exactly this (the C# side depends on the names):

```
initialize(String modelPath, String backend, InitCallback)
generate(String prompt, byte[] jpeg /* nullable, JPEG bytes */, StreamCallback)                       // text/multimodal, no tools
generate(String systemPrompt, String prompt, byte[] jpeg, String toolsJson, StreamCallback)           // tool-aware overload
provideToolResult(String callId, String result)
interface InitCallback   { onReady(); onError(String) }
interface StreamCallback { onChunk(String); onToolCall(String callId, String name, String argsJson); onComplete(String); onError(String) }
```

`generate` has two overloads; the 5-arg, `toolsJson`-carrying one is used when tools are registered.
The `StreamCallback` has four methods — note `onToolCall`, which the bridge fires for each tool call;
C# answers via `provideToolResult`. Image input is JPEG bytes from C#; the bridge decodes and
re-encodes to PNG for LiteRT-LM. Streaming deltas are emitted via `onChunk`.

## Backends

`AndroidLlmTransport` takes a backend string. Supported values map to LiteRT-LM backends:

| Backend | Notes |
|---------|-------|
| `GPU`   | OpenCL; **silently falls back to CPU** where OpenCL is unavailable. Most portable. |
| `CPU`   | Always available, slowest. |
| `NPU`   | Qualcomm Hexagon via QNN. Requires a matching NPU model export and a supported SoC/HTP arch. Falls back to CPU if init fails. |

> **Choosing a backend.** The repository ships with **`NPU` as the default** (set in `AgentBuilder`),
> targeting the **Galaxy Z Fold 7 (Qualcomm SM8750 / Hexagon v79)** NPU with an NPU-exported model —
> verified working (~25 tok/s, text + multimodal, streaming). NPU support is hardware-specific: it
> requires an NPU-exported `.litertlm` model and a matching Qualcomm SoC. If your target lacks a
> supported NPU, switch the backend to `GPU` (which falls back to CPU) in `AgentBuilder` and use the
> plain GPU/CPU `.litertlm` export instead of the NPU one.

## Model provisioning (development)

For development, side-load the model and let the provisioner copy it into app-private storage on
first run:

```bash
adb push gemma-4-E2B-it_qualcomm_sm8750.litertlm /data/local/tmp/
```

`LiteRtModelProvisioner` copies `/data/local/tmp/gemma-4-E2B-it_qualcomm_sm8750.litertlm`
(`LiteRtModelProvisioner.ModelFileName`, the SM8750/Hexagon-v79 NPU export) into
`persistentDataPath/LiteRtModels/` on first launch and hands that path to the engine. If the file is
absent it can download from Hugging Face (repo `litert-community/gemma-4-E2B-it-litert-lm`, revision
`main`) using a runtime-supplied token (the `HfToken` field on the provisioner — never commit a
token).

The model file name and backend are set in code today (`LiteRtModelProvisioner.ModelFileName`,
`AgentBuilder`); make them configurable for your device/model.

## Building the AAR

The prebuilt `llm-release.aar` ships **inside the framework package**
(`Runtime/Plugins/Android/`), so there is no build step to run the sample — just resolve the
transitive Maven deps via **Assets ▸ External Dependency Manager ▸ Android Resolver ▸ Resolve**.

The Kotlin/JNI source ships **inside this package** under `AndroidBridge~/` (hidden from the importer
by the trailing `~`); there is no separate bridge repo. To rebuild the AAR, run `./gradlew
assembleRelease` (JDK 17) from `AndroidBridge~/`, then replace `Runtime/Plugins/Android/llm-release.aar`
with the output. See `AndroidBridge~/README.md` for the build spec, JNI contract, and third-party
notices.

Player Settings: IL2CPP, ARM64, minSdk 31.

## Verifying on device

1. First launch: model-copy log, then an "engine ready" log.
2. Text prompt → streamed on-device reply (test in airplane mode to prove it is local).
3. Image prompt → the model describes the current camera frame (multimodal path).