| --- |
| license: other |
| pipeline_tag: text-generation |
| language: |
| - en |
| - ru |
| tags: |
| - android |
| - gguf |
| - llama-cpp |
| - on-device |
| - bonsai-27b |
| --- |
| |
| # Bonsai Local for Android |
|
|
| [Русский](README.md) · **English** · [Benchmarks / Замеры](BENCHMARKS.md) |
|
|
| Research Android application that runs |
| [Prism ML Bonsai-27B GGUF](https://huggingface.co/prism-ml/Bonsai-27B-gguf) |
| fully on-device through JNI and the PrismML `llama.cpp` fork. Once the GGUF is |
| copied to the device, chat inference does not use a PC server, cloud API, or |
| internet connection. |
|
|
|  |
|
|
| ## What works |
|
|
| - Loads the complete 3,803,452,480-byte `Bonsai-27B-Q1_0.gguf`. |
| - Performs inference inside the Android application process. |
| - Streams chat output and exposes the model's thinking trace. |
| - Auto-discovers GGUF files in private and app-specific external storage. |
| - Imports compatible GGUF files through Android Storage Access Framework. |
| - Includes an on-device prompt-processing/token-generation benchmark. |
| - Builds for physical `arm64-v8a` devices and `x86_64` emulators. |
| - Includes an adaptive Android launcher icon and raster mipmaps. |
| - Provides a debug-only Base64 UTF-8 prompt intent for reproducible tests. |
|
|
| The model is deliberately not embedded inside the APK because a 3.8 GB APK is |
| impractical. A verified copy is published in this repository at |
| `models/Bonsai-27B-Q1_0.gguf`, with upstream attribution preserved, so the app |
| and its exact tested model can be retrieved from one place. |
|
|
| ## Verified setup |
|
|
| | Component | Value | |
| |---|---| |
| | Host | Windows 11, Intel Core Ultra 5 125H | |
| | Android Studio | 2025.2 | |
| | AGP / Gradle | 8.13.2 / 8.14.3 | |
| | Build JDK | JetBrains Runtime 21.0.9 | |
| | compileSdk / targetSdk / minSdk | 36 / 36 / 30 | |
| | Android NDK / CMake | 28.2.13676358 / 3.22.1 | |
| | APK ABIs | `arm64-v8a`, `x86_64` | |
| | Test AVD | Android 17 preview, x86_64, 8 GB RAM, 16 GB data | |
| | PrismML fork commit | `62061f91088281e65071cc38c5f69ee95c39f14e` | |
| |
| ## Measured Android results |
| |
| All measurements below came from the Android emulator and the application JNI |
| runtime, not from desktop `llama-cli`. |
| |
| | Check | Result | |
| |---|---| |
| | Debug APK build, install, launch | passed | |
| | Recognized model | `qwen35 27B Q1_0`, 26.9B parameters, 3.53 GiB | |
| | Backend | CPU, dynamically selected x86_64 variant | |
| | Prompt processing, pp64 | **2.89 tokens/s** | |
| | Token generation, tg32 | **1.79 tokens/s** | |
| | Arithmetic prompt `37 × 19` | correct final answer: **703** | |
| | Thinking trace | operational | |
| | Offline inference after import | confirmed | |
| | Fatal exceptions | none observed | |
| |
| The Russian/Kotlin instruction-following test also produced a useful negative |
| result: the model understood the requested language, signature, and validation |
| constraints, but spent the complete 512-token limit in visible reasoning and |
| did not reach a clean final code answer. This is recorded rather than presented |
| as a successful result. |
| |
| Screenshots: |
| |
| - [`02-model-loaded.png`](screenshots/02-model-loaded.png) — final installed APK |
| with the local model loaded; |
| - [`03-benchmark.png`](screenshots/03-benchmark.png) — on-device benchmark; |
| - [`04-reasoning.png`](screenshots/04-reasoning.png) — live generation and 703; |
| - [`05-russian-kotlin.png`](screenshots/05-russian-kotlin.png) — 512-token |
| thinking-limit result. |
| |
| Emulator throughput should not be treated as a phone prediction. A physical |
| device has different SIMD support, thermal limits, memory bandwidth, and power |
| management. |
| |
| ## Architecture |
| |
| ```mermaid |
| flowchart TD |
| UI["Android UI · MainActivity"] --> API["InferenceEngine Kotlin API"] |
| API --> DISP["Single-thread coroutine dispatcher"] |
| DISP --> JNI["JNI · libai-chat.so"] |
| JNI --> COMMON["llama-common · chat template · sampler"] |
| JNI --> LLAMA["PrismML llama.cpp"] |
| LLAMA --> GGML["GGML CPU backend loader"] |
| GGML --> ABI{"Device ABI and CPU features"} |
| ABI --> ARM["ARM · NEON / DOTPROD / I8MM / SVE / SME"] |
| ABI --> X86["x86_64 · SSE4 / AVX2 / AVX512 / AMX"] |
| GGUF["Bonsai-27B-Q1_0.gguf · 3.80 GB"] --> LLAMA |
| ``` |
| |
| Project modules: |
|
|
| - `app/` — UI, model import, chat, benchmark, and Base64 test intent. |
| - `lib/` — Kotlin API, GGUF metadata reader, JNI, and native build. |
| - `third_party/llama.cpp/` — PrismML fork, kept next to this project when |
| building locally and not duplicated in this repository. |
| - `models/` — the exact GGUF used by the Android tests and its checksum. |
| - `screenshots/` — evidence captured from the Android emulator. |
| - `release/` — verified debug APK and checksum. |
|
|
| Model load flow: |
|
|
| 1. `MainActivity` waits for the native engine to initialize. |
| 2. It searches private and app-specific external `models` directories. |
| 3. Kotlin passes the absolute GGUF path to `loadModel()`. |
| 4. Native calls are serialized through one IO dispatcher. |
| 5. JNI creates an 8,192-token context, batch size 512, and sampler. |
| 6. GGML selects the best packaged CPU backend for the current ABI/features. |
| 7. The Qwen chat template is applied and token pieces stream as `Flow<String>`. |
|
|
| Critical JNI setup: |
|
|
| ```cpp |
| llama_model_params model_params = llama_model_default_params(); |
| g_model = llama_model_load_from_file(model_path, model_params); |
| |
| llama_context_params ctx_params = llama_context_default_params(); |
| ctx_params.n_ctx = 8192; |
| ctx_params.n_batch = 512; |
| ctx_params.n_threads = n_threads; |
| g_context = llama_init_from_model(g_model, ctx_params); |
| ``` |
|
|
| Native state is serialized because model, context, batch, and sampler are |
| global native resources: |
|
|
| ```kotlin |
| private val llamaDispatcher = Dispatchers.IO.limitedParallelism(1) |
| |
| override suspend fun loadModel(pathToModel: String) = |
| withContext(llamaDispatcher) { |
| load(pathToModel) |
| prepare() |
| } |
| ``` |
|
|
| ## Runtime capabilities verified |
|
|
| The real full-GGUF load reported: |
|
|
| - 64 transformer blocks and Q1_0 tensors; |
| - approximately 149.62 MiB recurrent state; |
| - approximately 523.02 MiB CPU compute buffer; |
| - automatic Flash Attention; |
| - fused Gated Delta Net in autoregressive and chunked paths; |
| - approximately 3,703 graph nodes and one split. |
| |
| Bonsai-27B is not a Mixture-of-Experts model. It is a dense 27B hybrid-attention |
| model, so there are no experts or routing network to enumerate. Roughly 75% of |
| its blocks use linear/recurrent attention and 25% full attention. |
| |
| This APK is text-only. The optional upstream vision projection |
| `Bonsai-27B-mmproj-Q8_0.gguf` and DSpark speculative drafter |
| `Bonsai-27B-dspark-Q4_1.gguf` are not integrated. |
|
|
| ## Build |
|
|
| Place the pinned PrismML fork next to this project: |
|
|
| ```text |
| gpt/ |
| ├── BonsaiAndroid/ |
| └── third_party/llama.cpp/ |
| ``` |
|
|
| Install SDK 36, NDK `28.2.13676358`, and CMake `3.22.1`, then open |
| `BonsaiAndroid` in Android Studio or run: |
|
|
| ```powershell |
| $env:JAVA_HOME = 'C:\Program Files\Android\Android Studio1\jbr' |
| java -classpath gradle\wrapper\gradle-wrapper.jar ` |
| org.gradle.wrapper.GradleWrapperMain :app:assembleDebug |
| ``` |
|
|
| The build output is `app/build/outputs/apk/debug/app-debug.apk`. The tested copy |
| is published as `release/BonsaiLocal-debug.apk`: |
|
|
| ```text |
| size: 120762843 bytes |
| SHA256: BDAF2D9EE7EE1BBB2A424242678D75AC35F2B770973E3F4C9E58639CE8F93E5C |
| ``` |
|
|
| The native runtime is wired through: |
|
|
| ```cmake |
| set(LLAMA_SRC ${CMAKE_CURRENT_LIST_DIR}/../../../../../third_party/llama.cpp) |
| add_subdirectory(${LLAMA_SRC} build-llama) |
| ``` |
|
|
| ## Install the model |
|
|
| Normal path: |
|
|
| 1. Copy `Bonsai-27B-Q1_0.gguf` to the Android device. |
| 2. Open Bonsai Local. |
| 3. Tap **Choose GGUF**. |
| 4. Select the file and wait for import/load. |
|
|
| Reproducible emulator path: |
|
|
| ```powershell |
| adb shell mkdir -p ` |
| /sdcard/Android/data/com.prismml.bonsailocal/files/models |
| adb push .\models\Bonsai-27B-Q1_0.gguf ` |
| /sdcard/Android/data/com.prismml.bonsailocal/files/models/ |
| ``` |
|
|
| Verified model checksum: |
|
|
| ```text |
| 17EF842E47450CAEB8EAA3EBFBBAB5D2F2278B62B79BE107985FB69A2F819AA0 |
| ``` |
|
|
| ## Problems encountered and fixes |
|
|
| 1. **No ready Android runtime in the collection.** The upstream Android sample |
| was adapted and the native runtime is built inside Gradle. |
| 2. **Mainline llama.cpp was insufficient.** Q1_0 and the hybrid architecture |
| require the pinned PrismML fork. |
| 3. **A separate JDK 17 toolchain was unavailable.** The project builds on JBR |
| 21 while explicitly targeting JVM bytecode 17. |
| 4. **Android logging API level mismatch.** A local priority filter was used and |
| minSdk was finalized at API 30. |
| 5. **The existing AVD had insufficient RAM/storage.** A dedicated 8 GB RAM, |
| 16 GB data AVD was created without modifying the user's existing Pixel AVD. |
| 6. **16 KB page-size checks.** NDK r28 and AGP 8.13.2 are used; ELF LOAD |
| alignment is `2**14` and `zipalign -P 16` passes. An unused DataStore bundle |
| and its `libdatastore_shared_counter.so` were removed. Android 17 preview may |
| still report experimental RELRO compatibility warnings for dynamically |
| loaded CPU variants; verify again on stable Android 15/16 before publishing |
| to production. |
| 7. **A 3.8 GB GGUF is easy to duplicate accidentally.** The test model was |
| pushed directly into app-specific external storage. |
| 8. **Thinking mode makes short CPU tests take minutes.** Generation remains |
| asynchronous, but production UI should separate/hide thinking and expose a |
| configurable token limit. |
| |
| ## Current limitations and next work |
| |
| - Text-only and CPU-only in the tested AVD. |
| - No resumable model downloader or in-app SHA-256 verification. |
| - Chat history is in-memory. |
| - Application context is currently 8,192 tokens. |
| - Thinking tags are displayed as normal text. |
| - 6 GB devices may be killed by Android LMK or run out of memory. |
| - Vision, Vulkan, compressed KV cache, and DSpark remain research tasks. |
| |
| Recommended next experiments: physical Snapdragon/Dimensity benchmarks, |
| first-token latency and thermal tests, Vulkan comparison, collapsible thinking |
| UI, WorkManager download/resume, vision `mmproj`, speculative decoding, and |
| per-ABI AAB delivery. |
| |
| ## Community value and related projects |
| |
| Running an LLM locally on Android is not itself novel. Official `llama.cpp` |
| already provides an Android Studio sample and runtime CPU-kernel selection; |
| PocketPal AI and ChatterUI are mature GGUF clients; MLC LLM, MNN, and |
| ExecuTorch provide alternative Android runtimes and demos. |
| |
| | Project | Existing capability | Difference in this work | |
| |---|---|---| |
| | [llama.cpp Android](https://github.com/ggml-org/llama.cpp/blob/master/docs/android.md) | official JNI/sample and CPU variants | validates this unusual Q1_0 hybrid model and publishes a tested APK | |
| | [PocketPal AI](https://github.com/a-ghorbani/pocketpal-ai) | mature GGUF client, HF downloads, benchmarks | general product versus a narrow reproducible Bonsai-27B test case | |
| | [ChatterUI](https://github.com/Vali-98/ChatterUI) | GGUF and API chat through React Native | this project keeps a minimal native Android/JNI path | |
| | [MLC LLM](https://llm.mlc.ai/docs/deploy/android.html) | GPU-oriented Android SDK and demo | different format/toolchain; this work consumes the original GGUF on CPU | |
| | [MNN](https://github.com/alibaba/MNN/tree/master/apps/Android/MnnLlmChat) | high-performance multimodal Android client | far broader and faster; this repository is simpler as a Q1_0/GDN regression fixture | |
| | [ExecuTorch](https://docs.pytorch.org/executorch/stable/llm/run-on-android.html) | AAR and experimental Java LLM API | exports to `.pte`; this project documents the GGUF/llama.cpp route | |
| |
| A public Hugging Face and GitHub search at publication time did not reveal |
| another reproducible package for **Bonsai-27B Q1_0 running inside an Android |
| APK**. This does not prove absolute priority, but the useful combination is: |
| |
| - the exact GGUF, APK, source, and checksums live together; |
| - real Android pp/tg measurements, runtime paths, and screenshots are recorded; |
| - Android preview 16 KB page-size and RELRO issues are documented; |
| - both a correct reasoning result and a failed Kotlin instruction test are kept; |
| - the research notes are available in English and Russian. |
| |
| The current result is best described as an **engineering baseline and regression |
| artifact**, not a new model architecture or a production competitor to |
| PocketPal/MNN. Physical ARM64 Snapdragon/Dimensity results, RAM/energy/ |
| first-token measurements, CI builds, and upstream Android fixes would turn it |
| into a substantially stronger community contribution. |
| |
| ## Licenses |
| |
| - Bonsai-27B GGUF: Apache-2.0 according to the upstream model card. |
| - PrismML fork / llama.cpp: see the upstream repository licenses. |
| - Preserve all applicable upstream `LICENSE` and `NOTICE` files when |
| redistributing the APK or model. |
| |
