Instructions to use litert-community/Phi-4-mini-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Phi-4-mini-instruct with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli litert-lm run \ --from-huggingface-repo=litert-community/Phi-4-mini-instruct \ --prompt="Write me a poem"
- Notebooks
- Google Colab
- Kaggle
Add measured Galaxy S26 GPU-vs-CPU rows for the .litertlm bundle
#2
by mlboydaisuke - opened
README.md
CHANGED
|
@@ -99,3 +99,14 @@ Note that all benchmark stats are from a Samsung S24 Ultra with
|
|
| 99 |
* Benchmark is done assuming XNNPACK cache is enabled
|
| 100 |
* Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ.
|
| 101 |
* dynamic_int8: quantized model with int8 weights and float activations.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
* Benchmark is done assuming XNNPACK cache is enabled
|
| 100 |
* Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ.
|
| 101 |
* dynamic_int8: quantized model with int8 weights and float activations.
|
| 102 |
+
|
| 103 |
+
### Galaxy S26 β `.litertlm` bundle, GPU vs CPU (litert-lm 0.16.0)
|
| 104 |
+
|
| 105 |
+
Community measurement of the `Phi-4-mini-instruct_multi-prefill-seq_q8_ekv4096.litertlm` file β a different runtime path and artifact than the S24 Ultra table above, so compare within one table, not across. Physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16), `litert_lm_advanced_main` from the litert-lm v0.16.0 release, GPU backend OpenCL (`LITERT_CL`). One fixed 205-token prompt text (201 tokens under this tokenizer), `--benchmark`. Two runs per backend taken back-to-back β cells show the range (the CPU's first run also pays cold file I/O on the 3.9 GB bundle). Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both produced a correct text answer.
|
| 106 |
+
|
| 107 |
+
| Backend | Prefill (201 tok) | Decode | Time-to-first-token | Init | Peak RSS |
|
| 108 |
+
|---|---|---|---|---|---|
|
| 109 |
+
| **GPU (OpenCL)** | **195β196 tok/s** | **10.6 tok/s** | **1.12β1.13 s** | 5.9β7.9 s | **1447 MB** |
|
| 110 |
+
| CPU (XNNPACK) | 78β128 tok/s | 6.8β6.9 tok/s | 1.72β2.72 s | 4.8β5.0 s | 5661 MB |
|
| 111 |
+
|
| 112 |
+
The GPU takes the whole graph β decode 1648/1648 ops and all four prefill signatures 1677/1677 on `LITERT_CL`. It wins everywhere on this bundle: prefill 1.5β2.5Γ, decode 1.5Γ, and peak RSS 3.9Γ lower (1447 against 5661 MB) β on a 4 GB-class model the memory difference is what decides whether the phone stays comfortable.
|