| # Running on a cell phone (on-device or thin-client) |
|
|
| "Runs on a cell phone" can mean two things; the app supports both via one env switch. |
|
|
| ## The inference switch |
|
|
| `server/model.py` reads `INFERENCE_BASE_URL`: |
|
|
| - **Unset (default):** the GGUF is loaded in-process via `llama-cpp-python` (the Space / a laptop). |
| - **Set:** generation is delegated to a remote **OpenAI-compatible / llama.cpp server** at that URL. |
| Same agent code, different inference location. |
|
|
| ```bash |
| export INFERENCE_BASE_URL="http://127.0.0.1:8080/v1" # a llama-server on the phone |
| export INFERENCE_API_KEY="..." # optional |
| export INFERENCE_MODEL="gemma-e4b" # optional label |
| ``` |
|
|
| So "on the phone" = run a `llama-server` **on the device** and point the agent at `127.0.0.1`. |
|
|
| ## On-device model profile (Gemma E4B edge) |
|
|
| A 31B Q4 GGUF (~18–20 GB) needs a GPU and will not run on a phone. Use the lightweight **Gemma E4B** |
| edge variant (see [PLAN.md](../PLAN.md) and the README *Accuracy upgrade* section), with a small |
| context window: |
|
|
| ```bash |
| export MODEL_REPO="<your-or-community gemma E4B GGUF repo>" |
| export MODEL_FILE="<gemma-e4b-*-Q4_K_M.gguf>" |
| export N_CTX=4096 # keep the KV cache small on a phone |
| export N_GPU_LAYERS=0 # CPU; on a Mac use Metal layers instead |
| ``` |
|
|
| ## Android (Termux) — genuinely on-device |
|
|
| ```bash |
| pkg install python git cmake clang |
| git clone <this repo> && cd imessage-calendar-agent |
| pip install -r requirements-ci.txt llama-cpp-python # CPU build |
| # Option 1: run the whole app (UI + /agent) on the phone |
| USE_STUB_EXTRACTOR=0 python app.py # http://127.0.0.1:7860 |
| # Option 2: run only a llama-server and point a client/app at it |
| # llama-server -m <gemma-e4b.gguf> --port 8080 |
| # then set INFERENCE_BASE_URL=http://127.0.0.1:8080/v1 |
| ``` |
|
|
| Expect multi-second latency per request on phone CPU — keep `N_CTX` small and threads short. |
|
|
| ## iOS — the honest limit |
|
|
| iOS does **not** allow background message access or a persistent background LLM server. You cannot |
| run an autonomous on-device agent for iMessage on an iPhone. The supported iOS path is the |
| foreground **Shortcut** in [automations.md](./automations.md), optionally pointing at a remote |
| `INFERENCE_BASE_URL` for the model. |
|
|
