ales27pm's picture
Use portable shell invocations
c786a70 verified
|
Raw
History Blame Contribute Delete
12.7 kB
# Dolphin Assistant for iPhone
Dolphin Assistant is a private, on-device iOS 18 assistant powered by the
stateful INT4 Dolphin 3.0 Core ML model. It can plan a short sequence of local
actions, ask for approval before sensitive work, execute registered tools, and
use the results to answer in the same conversation. Chats, typed knowledge,
context digests, session handoffs, memories, tasks, settings, and the activity
audit stay in the app's local container.
This is **bounded foreground autonomy**. A run follows an
observe → plan → approve → execute → record → reassess loop while the app is
active. iOS may suspend the process after it enters the background, so this is
not an indefinitely running background agent.
## What it can do
The assistant has one canonical registry of local tools:
- read the current date and time;
- evaluate arithmetic with a constrained calculator;
- search or save durable legacy memories;
- capture, search, and inspect typed project knowledge with exact provenance;
- synthesize bounded project context or a deterministic evidence-backed project
review;
- create, list, inspect, restore, and deactivate typed session handoffs without
replaying actions or deleting the checkpoint;
- derive deterministic continuity insights;
- inspect a bounded code snippet with non-executing static heuristics;
- list, search, add, complete, and reopen local tasks; and
- read a bounded list of Calendar event titles and times for today, tomorrow,
or the next seven days after explicit opt-in, iOS full-access authorization,
and one-time approval for the exact read.
The Chat tab is the command surface. Activity groups each durable run with its
timeline, counters, receipts, failures, and cancellation state. Knowledge
provides Knowledge, Handoffs, and Tasks sections for manual capture, local
hybrid lexical search, deterministic synthesis, proactive insights, continuity
packages, legacy memories, and directly completable tasks. Settings uses an
explicit draft-and-save flow for run limits and the system prompt, and
separately owns model lifecycle, resource diagnostics, and protected
system-context consent.
The **Ask Dolphin**, **Continue Dolphin**, and **Capture Context** App Shortcuts
open the app with a sanitized request placed in the composer for review. They
never send the request, load the model, invoke a tool, restore a handoff, save
knowledge, or request permission on their own.
The default policy is intentionally conservative:
- local writes require approval;
- Calendar is disabled until the user enables it in Dolphin Settings and iOS
grants full access; each exact bounded read still requires approval once;
- external network tools are unavailable until the connected peer can be
validated before a request is sent;
- unknown tools and malformed arguments are rejected;
- exact normalized calls and bounded observations are retained as structured
Activity receipts rather than copied into conversational text; local-write
receipts are committed atomically with their changes, and partial side
effects remain visible after a stop or failure;
- approval is tied to one exact action and argument payload, expires after ten
minutes, and is consumed once;
- duplicate tool calls are rejected within a run;
- deterministic commands declare their complete ordered action list and must
produce a receipt for every planned call before the run can succeed; these
plans can run without loading the language model because they generate no
model turn;
- broad model-assisted routing exposes read tools only; local writes require an
explicit deterministic command and normal approval;
- tool output is length-bounded and treated as untrusted data; and
- each run is limited to three tool steps by default, with cancellation
requested at a 300-second deadline and a 256-token response budget.
Continuity deletions show their cascade before removing linked handoffs or
digests. The original request and receipts remain auditable until the separate
**Clear Chat & Run Activity** action removes messages, run records, run-scoped
Activity, and copied recent-chat excerpts from retained handoffs.
## Prepare the model and project
The checked-in Xcode project is generated from `project.yml`. Prepare the
pinned model and regenerate the project with:
```bash
cd examples/ios/DolphinCoreMLTestApp
zsh prepare-model.sh
```
The script downloads
`ales27pm/Dolphin3.0-CoreML@v2.0.0`, verifies all three package-file SHA-256
hashes and the five tokenizer/config hashes, and runs XcodeGen when it is
installed. Install XcodeGen with `brew install xcodegen` if necessary, or
regenerate explicitly:
```bash
xcodegen generate
open DolphinCoreMLTestApp.xcodeproj
```
The app target remains `DolphinCoreMLTestApp`, its bundle identifier remains
`com.ales27pm.DolphinCoreMLTestApp`, and the installed display name is
**Dolphin Assistant**.
## Build and run
In Xcode:
1. Select the `DolphinCoreMLTestApp` target.
2. Choose your Apple Developer Team under Signing & Capabilities.
3. Select an iPhone running iOS 18 or newer, or a suitable simulator for UI
work.
4. Choose Product → Run.
5. Explicit local commands can run before model load. In Settings, load the
model and wait for the ready state before starting a generated conversation.
6. Optional: use **Grant and Enable Calendar** in Settings. The permission
prompt can originate only from this button, never from an agent run.
For a command-line compile check:
```bash
xcodebuild \
-project DolphinCoreMLTestApp.xcodeproj \
-scheme DolphinCoreMLTestApp \
-configuration Debug \
-destination 'generic/platform=iOS Simulator' \
build
```
Run the agent-logic and protocol tests on the macOS host without launching a
Simulator or compiling and embedding the 1.81 GB model package:
```bash
zsh scripts/test-agent-logic.sh
```
The equivalent direct command is:
```bash
swift test \
--package-path . \
--scratch-path DerivedDataHostTests
```
The unit tests cover canonical JSON/signature generation, strict parsing of the
model's `<tool_call>` envelope, exact argument normalization, the production
tool surface, whole-turn context trimming, structured receipt compatibility,
oversized-result rejection, deterministic read-observation grounding, durable
run-checkpoint migration and recovery, and mixed-result fail-closed behavior.
They verify natural-language multi-step planning, plan-limit refusal before any
execution, no-model deterministic execution, receipt correlation before trust,
and rejection or removal of hidden-reasoning output.
They also cover Calendar privacy minimization and authorization denial, intent
handoff sanitization and one-shot consumption, persistence schema
compatibility, resource preflight policy, typed knowledge bounds and hybrid
lexical ranking, deterministic synthesis and insights, handoff lifecycle,
schema-v5 continuity round trips, grounded continuity-tool answers, and bounded
non-executing code analysis. The host package excludes SwiftUI, app lifecycle,
Core ML inference, and App Intents metadata generation.
These tests exercise app logic, not Core ML inference. A successful compile or
unit-test run is not proof of model-backed behavior; verify an actual model run
and inspect Activity for the corresponding model and tool evidence.
## Runtime design
Dolphin's native tool protocol is rendered directly into the prompt. Each
model turn starts with a fresh Core ML state. Prompt prefill is split into
chunks of at most 512 tokens while reusing that state within the turn, unlocking
the model's full 2,048-token state capacity for prompt plus generated tokens.
Greedy decoding favors stable tool-call JSON, and public conversation history is
trimmed to fit the context budget. Scratch tool exchanges are scoped to the
current run; private chain-of-thought is neither requested nor persisted.
User text, tool observations, and reconstructed tool-call arguments have all
tokenizer control-token sequences neutralized before they re-enter a prompt.
Explicit memory and task writes are routed deterministically through the same
approval and receipt path, while ordinary conversation receives no irrelevant
tool definitions. Legacy receipt prose from earlier builds is removed from
saved chat during workspace migration so it cannot contaminate later prompts.
Explicit multi-action requests joined with **then** become a bounded sequential
plan. The plan records tool names and required receipt identities—not private
chain-of-thought—and appears in Activity. Once every deterministic step has a
trusted receipt, Dolphin renders the answer directly from those receipts with
zero follow-up model turns. A fail-closed tool-only runtime lets this path work
without loading the 1.81 GB Core ML model; an accidental generation request is
rejected. Missing tools, denied actions, missing receipts, and plans over the
configured step limit fail closed. Complete hidden-reasoning blocks are removed
from model output, while malformed or reasoning-only output is rejected as an
empty response.
The schema-v5 workspace adds typed knowledge, deterministic digests, session
handoffs, and active-handoff selection. Saving a legacy memory also mirrors the
same content into typed knowledge when capacity permits. Search uses bounded,
deterministic BM25-like, phrase, coverage, and trigram signals with stable tie
breaking; it is hybrid lexical retrieval, not vector or embedding search. A
digest is derived after every fifth captured knowledge item, and restored
handoffs contribute context only—they cannot replay tools or approvals.
Stop requests are observed between Core ML predictions. Cancellation or failure
discards the current state, and overlapping runs are prevented. The app cancels
foreground work when it moves to the background rather than claiming work that
iOS may suspend.
Protected Calendar reads have four independent gates: explicit app opt-in, iOS
full-access authorization, approval for the normalized request, and an active
foreground scene before and after EventKit access. Only sanitized titles,
start/end times, and all-day flags cross the system-tool boundary; identifiers,
notes, attendees, URLs, and locations never do.
Each request also has a durable run record and checkpoint. The journal records
the current phase, active tool call, completed receipts, counters, and terminal
status. If iOS ends the process during a run, the next launch preserves that
evidence and marks the run interrupted instead of pretending it completed or
silently replaying an action. Local mutations and their corresponding receipts
remain one atomic persistence operation.
## Limitations
- The package is about 1.81 GB. Its fixed FP16 KV caches use roughly 224 MiB
before model weights, activations, and runtime buffers. Use a recent,
high-memory iPhone with several gigabytes of free storage.
- The context window is 2,048 tokens total, including tool definitions,
conversation, tool observations, and generated output. Older history may be
omitted from a turn to stay within that limit.
- Generation is deliberately bounded and can be slow on older devices. A stop
cannot interrupt a Core ML prediction already in progress.
- Direct networking is intentionally unavailable in this build. URL syntax and
a preliminary DNS lookup are not enough to prevent DNS rebinding; a future
network tool must pin and validate the actual connected peer before sending.
- Tools operate only on the assistant's local store and explicitly approved
Calendar context. The app does not yet control Mail, read Contacts or
Location, execute arbitrary Shortcuts, or access other system services.
- Create State-inspired continuity is an independent, single-workspace local
implementation. It has no Create State account or compatibility layer,
embeddings, cloud sync, repository/media imports, team graph or RBAC,
background monitoring, or hosted MCP dependency. See
[CREATE_STATE_INTEGRATION.md](CREATE_STATE_INTEGRATION.md) for the immutable
public-source inventory and exact parity boundary.
- Local persistence is app-container storage, not a synced account or an
encrypted secrets vault. Deleting the app deletes that data unless it is
otherwise backed up by iOS. If the store cannot be opened or decoded, the
assistant fails closed instead of presenting volatile data as saved.
- Conversational responses and model-assisted read selection can be wrong.
Deterministic plans remain narrow host rules, and verified tool observations
remain the source of truth. The approval boundary reduces side effects; it
does not make model output factual.