ARCHITECTURE.md

# UMoX architecture

## Definition

**UMoX (Unified Mixture of Executors)** treats heterogeneous processors as
execution specialists. It borrows the organizational idea of a model-side
mixture of experts without pretending the hardware engines are interchangeable
tensor units.

The admitted prototype is task-parallel:

| Lane | Current engine/model | Responsibility |
|---|---|---|
| Primary | Adreno GPU / Ornith 35B Q4_K_M | planning, difficult reasoning, coding, final synthesis |
| Utility | Hexagon NPU / Qwen3 4B Instruct | bounded summarization, classification, extraction, rewriting |
| Control/tool | Oryon CPU | routing, MCP/tools, retrieval, filesystem, validation, transport |

The models remain resident and share the platform's unified memory system, but
they do not recompute the same request. Independent work overlaps so that the
NPU removes safe utility demand from the GPU queue.

## Request lifecycle

1. A loopback OpenAI-compatible endpoint receives the request.
2. A deterministic classifier inspects request shape, explicit task tags,
   estimated input size, and static safety markers.
3. Tools, structured output, high-risk markers, and default/unknown work select
   the GPU lane.
4. Only allowlisted task tags below the configured input cap select the NPU.
5. For tool-free requests, retryable NPU failure may fall back to the GPU before
   the first body fragment is committed.
6. Once streaming begins, the route never changes. Stream failure terminates
   honestly rather than silently producing a mixed answer.
7. Logs contain route metadata and timings, never prompt or response bodies.

## Safety invariants

- GPU is the default.
- Token length can disqualify NPU routing but cannot qualify it.
- Model-generated content cannot select a route.
- Tools always use the GPU lane.
- High-risk vocabulary outranks explicit routing overrides.
- Unsafe overrides are disabled by default and unmistakably labeled when
  enabled in a development configuration.
- Loopback binding is mandatory for the admitted topology.
- No retry occurs after response commitment.

These controls are part of the measured architecture. A future learned router
may advise scheduling, but it cannot replace the hard safety boundary.

## Why direct layer splitting is not the admitted path

The research program measured three direct NPU lanes:

| Attempt | Result |
|---|---|
| Batch-1 decode components/layers | Closed: transfer/session boundaries exceeded the token-path budget |
| Prompt prefill offload | Closed through 32K context: qualified GPU prompt processing remained faster |
| NPU speculative verifier | Closed under the registered acceptance regime: per-layer session overhead dominated |

The work produced useful kernels, provenance tooling, numerical gates, and
prompt-processing maps, but it did not earn a place in production inference.
Those conclusions remain conditional on the measured software stack; a future
fused persistent runtime could reopen them with new evidence.

## Where the CPU contributes

The CPU is not a spare transformer engine. It is the control plane:

- filesystem and repository scans;
- token counting and prompt assembly;
- retrieval and context compression;
- MCP/tool execution;
- schema and policy validation;
- queueing, cancellation, backpressure, health, and recovery.

Offloading ordinary 35B layers to the CPU would compete for shared memory
bandwidth and add synchronization. Removing non-model work from accelerator
critical paths is usually more valuable.

## Scaling beyond the laptop

The planned topology adds an optional deep lane on a workstation reachable over
an authenticated private network:

- local NPU: fast utility work;
- local GPU: normal 35B reasoning;
- remote GPU: 70B+ or expensive visual/deep work;
- local CPU: policy, tools, validation, and failover.

Remote failure must leave the local two-model system functional. No remote lane
will be described as production-ready until latency, privacy, authentication,
and local-fallback gates pass.