request_lifecycle.md

QualEdge: Query Lifecycle & Silicon Execution Architecture

This document describes the step-by-step lifecycle of a user query through the QualEdge hybrid edge-cloud router down to the processor silicon (Hexagon NPU vs. Kryo CPU), showing what optimizations solve quantization errors and what telemetry metrics are reported.

---

🗺️ Execution Sequence Flow

sequenceDiagram
    autonumber
    actor User as User / App
    participant Router as Hybrid Router (FastAPI)
    participant Classifier as TF-IDF & Logistic Regression
    participant LocalEngine as Local Inference (Qwen-0.5B)
    participant NPU as Hexagon NPU (HTP)
    participant CPU as Kryo CPU (Fallback)
    participant Verifier as Self-Verification Checker
    participant Cloud as Cloud Engine (Claude/Gemini)

    User->>Router: POST /api/router/route (Query prompt)
    
    rect rgb(15, 23, 42)
        Note over Router, Classifier: Pre-Processing & Routing Gate
        Router->>Classifier: Vectorize query prompt
        Classifier->>Classifier: Compute TF-IDF weights
        Classifier->>Classifier: Calculate complexity probability P(complex)
        Classifier-->>Router: Return P(complex) & pathway decision
    end

    alt P(complex) <= Threshold (Route On-Device)
        rect rgb(12, 74, 96)
            Note over Router, LocalEngine: Local Edge Inference
            Router->>LocalEngine: Ingest prompt & initialize model graph
            
            Note over LocalEngine, NPU: NPU Native Execution (Optimized with CLE & AdaRound)
            LocalEngine->>NPU: Offload INT8 Conv/GEMM operators (0.08 Joules)
            
            Note over LocalEngine, CPU: CPU Fallback Execution (Unmapped Operators)
            LocalEngine->>CPU: Run LayerNorm/Softmax on Kryo CPU (2.10 Joules penalty)
            
            LocalEngine-->>Router: Return generated text output
        end

        rect rgb(20, 83, 45)
            Note over Router, Verifier: Self-Verification check
            Router->>Verifier: Evaluate generated text for degradation
            Verifier->>Verifier: Calculate token repetition frequency
            
            alt Repetition < 40% & Not Null (Verification PASSED)
                Verifier-->>Router: Status: VERIFIED
                Router-->>User: Final Output + CLI Telemetry (NPU Energy, 0$ Cloud Cost)
            else Repetition >= 40% or Null (Verification FAILED - Local Model Collapse)
                Verifier-->>Router: Status: FAILED
                Note over Router, Cloud: Cloud Escalation Cascade (AutoMix)
                Router->>Cloud: Forward query to Claude/Gemini API
                Cloud-->>Router: Return high-quality cloud text
                Router-->>User: Final Output + CLI Telemetry (Escalated, Cloud Latency, $0.0055 Cost)
            end
        end

    else P(complex) > Threshold (Route Cloud Direct)
        rect rgb(15, 23, 42)
            Note over Router, Cloud: Direct Cloud Path
            Router->>Cloud: Forward query to Claude/Gemini API
            Cloud-->>Router: Return cloud response text
            Router-->>User: Final Output + CLI Telemetry (Direct Cloud, $0.0055 Cost)
        end
    end

---

🔍 Detailed Lifecycle Stages

Stage 1: Ingestion & Feature Extraction (TF-IDF)

When the user submits a text query, the request hits the /api/router/route endpoint. Before passing the query to any neural network, it is pre-processed using a sub-1ms TF-IDF Vectorizer. This extracts semantic keyword features (e.g. keywords like "scrapes", "write", "script" indicate high reasoning complexity, whereas "capital", "what is" indicate simple lookup tasks).

Stage 2: Decision Routing Gate (Logistic Regression)

The extracted features are input to a Logistic Regression classifier that outputs the probability $P(\text{complex})$.

• If $P(\text{complex}) \le \theta$ (classification threshold, default 0.5), the query is routed to the on-device path.
• If $P(\text{complex}) > \theta$, the query bypasses local execution and goes directly to the cloud, preventing local model failure on complex prompts.

Stage 3: Edge NPU Execution & Operator Graph Allocation

When executing on-device, the model graph is partitioned:

1. Hexagon Tensor Processor (HTP - NPU): Handles the core weight-matrix multiplies.
   • What We Solved: Naive INT4/INT8 quantization causes accuracy degradation. We applied Cross-Layer Equalization (CLE) to balance convolution weight channels and AdaRound to dynamically calculate optimal weight rounding values per layer, maintaining Top-1 accuracy.
   • Silicon Draw: Drew only ~0.08 Joules of energy per query on native HTP silicon.
2. Kryo CPU (Fallback): General-purpose CPU handles layers that the NPU compiler lacks kernel maps for (e.g. specialized normalization, resizing, or custom activation layers).
   • The Penalty: Moving tensors back-and-forth from NPU to CPU cache triggers an energy penalty of ~2.10 Joules per query.

Stage 4: Output Self-Verification Check

Quantized edge models (like Qwen-0.5B) are prone to quantization collapse (loops of repetitive tokens, e.g. "tokyo tokyo tokyo..." or empty strings) when facing challenging inputs.

• Our Solution: A self-verification checker evaluates the generated output in real-time. If the token repetition rate exceeds 40%, it rejects the local output.

Stage 5: Cloud Escalation Cascade (AutoMix)

If the verification checker fails the local response, the router automatically spins up a cloud client fallback, sending the query to the Cloud Inference Engine (Gemini/Claude) and recovering response quality seamlessly for the user.

---

💻 CLI Telemetry Output (What You See)

When a query is processed, the system prints the exact execution trace and silicon metrics to the command line.

Scenario A: Local NPU Execution (Success)

[ROUTER] Ingesting query: "What is the capital of Japan?"
[ROUTER] Pre-processing TF-IDF features... (Time: 0.15ms)
[ROUTER] Routing Decision: ON_DEVICE (Complexity Score: 0.12, Threshold: 0.50)
[ROUTER] Executing locally on Snapdragon X Elite NPU...
[ROUTER] HTP Silicon Allocation: 98% native HTP execution, 2% Kryo CPU fallback.
[ROUTER] Local generation complete. Checking output quality...
[ROUTER] Output quality self-verification: PASSED (Repetition index: 0.00)
[ROUTER] ================= TELEMETRY =================
[ROUTER] Latency: 87.2ms
[ROUTER] Energy Consumed: 0.12 Joules
[ROUTER] Cloud API Cost: $0.0000 USD
[ROUTER] =============================================

Scenario B: Local Execution with Cloud Escalation (Self-Verification Fallback)

[ROUTER] Ingesting query: "Draft a polite email asking for feedback on my project."
[ROUTER] Pre-processing TF-IDF features... (Time: 0.18ms)
[ROUTER] Routing Decision: ON_DEVICE (Complexity Score: 0.44, Threshold: 0.50)
[ROUTER] Executing locally on Snapdragon X Elite NPU...
[ROUTER] Local generation complete. Checking output quality...
[ROUTER] WARNING: Output failed quality validation (Repetitive loop detected, score: 0.76).
[ROUTER] Escalating query to cloud fallback (AutoMix cascade)...
[ROUTER] Cloud model response received successfully.
[ROUTER] ================= TELEMETRY =================
[ROUTER] Latency: 896.5ms (86.5ms local + 810.0ms cloud)
[ROUTER] Energy Consumed: 2.22 Joules (0.12J local NPU + 2.10J CPU fallback penalty)
[ROUTER] Cloud API Cost: $0.0055 USD
[ROUTER] =============================================