# Phase Eight — Cold-Start & Model-Loading Evaluation Report

## 1. Motivation & Practical Edge Context

This evaluation is motivated by the repository author's practical developer experience when deploying medical and edge AI models on memory-constrained hardware: **loading a 1.5B-parameter model (such as MedGemma 1.5B) into constrained VRAM took approximately 200 seconds before the first response could be generated.**
*(Attribution Note: The ~200s MedGemma 1.5B figure is cited strictly as the developer's motivating user experience and is not claimed as a benchmark run inside this repository.)*

On edge devices (phones, point-of-care medical diagnostic units, drones, embedded controllers), total user-perceived response time is governed by two distinct regimes:

$$T_{\text{total}}(N_{\text{tok}}) = \underbrace{T_{\text{disk\_read}} + T_{\text{deserialize}} + T_{\text{init}} + T_{\text{load\_state\_dict}} + \text{TTFT}}_{\text{Cold-Start Overhead } T_{\text{cold}}} + \underbrace{N_{\text{tok}} \times \text{TPOT}}_{\text{Steady-State Decode}}$$

---

## 2. Measured Cold-Start Breakdown (`RUN-20261013-QTF-HARDENING`)

Data source: `results/raw/cold_start_and_edge.json` (`MEASURED` on Intel 20-thread CPU, Windows 10, PyTorch `2.10.0+cpu`, 5 repeats per model variant, $d_{\text{model}}=256, L_{\text{layers}}=4, H=8, V=10000$).

| Model Variant | Dtype | Total Params | Checkpoint Size (MB) | Model Init (ms) | Disk Deserialization (ms) | `load_state_dict` (`strict=True`, ms) | First Inference `TTFT` ($L=16$, ms) | Total Cold-Start to 1st Token (ms) | Classification |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| `DenseBaseline_FP32` | `float32` | `5,719,552` | `22.086 MB` | `115.25 ± 1.49` | `25.87 ± 5.59` | `5.19` | `17.76` | `164.08 ± 24.90 ms` | `MEASURED` |
| `QTensorFormer_Balanced_r8_FP32` | `float32` | `3,927,288` | `15.320 MB` (**-30.6%**) | `137.66 ± 48.97` | `58.97 ± 15.22` | `10.59` | `37.63` | `244.87 ± 83.37 ms` | `MEASURED` |
| `QTensorFormer_Edge_r2_FP32` | `float32` | `3,726,648` | `14.553 MB` (**-34.1%**) | `108.96 ± 0.56` | `54.79 ± 5.71` | `10.08` | `50.48` | `224.32 ± 37.31 ms` | `MEASURED` |
| `HF_QTensorFormer_GQA_r8_FP32` | `float32` | `5,852,928` | `22.353 MB` | `125.15 ± 2.92` | `32.85 ± 12.52` | `4.88` | `25.82` | `188.70 ± 17.42 ms` | `MEASURED` |
| `HF_QTensorFormer_GQA_r8_FP16` | `float16` | `5,852,928` | `11.189 MB` (**-49.3%**) | `132.72 ± 1.05` | `21.64 ± 5.81` (**fastest**) | `4.27` (**fastest**) | `335.66` (CPU FP16 emulation) | `494.30 ± 11.10 ms` | `MEASURED` |

---

## 3. Honest Scientific Analysis of Cold-Start Findings

1. **Checkpoint Storage & Disk Footprint Reduction (`MEASURED`):**
   - Replacing dense FFN matrices with nested TT cores reduces on-disk checkpoint size from `22.086 MB` (`DenseBaseline_FP32`) to `15.320 MB` (`QTensorFormer_Balanced_r8_FP32`, **-30.63%**) and `14.553 MB` (`QTensorFormer_Edge_r2_FP32`, **-34.11%**).
   - Storing the Hugging Face GQA variant in `float16` (`HF_QTensorFormer_GQA_r8_FP16`) cuts checkpoint size to `11.189 MB` (**-49.34%**) and achieves the fastest disk deserialization (`21.64 ms` vs `25.87 ms`) and `load_state_dict` (`4.27 ms` vs `5.19 ms`).
   - At the individual FFN layer level ($512 \times 2048$, where embedding tables do not dominate parameter count), `TTLinear` reduces weight storage from `4.20 MB` to `0.53 MB` at $r=8$ (**-87.28%**) and `0.14 MB` at $r=2$ (**-96.68%**).

2. **Object-Count Deserialization Overhead in PyTorch Zip Archives (`MEASURED`):**
   - Although `QTensorFormer_Balanced_r8_FP32` has `30.6%` fewer bytes on disk than `DenseBaseline_FP32`, its state-dict has more individual tensor objects (`185` keys vs `38` keys in `DenseBaseline` due to 4 TT cores per projection, information probes, and allocator MLPs). On fast local SSD cache, `torch.load` zip-entry parsing overhead (`54.8–59.0 ms`) exceeds the byte-read savings at small model scales (`15–22 MB`).
   - **Implication for Large Edge Models ($\ge 1\text{B}$ parameters):** When checkpoints are gigabytes in size and I/O-bound by flash storage or PCIe/VRAM paging (as in the MedGemma 1.5B scenario), byte volume dominates zip-entry metadata overhead, so a $3\times$–$10\times$ weight byte reduction directly reduces flash-to-VRAM staging time.

3. **CPU FP16 Emulation Penalty (`MEASURED`):**
   - While `HF_QTensorFormer_GQA_r8_FP16` deserializes and loads into memory fastest (`25.9 ms` combined load), executing `float16` `einsum` contractions on an x86 CPU without native FP16 tensor cores triggers software upcasting/downcasting (`TTFT = 335.66 ms`). On CPU, `float32` (`TTFT = 25.82 ms`) should be used for execution while `float16` can be used for on-disk storage.