reports/COLD_START_EVALUATION.md

Phase Eight — Cold-Start & Model-Loading Evaluation Report

1. Motivation & Practical Edge Context

This evaluation is motivated by the repository author's practical developer experience when deploying medical and edge AI models on memory-constrained hardware: loading a 1.5B-parameter model (such as MedGemma 1.5B) into constrained VRAM took approximately 200 seconds before the first response could be generated. (Attribution Note: The ~200s MedGemma 1.5B figure is cited strictly as the developer's motivating user experience and is not claimed as a benchmark run inside this repository.)

On edge devices (phones, point-of-care medical diagnostic units, drones, embedded controllers), total user-perceived response time is governed by two distinct regimes:

$$T_{\text{total}}(N_{\text{tok}})=\underset{\text{Cold-Start Overhead }{T}_{\text{cold}}}{\underbrace{T_{\text{disk\_read}}+T_{\text{deserialize}}+T_{\text{init}}+T_{\text{load\_state\_dict}}+\text{TTFT}}}+\underset{\text{Steady-State Decode}}{\underbrace{N_{\text{tok}}\times \text{TPOT}}}T\_\{\backslash text\{total\}\}(N\_\{\backslash text\{tok\}\})\; =\; \backslash underbrace\{T\_\{\backslash text\{disk\backslash \_read\}\}\; +\; T\_\{\backslash text\{deserialize\}\}\; +\; T\_\{\backslash text\{init\}\}\; +\; T\_\{\backslash text\{load\backslash \_state\backslash \_dict\}\}\; +\; \backslash text\{TTFT\}\}\_\{\backslash text\{Cold-Start\; Overhead\; \}\; T\_\{\backslash text\{cold\}\}\}\; +\; \backslash underbrace\{N\_\{\backslash text\{tok\}\}\; \backslash times\; \backslash text\{TPOT\}\}\_\{\backslash text\{Steady-State\; Decode\}\}$$

---

2. Measured Cold-Start Breakdown (RUN-20261013-QTF-HARDENING)

Data source: results/raw/cold_start_and_edge.json (MEASURED on Intel 20-thread CPU, Windows 10, PyTorch 2.10.0+cpu, 5 repeats per model variant, $d_{\text{model}}=256, L_{\text{layers}}=4, H=8, V=10000$).

Model Variant	Dtype	Total Params	Checkpoint Size (MB)	Model Init (ms)	Disk Deserialization (ms)	load_state_dict (strict=True, ms)	First Inference TTFT ($L=16$, ms)	Total Cold-Start to 1st Token (ms)	Classification
DenseBaseline_FP32	float32	5,719,552	22.086 MB	115.25 ± 1.49	25.87 ± 5.59	5.19	17.76	164.08 ± 24.90 ms	MEASURED
QTensorFormer_Balanced_r8_FP32	float32	3,927,288	15.320 MB (-30.6%)	137.66 ± 48.97	58.97 ± 15.22	10.59	37.63	244.87 ± 83.37 ms	MEASURED
QTensorFormer_Edge_r2_FP32	float32	3,726,648	14.553 MB (-34.1%)	108.96 ± 0.56	54.79 ± 5.71	10.08	50.48	224.32 ± 37.31 ms	MEASURED
HF_QTensorFormer_GQA_r8_FP32	float32	5,852,928	22.353 MB	125.15 ± 2.92	32.85 ± 12.52	4.88	25.82	188.70 ± 17.42 ms	MEASURED
HF_QTensorFormer_GQA_r8_FP16	float16	5,852,928	11.189 MB (-49.3%)	132.72 ± 1.05	21.64 ± 5.81 (fastest)	4.27 (fastest)	335.66 (CPU FP16 emulation)	494.30 ± 11.10 ms	MEASURED

---

3. Honest Scientific Analysis of Cold-Start Findings

1. Checkpoint Storage & Disk Footprint Reduction (MEASURED):

   • Replacing dense FFN matrices with nested TT cores reduces on-disk checkpoint size from 22.086 MB (DenseBaseline_FP32) to 15.320 MB (QTensorFormer_Balanced_r8_FP32, -30.63%) and 14.553 MB (QTensorFormer_Edge_r2_FP32, -34.11%).
   • Storing the Hugging Face GQA variant in float16 (HF_QTensorFormer_GQA_r8_FP16) cuts checkpoint size to 11.189 MB (-49.34%) and achieves the fastest disk deserialization (21.64 ms vs 25.87 ms) and load_state_dict (4.27 ms vs 5.19 ms).
   • At the individual FFN layer level ($512 \times 2048$, where embedding tables do not dominate parameter count), TTLinear reduces weight storage from 4.20 MB to 0.53 MB at $r=8$ (-87.28%) and 0.14 MB at $r=2$ (-96.68%).

2. Object-Count Deserialization Overhead in PyTorch Zip Archives (MEASURED):

   • Although QTensorFormer_Balanced_r8_FP32 has 30.6% fewer bytes on disk than DenseBaseline_FP32, its state-dict has more individual tensor objects (185 keys vs 38 keys in DenseBaseline due to 4 TT cores per projection, information probes, and allocator MLPs). On fast local SSD cache, torch.load zip-entry parsing overhead (54.8–59.0 ms) exceeds the byte-read savings at small model scales (15–22 MB).
   • Implication for Large Edge Models ($\ge 1\text{B}$ parameters): When checkpoints are gigabytes in size and I/O-bound by flash storage or PCIe/VRAM paging (as in the MedGemma 1.5B scenario), byte volume dominates zip-entry metadata overhead, so a $3\times$–$10\times$ weight byte reduction directly reduces flash-to-VRAM staging time.

3. CPU FP16 Emulation Penalty (MEASURED):

   • While HF_QTensorFormer_GQA_r8_FP16 deserializes and loads into memory fastest (25.9 ms combined load), executing float16 einsum contractions on an x86 CPU without native FP16 tensor cores triggers software upcasting/downcasting (TTFT = 335.66 ms). On CPU, float32 (TTFT = 25.82 ms) should be used for execution while float16 can be used for on-disk storage.