Document measured Q4TP CPU and 16 GB policy

README.md

@@ -23,9 +23,10 @@ language:
 This repository contains the **text tower** of
 [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash),
 converted directly from the official FP8 checkpoint to one memory-mapped CMF
-file. The same artifact runs on CPU and on Cortiq's portable GPU backends
-(Vulkan for NVIDIA/AMD/Intel, DX12, and Metal); inference does not require
-Python, PyTorch, a CUDA toolkit, or full-model GPU residency.
+file. The same artifact is mmap-backed for CPU and Cortiq's portable GPU
+backends; inference does not require Python, PyTorch, a CUDA toolkit, or
+full-model GPU residency. This release was validated on CPU and Vulkan;
+Metal validation is still pending.
 
 The upstream model is a 320B/18B-active native multimodal MoE. This release is
 text-generation only: it intentionally omits the vision tower and the optional
@@ -94,29 +95,54 @@ marker. `cortiq verify` independently checks every payload hash.
 ## Models larger than VRAM
 
 The 155.68 GiB file does not need to fit in VRAM. CMF keeps weights mmap-backed
-in host memory, detects the available adapter budget, and manages routed
-experts through a bounded global GPU pool. Resident experts run on the GPU;
-cache misses are completed exactly on CPU and accumulated into the same MoE
-result. Systems without a supported adapter use the same file on CPU.
+in host memory and detects the available adapter budget. When dynamic pooling
+is enabled, routed experts use a bounded global GPU pool: resident experts run
+on the GPU, while cache misses are completed exactly on CPU and accumulated
+into the same MoE result. Q4TP automatic mode and systems without a supported
+adapter use the same file on CPU.
 
 Useful controls:
 
 ```bash
 cortiq gpu
 CMF_GPU=0 cortiq run glm-5.3-flash-q4tp.cmf --prompt "CPU check" --no-think
-CMF_GPU_VRAM_MB=13500 cortiq run glm-5.3-flash-q4tp.cmf \
-  --prompt "bounded GPU memory" --no-think
+CMF_GPU_VRAM_MB=16000 CMF_GLM_DYNAMIC_MOE=1 \
+  cortiq run glm-5.3-flash-q4tp.cmf \
+  --prompt "bounded forced GPU pool" --no-think
 CMF_GPU_ADAPTER=0 cortiq run glm-5.3-flash-q4tp.cmf \
   --prompt "select adapter 0" --no-think
-CMF_GLM_DYNAMIC_MOE=1 cortiq run glm-5.3-flash-q4tp.cmf \
-  --prompt "force dynamic expert pool" --no-think
+CMF_GLM_DYNAMIC_MOE=0 cortiq run glm-5.3-flash-q4tp.cmf \
+  --prompt "force exact CPU MoE" --no-think
 ```
 
-The automatic dynamic-MoE policy enables on supported Vulkan devices with at
-least a 14 GB detected budget. Smaller cards and CPU-only machines remain
-valid targets through the exact fallback. RTX 3090 dynamic-pool and constrained
-16 GB measurements are still in progress and will be added without inventing
-unmeasured throughput. Metal validation will use this same uploaded artifact.
+For this **Q4TP** artifact, automatic mode deliberately keeps the MoE on exact
+CPU. On the measured host, forcing Q4TP expert pooling was substantially slower
+than CPU execution. Automatic dynamic pooling is therefore reserved for Q2TP
+gate/up tensors on supported Vulkan devices with at least a 14 GB detected
+budget. Explicit `CMF_GLM_DYNAMIC_MOE=1` remains available for bounded-memory
+compatibility testing; `CMF_GLM_DYNAMIC_MOE=0` always disables it.
+
+Measured on an RTX 3090 with dual AMD EPYC 7H12 CPUs and 32 CPU worker threads:
+
+| execution path | steady decode | notes |
+|---|---:|---|
+| Q4TP CPU, final run A | 1.912 tok/s | exact path, zero GPU submissions |
+| Q4TP CPU, final run B | 2.350 tok/s | exact path, zero GPU submissions |
+| Q4TP CPU, final run C | 2.207 tok/s | exact path, zero GPU submissions |
+| Q4TP CPU, three-run mean | **2.156 tok/s** | 1.26% below the earlier 2.184 tok/s baseline |
+| Q4TP forced dynamic Vulkan | **0.634 tok/s** | 40-token run, 42 submissions/token, 13.9 GB pool |
+
+The CPU result varies with NUMA placement; the three-run spread is reported
+instead of presenting one noisy maximum. Forced Vulkan was about 3.4× slower,
+which is why it is not the Q4TP default.
+
+A bounded compatibility run with
+`CMF_GPU_VRAM_MB=16000 CMF_GLM_DYNAMIC_MOE=1` completed successfully, emitted
+`Hello`, and used a 9.9 GB/792-slot expert pool. Process VRAM samples peaked at
+10,260 MiB on the 24 GB test card. This proves that the dynamic pool respects a
+16,000 MB budget; the one-token run is not a throughput claim. CPU-only systems
+and smaller adapters remain valid through the exact CPU fallback. Metal will be
+tested from this same immutable artifact.
 
 ## Scope and license
 
@@ -125,4 +151,3 @@ This CMF is a redistribution/quantization of the official
 The upstream model card is MIT-licensed. Refer to the upstream repository for
 model limitations, intended use, citation, and the authoritative training and
 benchmark description.
-