Document GLM-5.3-Flash Q2TP and bounded runtime

README.md

@@ -8,17 +8,19 @@ tags:
   - cmf
   - cortiq
   - quantized
+  - q2tp
   - q4tp
   - mixed-precision
   - moe
   - hybrid-attention
+  - 2-bit
   - 4-bit
 language:
   - en
   - zh
 ---
 
-# GLM-5.3-Flash Q4TP — universal CPU/GPU CMF
+# GLM-5.3-Flash Q2TP and Q4TP — universal CPU/GPU CMF
 
 This repository contains the **text tower** of
 [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash),
@@ -35,12 +37,15 @@ speculative MTP block. `cortiq info` reports 313.33B retained text parameters,
 and 37,534 payload tensors.
 
 ```bash
-hf download infosave/GLM-5.3-Flash-cmf glm-5.3-flash-q4tp.cmf --local-dir .
-cortiq verify glm-5.3-flash-q4tp.cmf
-cortiq run glm-5.3-flash-q4tp.cmf \
+hf download infosave/GLM-5.3-Flash-cmf glm-5.3-flash-q2tp.cmf --local-dir .
+cortiq verify glm-5.3-flash-q2tp.cmf
+cortiq run glm-5.3-flash-q2tp.cmf \
   --prompt "What is 2+2? Answer briefly." --no-think
 ```
 
+Replace `q2tp` with `q4tp` in these commands for the larger,
+quality-oriented profile.
+
 The file requires a GLM-5.3-compatible Cortiq build containing the dedicated
 `glm5_next` runtime. Older binaries reject this architecture instead of
 silently treating it as a generic Transformer. A packaged version will be
@@ -50,19 +55,23 @@ listed here after the remaining GPU release gate.
 
 | file | profile | exact size | status |
 |---|---|---:|---|
+| `glm-5.3-flash-q2tp.cmf` | mixed q2tp + q4tp + q8_2f + f16 | 116,254,068,322 bytes (108.27 GiB) | 37,534/37,534 tensor hashes pass |
 | `glm-5.3-flash-q4tp.cmf` | mixed q4tp + q8_2f + f16 | 167,164,530,274 bytes (155.68 GiB) | 37,534/37,534 tensor hashes pass |
 
 SHA-256:
 
 ```text
+818d997b7c19dd0da1b3424ed12d869e8ff6fc4a187154c23b08f2efc4198cde  glm-5.3-flash-q2tp.cmf
 6bd350591e1efd9dd098a189abf603467eed8387ee38d28f805e0891cedf36c6  glm-5.3-flash-q4tp.cmf
 ```
 
-This is a quality-oriented mixed profile, not a uniform four-bit dump. The
-large routed/shared expert payloads use q4tp. Always-active KDA/DSA attention
-projections and both vocabulary edges use q8_2f to protect recurrent error and
-input-channel outliers. Small norms, routers, convolution taps, mHC controls,
-and other sensitive nonlinear control tensors remain f16.
+Neither file is a uniform low-bit dump. In Q2TP, routed/shared expert gate and
+up projections use q2tp while their down projections remain q4tp. In Q4TP,
+all large routed/shared expert projections use q4tp. Both profiles preserve
+always-active KDA/DSA attention projections and both vocabulary edges as
+q8_2f to protect recurrent error and input-channel outliers. Small norms,
+routers, convolution taps, mHC controls, and other sensitive nonlinear control
+tensors remain f16.
 
 The source checkpoint's fine-grained E4M3 weights are decoded with their
 F32 `weight_scale_inv` planes using the official 128×128 tile orientation,
@@ -94,7 +103,7 @@ marker. `cortiq verify` independently checks every payload hash.
 
 ## Models larger than VRAM
 
-The 155.68 GiB file does not need to fit in VRAM. CMF keeps weights mmap-backed
+Neither the 108.27 GiB nor the 155.68 GiB file needs to fit in VRAM. CMF keeps weights mmap-backed
 in host memory and detects the available adapter budget. When dynamic pooling
 is enabled, routed experts use a bounded global GPU pool: resident experts run
 on the GPU, while cache misses are completed exactly on CPU and accumulated
@@ -105,44 +114,48 @@ Useful controls:
 
 ```bash
 cortiq gpu
-CMF_GPU=0 cortiq run glm-5.3-flash-q4tp.cmf --prompt "CPU check" --no-think
+CMF_GPU=0 cortiq run glm-5.3-flash-q2tp.cmf --prompt "CPU check" --no-think
 CMF_GPU_VRAM_MB=16000 CMF_GLM_DYNAMIC_MOE=1 \
-  cortiq run glm-5.3-flash-q4tp.cmf \
+  cortiq run glm-5.3-flash-q2tp.cmf \
   --prompt "bounded forced GPU pool" --no-think
-CMF_GPU_ADAPTER=0 cortiq run glm-5.3-flash-q4tp.cmf \
+CMF_GPU_ADAPTER=0 cortiq run glm-5.3-flash-q2tp.cmf \
   --prompt "select adapter 0" --no-think
-CMF_GLM_DYNAMIC_MOE=0 cortiq run glm-5.3-flash-q4tp.cmf \
+CMF_GLM_DYNAMIC_MOE=0 cortiq run glm-5.3-flash-q2tp.cmf \
   --prompt "force exact CPU MoE" --no-think
 ```
 
-For this **Q4TP** artifact, automatic mode deliberately keeps the MoE on exact
-CPU. On the measured host, forcing Q4TP expert pooling was substantially slower
-than CPU execution. Automatic dynamic pooling is therefore reserved for Q2TP
-gate/up tensors on supported Vulkan devices with at least a 14 GB detected
-budget. Explicit `CMF_GLM_DYNAMIC_MOE=1` remains available for bounded-memory
-compatibility testing; `CMF_GLM_DYNAMIC_MOE=0` always disables it.
+Automatic mode deliberately keeps both profiles on the exact CPU MoE path. On
+the measured host, bounded Vulkan expert pooling was slower for both Q2TP and
+Q4TP. Explicit `CMF_GLM_DYNAMIC_MOE=1` remains available for compatibility and
+experimentation; `CMF_GLM_DYNAMIC_MOE=0` always disables it. The GLM-specific
+pool is limited to 20–40% of the configured physical-memory envelope, and an
+explicit `CMF_GLM_EXPERT_SLOTS` value cannot bypass that cap.
 
 Measured on an RTX 3090 with dual AMD EPYC 7H12 CPUs and 32 CPU worker threads:
 
 | execution path | steady decode | notes |
 |---|---:|---|
+| Q2TP CPU32, automatic | **2.498 tok/s** | 20-token clean run, zero GPU submissions |
+| Q2TP CPU32, explicit CPU | **2.492 tok/s** | 20-token clean run |
+| Q2TP forced dynamic Vulkan, 16-GB target | **1.413 tok/s** | 512 slots / 4,352 MB pool, 42 submissions/token |
 | Q4TP CPU, final run A | 1.912 tok/s | exact path, zero GPU submissions |
 | Q4TP CPU, final run B | 2.350 tok/s | exact path, zero GPU submissions |
 | Q4TP CPU, final run C | 2.207 tok/s | exact path, zero GPU submissions |
 | Q4TP CPU, three-run mean | **2.156 tok/s** | 1.26% below the earlier 2.184 tok/s baseline |
 | Q4TP forced dynamic Vulkan | **0.634 tok/s** | 40-token run, 42 submissions/token, 13.9 GB pool |
 
-The CPU result varies with NUMA placement; the three-run spread is reported
+The Q4TP CPU result varies with NUMA placement; the three-run spread is reported
 instead of presenting one noisy maximum. Forced Vulkan was about 3.4× slower,
 which is why it is not the Q4TP default.
 
-A bounded compatibility run with
-`CMF_GPU_VRAM_MB=16000 CMF_GLM_DYNAMIC_MOE=1` completed successfully, emitted
-`Hello`, and used a 9.9 GB/792-slot expert pool. Process VRAM samples peaked at
-10,260 MiB on the 24 GB test card. This proves that the dynamic pool respects a
-16,000 MB budget; the one-token run is not a throughput claim. CPU-only systems
-and smaller adapters remain valid through the exact CPU fallback. Metal will be
-tested from this same immutable artifact.
+For Q2TP, a bounded compatibility run with
+`CMF_GPU_VRAM_MB=16000 CMF_GLM_DYNAMIC_MOE=1` completed successfully with the
+official-template answer `4`. Its 4,352-MB/512-slot expert pool peaked at
+6,563 MiB of total sampled process VRAM on the physical 24-GB test card, safely
+below 16 GiB. The automatic normal-budget run also emitted `4`, used the exact
+CPU path, and peaked at 331 MiB of VRAM. CPU-only systems and smaller adapters
+remain valid through the same exact fallback. Metal will be tested from these
+immutable artifacts.
 
 ## Scope and license