Hugging Face's logo

HelloSun
/
ling30tiny_EPQ

Text Generation
GGUF
bailingmoe3
ling
Mixture of Experts
expert-paging
ssd-to-ram
cpu
Model card Files
xet
 Community
HelloSun commited on
Commit
afafadc
·
verified ·
1 Parent(s): f281613

Add strict memory benchmark results

Browse files
Files changed (10) hide show
  1. results/host_capability_audit.txt +15 -0
  2. results/io_audit.md +12 -0
  3. results/io_buffered_watch.txt +31 -0
  4. results/io_direct_watch.txt +36 -0
  5. results/strict_memory_levels.json +416 -0
  6. results/strict_memory_levels.md +24 -0
  7. results/verify_dequant.txt +23 -0
  8. results/verify_moe_layer.txt +4 -0
  9. results/verify_selftest.txt +0 -0
  10. results/verify_selftest_numpy.txt +4 -0
results/host_capability_audit.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Host: Intel(R) Xeon(R) Platinum 8559C
2
+ Host threads: 192
3
+ Host MemTotal: 2147448086528 bytes (reported by /proc/meminfo)
4
+ Container cgroup memory.max: 104000000000 bytes
5
+ Container cgroup memory.swap.max: 2936848384 bytes
6
+ Container cgroup memory.high: max
7
+ GPU: unavailable; all tests ran on CPU
8
+ Model file: Ling-3.0-tiny-Q4_K_M.gguf
9
+ Model size: 4823894944 bytes
10
+ Model source: inclusionAI/Ling-3.0-tiny-GGUF
11
+ llama.cpp base commit: c811cb8f0ac91b8ac72a32f970bdd45037f20da7
12
+ Build: Release, GGML_NATIVE=OFF, GGML_CPU_REPACK=OFF, AVX512/VNNI enabled, AMX disabled
13
+ Hard limit: prlimit --data (RLIMIT_DATA)
14
+ Benchmark: ctx=256 ubatch=32 threads=4 temp=0 n_predict=32 repeats=2
15
+ Page cache: per-run posix_fadvise(POSIX_FADV_DONTNEED)
results/io_audit.md ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # I/O 稽核：O_DIRECT 與 buffered page cache 對照
2
+
3
+ 在 1 GiB RLIMIT_DATA 條件下，先將 GGUF page cache 驅逐，再以 0.25 秒間隔量測模型檔 `mincore` resident 頁面與 `/proc/<pid>/io`。
4
+
5
+ | 測試 | 真實裝置 read_bytes 峰值 | GGUF page-cache resident 峰值 | Cached 增量峰值 |
6
+ |---|---:|---:|---:|
7
+ | 預設 O_DIRECT pager | 8.981 GiB | 0.496 GiB | +0.768 GiB |
8
+ | Buffered 對照 (`SDQ_NO_DIRECT_IO=1`) | 3.341 GiB | 3.341 GiB | +3.336 GiB |
9
+
10
+ 在 buffered 對照中，GGUF resident 隨 buffered reads 上升至 3.341 GiB；O_DIRECT 組只有 0.496 GiB resident，顯示 direct I/O 沒有將相同專家權重工作集留在模型檔 page cache。模型載入仍會讀取/快取非 expert tensors，因此 resident 並非零。兩次 I/O 數字因 O_DIRECT 與 buffered 的核心讀取路徑不同，不應視為完全相同負載下的效能對比。
11
+
12
+ 原始時間序列：[`io_direct_watch.txt`](io_direct_watch.txt) 與 [`io_buffered_watch.txt`](io_buffered_watch.txt)。
results/io_buffered_watch.txt ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [watch] evicting 1 shard(s) from the page cache
2
+ Ling-3.0-tiny-Q4_K_M.gguf resident 0.000 GiB / 4.493 GiB ( 0.0 %)
3
+ [watch] t= 0.0s read_bytes= 0.000 GiB rchar= 0.000 GiB model_file_resident= 0.000 GiB RssAnon= 0.000 RssFile= 0.000 Cached_delta= +0.000 GiB
4
+ [watch] t= 0.3s read_bytes= 0.063 GiB rchar= 0.063 GiB model_file_resident= 0.063 GiB RssAnon= 0.061 RssFile= 0.007 Cached_delta= +0.061 GiB
5
+ [watch] t= 0.5s read_bytes= 0.063 GiB rchar= 0.069 GiB model_file_resident= 0.063 GiB RssAnon= 0.336 RssFile= 0.007 Cached_delta= +0.062 GiB
6
+ [watch] t= 0.8s read_bytes= 0.093 GiB rchar= 0.069 GiB model_file_resident= 0.093 GiB RssAnon= 0.392 RssFile= 0.035 Cached_delta= +0.096 GiB
7
+ [watch] t= 1.1s read_bytes= 0.383 GiB rchar= 0.308 GiB model_file_resident= 0.386 GiB RssAnon= 0.394 RssFile= 0.063 Cached_delta= +0.397 GiB
8
+ [watch] t= 1.4s read_bytes= 0.668 GiB rchar= 0.526 GiB model_file_resident= 0.689 GiB RssAnon= 0.396 RssFile= 0.090 Cached_delta= +0.691 GiB
9
+ [watch] t= 1.6s read_bytes= 0.959 GiB rchar= 0.755 GiB model_file_resident= 0.984 GiB RssAnon= 0.397 RssFile= 0.119 Cached_delta= +0.987 GiB
10
+ [watch] t= 1.9s read_bytes= 1.270 GiB rchar= 0.994 GiB model_file_resident= 1.283 GiB RssAnon= 0.398 RssFile= 0.146 Cached_delta= +1.277 GiB
11
+ [watch] t= 2.2s read_bytes= 1.566 GiB rchar= 1.218 GiB model_file_resident= 1.592 GiB RssAnon= 0.400 RssFile= 0.178 Cached_delta= +1.583 GiB
12
+ [watch] t= 2.5s read_bytes= 1.878 GiB rchar= 1.456 GiB model_file_resident= 1.922 GiB RssAnon= 0.402 RssFile= 0.209 Cached_delta= +1.920 GiB
13
+ [watch] t= 2.8s read_bytes= 2.176 GiB rchar= 1.689 GiB model_file_resident= 2.254 GiB RssAnon= 0.404 RssFile= 0.241 Cached_delta= +2.254 GiB
14
+ [watch] t= 3.1s read_bytes= 2.519 GiB rchar= 1.968 GiB model_file_resident= 2.579 GiB RssAnon= 0.406 RssFile= 0.274 Cached_delta= +2.582 GiB
15
+ [watch] t= 3.4s read_bytes= 2.787 GiB rchar= 2.129 GiB model_file_resident= 2.788 GiB RssAnon= 0.408 RssFile= 0.412 Cached_delta= +2.815 GiB
16
+ [watch] t= 3.8s read_bytes= 2.921 GiB rchar= 2.679 GiB model_file_resident= 2.948 GiB RssAnon= 0.409 RssFile= 0.473 Cached_delta= +2.938 GiB
17
+ [watch] t= 4.1s read_bytes= 3.020 GiB rchar= 3.475 GiB model_file_resident= 3.025 GiB RssAnon= 0.411 RssFile= 0.473 Cached_delta= +3.020 GiB
18
+ [watch] t= 4.4s read_bytes= 3.070 GiB rchar= 4.222 GiB model_file_resident= 3.071 GiB RssAnon= 0.411 RssFile= 0.473 Cached_delta= +3.076 GiB
19
+ [watch] t= 4.7s read_bytes= 3.115 GiB rchar= 4.977 GiB model_file_resident= 3.122 GiB RssAnon= 0.411 RssFile= 0.473 Cached_delta= +3.119 GiB
20
+ [watch] t= 5.0s read_bytes= 3.151 GiB rchar= 5.838 GiB model_file_resident= 3.151 GiB RssAnon= 0.411 RssFile= 0.474 Cached_delta= +3.166 GiB
21
+ [watch] t= 5.3s read_bytes= 3.214 GiB rchar= 6.582 GiB model_file_resident= 3.215 GiB RssAnon= 0.411 RssFile= 0.474 Cached_delta= +3.212 GiB
22
+ [watch] t= 5.7s read_bytes= 3.255 GiB rchar= 7.423 GiB model_file_resident= 3.260 GiB RssAnon= 0.411 RssFile= 0.474 Cached_delta= +3.256 GiB
23
+ [watch] t= 6.0s read_bytes= 3.300 GiB rchar= 8.154 GiB model_file_resident= 3.303 GiB RssAnon= 0.411 RssFile= 0.474 Cached_delta= +3.303 GiB
24
+ [watch] t= 6.3s read_bytes= 3.341 GiB rchar= 8.812 GiB model_file_resident= 3.341 GiB RssAnon= 0.360 RssFile= 0.009 Cached_delta= +3.336 GiB
25
+
26
+ [watch] === summary over 22 samples, 6.6s ===
27
+ [watch] peak model_file_resident in page cache : 3.341 GiB
28
+ [watch] peak process RssAnon : 0.411 GiB
29
+ [watch] peak process RssFile : 0.474 GiB
30
+ [watch] system Cached delta (peak) : +3.336 GiB
31
+ [watch] final model_file_resident : 3.341 GiB
results/io_direct_watch.txt ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [watch] evicting 1 shard(s) from the page cache
2
+ Ling-3.0-tiny-Q4_K_M.gguf resident 0.000 GiB / 4.493 GiB ( 0.0 %)
3
+ [watch] t= 0.0s read_bytes= 0.000 GiB rchar= 0.000 GiB model_file_resident= 0.000 GiB RssAnon= 0.000 RssFile= 0.000 Cached_delta= +0.000 GiB
4
+ [watch] t= 0.3s read_bytes= 0.063 GiB rchar= 0.063 GiB model_file_resident= 0.063 GiB RssAnon= 0.052 RssFile= 0.006 Cached_delta= +0.312 GiB
5
+ [watch] t= 0.5s read_bytes= 0.063 GiB rchar= 0.069 GiB model_file_resident= 0.063 GiB RssAnon= 0.328 RssFile= 0.007 Cached_delta= +0.312 GiB
6
+ [watch] t= 0.8s read_bytes= 0.087 GiB rchar= 0.069 GiB model_file_resident= 0.088 GiB RssAnon= 0.392 RssFile= 0.029 Cached_delta= +0.341 GiB
7
+ [watch] t= 1.1s read_bytes= 0.452 GiB rchar= 0.395 GiB model_file_resident= 0.126 GiB RssAnon= 0.395 RssFile= 0.065 Cached_delta= +0.375 GiB
8
+ [watch] t= 1.3s read_bytes= 0.771 GiB rchar= 0.673 GiB model_file_resident= 0.168 GiB RssAnon= 0.396 RssFile= 0.103 Cached_delta= +0.421 GiB
9
+ [watch] t= 1.6s read_bytes= 1.081 GiB rchar= 0.938 GiB model_file_resident= 0.212 GiB RssAnon= 0.398 RssFile= 0.145 Cached_delta= +0.467 GiB
10
+ [watch] t= 1.9s read_bytes= 1.427 GiB rchar= 1.249 GiB model_file_resident= 0.250 GiB RssAnon= 0.400 RssFile= 0.177 Cached_delta= +0.505 GiB
11
+ [watch] t= 2.2s read_bytes= 1.733 GiB rchar= 1.510 GiB model_file_resident= 0.294 GiB RssAnon= 0.402 RssFile= 0.218 Cached_delta= +0.552 GiB
12
+ [watch] t= 2.4s read_bytes= 2.092 GiB rchar= 1.827 GiB model_file_resident= 0.329 GiB RssAnon= 0.405 RssFile= 0.252 Cached_delta= +0.589 GiB
13
+ [watch] t= 2.7s read_bytes= 2.447 GiB rchar= 2.147 GiB model_file_resident= 0.369 GiB RssAnon= 0.407 RssFile= 0.345 Cached_delta= +0.643 GiB
14
+ [watch] t= 3.0s read_bytes= 2.578 GiB rchar= 2.150 GiB model_file_resident= 0.494 GiB RssAnon= 0.409 RssFile= 0.470 Cached_delta= +0.754 GiB
15
+ [watch] t= 3.3s read_bytes= 3.147 GiB rchar= 2.717 GiB model_file_resident= 0.494 GiB RssAnon= 0.409 RssFile= 0.471 Cached_delta= +0.759 GiB
16
+ [watch] t= 3.5s read_bytes= 3.618 GiB rchar= 3.187 GiB model_file_resident= 0.494 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.762 GiB
17
+ [watch] t= 3.8s read_bytes= 3.986 GiB rchar= 3.560 GiB model_file_resident= 0.494 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.763 GiB
18
+ [watch] t= 4.1s read_bytes= 4.316 GiB rchar= 3.885 GiB model_file_resident= 0.494 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.763 GiB
19
+ [watch] t= 4.4s read_bytes= 4.744 GiB rchar= 4.316 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.763 GiB
20
+ [watch] t= 4.6s read_bytes= 5.141 GiB rchar= 4.710 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.764 GiB
21
+ [watch] t= 4.9s read_bytes= 5.572 GiB rchar= 5.144 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.744 GiB
22
+ [watch] t= 5.2s read_bytes= 5.968 GiB rchar= 5.542 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.471 Cached_delta= +0.765 GiB
23
+ [watch] t= 5.5s read_bytes= 6.379 GiB rchar= 5.953 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.765 GiB
24
+ [watch] t= 5.7s read_bytes= 6.806 GiB rchar= 6.373 GiB model_file_resident= 0.495 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.766 GiB
25
+ [watch] t= 6.0s read_bytes= 7.282 GiB rchar= 6.851 GiB model_file_resident= 0.496 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.766 GiB
26
+ [watch] t= 6.3s read_bytes= 7.711 GiB rchar= 7.278 GiB model_file_resident= 0.496 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.767 GiB
27
+ [watch] t= 6.6s read_bytes= 8.108 GiB rchar= 7.680 GiB model_file_resident= 0.496 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.767 GiB
28
+ [watch] t= 6.9s read_bytes= 8.540 GiB rchar= 8.109 GiB model_file_resident= 0.496 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.767 GiB
29
+ [watch] t= 7.1s read_bytes= 8.981 GiB rchar= 8.548 GiB model_file_resident= 0.496 GiB RssAnon= 0.411 RssFile= 0.472 Cached_delta= +0.768 GiB
30
+
31
+ [watch] === summary over 27 samples, 7.4s ===
32
+ [watch] peak model_file_resident in page cache : 0.496 GiB
33
+ [watch] peak process RssAnon : 0.411 GiB
34
+ [watch] peak process RssFile : 0.472 GiB
35
+ [watch] system Cached delta (peak) : +0.768 GiB
36
+ [watch] final model_file_resident : 0.496 GiB
results/strict_memory_levels.json ADDED
@@ -0,0 +1,416 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "host": {
3
+ "cpu_model": "Intel(R) Xeon(R) Platinum 8559C",
4
+ "cpu_threads": 192,
5
+ "mem_total_bytes": 2147448086528,
6
+ "cgroup": {
7
+ "memory.max": "104000000000",
8
+ "memory.swap.max": "2936848384",
9
+ "memory.high": "max"
10
+ }
11
+ },
12
+ "config": {
13
+ "model": "models/Ling-3.0-tiny-Q4_K_M.gguf",
14
+ "shards": [
15
+ "models/Ling-3.0-tiny-Q4_K_M.gguf"
16
+ ],
17
+ "bin": "llama.cpp/build/bin/sdq-chat",
18
+ "levels_mb": [
19
+ 1024,
20
+ 2048,
21
+ 4096,
22
+ 8192,
23
+ 16384
24
+ ],
25
+ "repeat": 2,
26
+ "ctx": 256,
27
+ "ubatch": 32,
28
+ "threads": 4,
29
+ "n_predict": 32,
30
+ "reserve_mb": 1024,
31
+ "timeout": 3600,
32
+ "evict_helper": "./pagecache_resident",
33
+ "reserve_map": "1024:700,2048:700,4096:700,8192:1000,16384:1400",
34
+ "output_dir": "results/strict_runs",
35
+ "out_json": "results/strict_memory_levels.json",
36
+ "_reserve": {
37
+ "1024": 700,
38
+ "2048": 700,
39
+ "4096": 700,
40
+ "8192": 1000,
41
+ "16384": 1400
42
+ }
43
+ },
44
+ "runs": [
45
+ {
46
+ "level_mb": 1024,
47
+ "level_gib": 1.0,
48
+ "repeat": 1,
49
+ "wall_s": 7.63,
50
+ "exit_code": 0,
51
+ "peak_rss_mb": 904.9,
52
+ "peak_anon_mb": 420.9,
53
+ "peak_swap_mb": 0.0,
54
+ "swap_free": true,
55
+ "anon_within_limit": true,
56
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
57
+ "status": "OK",
58
+ "decode_tok_s": 8.23,
59
+ "decode_tok": 31,
60
+ "prefill_tok_s": 17.7,
61
+ "prefill_tok": 56,
62
+ "final_rss_mb": 390,
63
+ "selfpeak_rss_mb": 390,
64
+ "self_swap_mb": 0,
65
+ "rss_anon_mb": 382,
66
+ "rss_file_mb": 8,
67
+ "rss_total_mb": 390,
68
+ "arena_gib": 0.3,
69
+ "n_slots": 225,
70
+ "total_ssd_gb": 10.33,
71
+ "decode_ssd_mb": 6189.6,
72
+ "ssd_mb_per_token": 199.7,
73
+ "expert_requests": 8769,
74
+ "expert_hits": 1470,
75
+ "hot_rate_pct": 16.7636,
76
+ "cold_rate_pct": 83.2364,
77
+ "evictions": 7074,
78
+ "per_layer_hit_min": 0,
79
+ "per_layer_hit_max": 37.9121,
80
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
81
+ },
82
+ {
83
+ "level_mb": 1024,
84
+ "level_gib": 1.0,
85
+ "repeat": 2,
86
+ "wall_s": 7.65,
87
+ "exit_code": 0,
88
+ "peak_rss_mb": 904.3,
89
+ "peak_anon_mb": 420.9,
90
+ "peak_swap_mb": 0.0,
91
+ "swap_free": true,
92
+ "anon_within_limit": true,
93
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
94
+ "status": "OK",
95
+ "decode_tok_s": 8.27,
96
+ "decode_tok": 31,
97
+ "prefill_tok_s": 17.7,
98
+ "prefill_tok": 56,
99
+ "final_rss_mb": 390,
100
+ "selfpeak_rss_mb": 390,
101
+ "self_swap_mb": 0,
102
+ "rss_anon_mb": 382,
103
+ "rss_file_mb": 7,
104
+ "rss_total_mb": 390,
105
+ "arena_gib": 0.3,
106
+ "n_slots": 225,
107
+ "total_ssd_gb": 10.34,
108
+ "decode_ssd_mb": 6193.5,
109
+ "ssd_mb_per_token": 199.8,
110
+ "expert_requests": 8769,
111
+ "expert_hits": 1467,
112
+ "hot_rate_pct": 16.7294,
113
+ "cold_rate_pct": 83.2706,
114
+ "evictions": 7077,
115
+ "per_layer_hit_min": 0,
116
+ "per_layer_hit_max": 37.9121,
117
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
118
+ },
119
+ {
120
+ "level_mb": 2048,
121
+ "level_gib": 2.0,
122
+ "repeat": 1,
123
+ "wall_s": 7.13,
124
+ "exit_code": 0,
125
+ "peak_rss_mb": 1894.9,
126
+ "peak_anon_mb": 1411.9,
127
+ "peak_swap_mb": 0.0,
128
+ "swap_free": true,
129
+ "anon_within_limit": true,
130
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
131
+ "status": "OK",
132
+ "decode_tok_s": 12.01,
133
+ "decode_tok": 31,
134
+ "prefill_tok_s": 17.7,
135
+ "prefill_tok": 56,
136
+ "final_rss_mb": 1381,
137
+ "selfpeak_rss_mb": 1381,
138
+ "self_swap_mb": 0,
139
+ "rss_anon_mb": 1373,
140
+ "rss_file_mb": 7,
141
+ "rss_total_mb": 1381,
142
+ "arena_gib": 1.3,
143
+ "n_slots": 979,
144
+ "total_ssd_gb": 6.46,
145
+ "decode_ssd_mb": 2287.7,
146
+ "ssd_mb_per_token": 73.8,
147
+ "expert_requests": 8769,
148
+ "expert_hits": 4338,
149
+ "hot_rate_pct": 49.4697,
150
+ "cold_rate_pct": 50.5303,
151
+ "evictions": 3452,
152
+ "per_layer_hit_min": 26.7157,
153
+ "per_layer_hit_max": 75.0663,
154
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
155
+ },
156
+ {
157
+ "level_mb": 2048,
158
+ "level_gib": 2.0,
159
+ "repeat": 2,
160
+ "wall_s": 7.19,
161
+ "exit_code": 0,
162
+ "peak_rss_mb": 1895.3,
163
+ "peak_anon_mb": 1411.9,
164
+ "peak_swap_mb": 0.0,
165
+ "swap_free": true,
166
+ "anon_within_limit": true,
167
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
168
+ "status": "OK",
169
+ "decode_tok_s": 11.75,
170
+ "decode_tok": 31,
171
+ "prefill_tok_s": 17.3,
172
+ "prefill_tok": 56,
173
+ "final_rss_mb": 1381,
174
+ "selfpeak_rss_mb": 1381,
175
+ "self_swap_mb": 0,
176
+ "rss_anon_mb": 1373,
177
+ "rss_file_mb": 7,
178
+ "rss_total_mb": 1381,
179
+ "arena_gib": 1.3,
180
+ "n_slots": 979,
181
+ "total_ssd_gb": 6.46,
182
+ "decode_ssd_mb": 2287.5,
183
+ "ssd_mb_per_token": 73.8,
184
+ "expert_requests": 8769,
185
+ "expert_hits": 4339,
186
+ "hot_rate_pct": 49.4811,
187
+ "cold_rate_pct": 50.5189,
188
+ "evictions": 3451,
189
+ "per_layer_hit_min": 26.7157,
190
+ "per_layer_hit_max": 75.0663,
191
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
192
+ },
193
+ {
194
+ "level_mb": 4096,
195
+ "level_gib": 4.0,
196
+ "repeat": 1,
197
+ "wall_s": 7.9,
198
+ "exit_code": 0,
199
+ "peak_rss_mb": 3887.7,
200
+ "peak_anon_mb": 3404.0,
201
+ "peak_swap_mb": 0.0,
202
+ "swap_free": true,
203
+ "anon_within_limit": true,
204
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
205
+ "status": "OK",
206
+ "decode_tok_s": 15.83,
207
+ "decode_tok": 31,
208
+ "prefill_tok_s": 17.0,
209
+ "prefill_tok": 56,
210
+ "final_rss_mb": 3362,
211
+ "selfpeak_rss_mb": 3362,
212
+ "self_swap_mb": 0,
213
+ "rss_anon_mb": 3354,
214
+ "rss_file_mb": 8,
215
+ "rss_total_mb": 3362,
216
+ "arena_gib": 3.3,
217
+ "n_slots": 2486,
218
+ "total_ssd_gb": 4.43,
219
+ "decode_ssd_mb": 767.5,
220
+ "ssd_mb_per_token": 24.8,
221
+ "expert_requests": 8786,
222
+ "expert_hits": 5864,
223
+ "hot_rate_pct": 66.7425,
224
+ "cold_rate_pct": 33.25749999999999,
225
+ "evictions": 1231,
226
+ "per_layer_hit_min": 40.9314,
227
+ "per_layer_hit_max": 78.1003,
228
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
229
+ },
230
+ {
231
+ "level_mb": 4096,
232
+ "level_gib": 4.0,
233
+ "repeat": 2,
234
+ "wall_s": 7.83,
235
+ "exit_code": 0,
236
+ "peak_rss_mb": 3887.5,
237
+ "peak_anon_mb": 3403.9,
238
+ "peak_swap_mb": 0.0,
239
+ "swap_free": true,
240
+ "anon_within_limit": true,
241
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
242
+ "status": "OK",
243
+ "decode_tok_s": 16.92,
244
+ "decode_tok": 31,
245
+ "prefill_tok_s": 17.0,
246
+ "prefill_tok": 56,
247
+ "final_rss_mb": 3362,
248
+ "selfpeak_rss_mb": 3362,
249
+ "self_swap_mb": 0,
250
+ "rss_anon_mb": 3354,
251
+ "rss_file_mb": 8,
252
+ "rss_total_mb": 3362,
253
+ "arena_gib": 3.3,
254
+ "n_slots": 2486,
255
+ "total_ssd_gb": 4.43,
256
+ "decode_ssd_mb": 767.5,
257
+ "ssd_mb_per_token": 24.8,
258
+ "expert_requests": 8786,
259
+ "expert_hits": 5864,
260
+ "hot_rate_pct": 66.7425,
261
+ "cold_rate_pct": 33.25749999999999,
262
+ "evictions": 1231,
263
+ "per_layer_hit_min": 40.9314,
264
+ "per_layer_hit_max": 78.1003,
265
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
266
+ },
267
+ {
268
+ "level_mb": 8192,
269
+ "level_gib": 8.0,
270
+ "repeat": 1,
271
+ "wall_s": 11.19,
272
+ "exit_code": 0,
273
+ "peak_rss_mb": 7548.3,
274
+ "peak_anon_mb": 7064.2,
275
+ "peak_swap_mb": 0.0,
276
+ "swap_free": true,
277
+ "anon_within_limit": true,
278
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
279
+ "status": "OK",
280
+ "decode_tok_s": 17.72,
281
+ "decode_tok": 31,
282
+ "prefill_tok_s": 14.0,
283
+ "prefill_tok": 56,
284
+ "final_rss_mb": 7034,
285
+ "selfpeak_rss_mb": 7034,
286
+ "self_swap_mb": 0,
287
+ "rss_anon_mb": 7026,
288
+ "rss_file_mb": 8,
289
+ "rss_total_mb": 7034,
290
+ "arena_gib": 7.01,
291
+ "n_slots": 5279,
292
+ "total_ssd_gb": 3.22,
293
+ "decode_ssd_mb": 220.5,
294
+ "ssd_mb_per_token": 7.1,
295
+ "expert_requests": 8769,
296
+ "expert_hits": 6689,
297
+ "hot_rate_pct": 76.2801,
298
+ "cold_rate_pct": 23.719899999999996,
299
+ "evictions": 0,
300
+ "per_layer_hit_min": 73.75,
301
+ "per_layer_hit_max": 78.1003,
302
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
303
+ },
304
+ {
305
+ "level_mb": 8192,
306
+ "level_gib": 8.0,
307
+ "repeat": 2,
308
+ "wall_s": 10.57,
309
+ "exit_code": 0,
310
+ "peak_rss_mb": 7547.8,
311
+ "peak_anon_mb": 7064.2,
312
+ "peak_swap_mb": 0.0,
313
+ "swap_free": true,
314
+ "anon_within_limit": true,
315
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
316
+ "status": "OK",
317
+ "decode_tok_s": 19.57,
318
+ "decode_tok": 31,
319
+ "prefill_tok_s": 15.7,
320
+ "prefill_tok": 56,
321
+ "final_rss_mb": 7034,
322
+ "selfpeak_rss_mb": 7034,
323
+ "self_swap_mb": 0,
324
+ "rss_anon_mb": 7026,
325
+ "rss_file_mb": 8,
326
+ "rss_total_mb": 7034,
327
+ "arena_gib": 7.01,
328
+ "n_slots": 5279,
329
+ "total_ssd_gb": 3.22,
330
+ "decode_ssd_mb": 220.5,
331
+ "ssd_mb_per_token": 7.1,
332
+ "expert_requests": 8769,
333
+ "expert_hits": 6689,
334
+ "hot_rate_pct": 76.2801,
335
+ "cold_rate_pct": 23.719899999999996,
336
+ "evictions": 0,
337
+ "per_layer_hit_min": 73.75,
338
+ "per_layer_hit_max": 78.1003,
339
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
340
+ },
341
+ {
342
+ "level_mb": 16384,
343
+ "level_gib": 16.0,
344
+ "repeat": 1,
345
+ "wall_s": 17.55,
346
+ "exit_code": 0,
347
+ "peak_rss_mb": 15084.0,
348
+ "peak_anon_mb": 14600.4,
349
+ "peak_swap_mb": 0.0,
350
+ "swap_free": true,
351
+ "anon_within_limit": true,
352
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
353
+ "status": "OK",
354
+ "decode_tok_s": 19.23,
355
+ "decode_tok": 31,
356
+ "prefill_tok_s": 10.4,
357
+ "prefill_tok": 56,
358
+ "final_rss_mb": 14571,
359
+ "selfpeak_rss_mb": 14571,
360
+ "self_swap_mb": 0,
361
+ "rss_anon_mb": 14563,
362
+ "rss_file_mb": 7,
363
+ "rss_total_mb": 14571,
364
+ "arena_gib": 14.62,
365
+ "n_slots": 11013,
366
+ "total_ssd_gb": 3.22,
367
+ "decode_ssd_mb": 223.4,
368
+ "ssd_mb_per_token": 7.2,
369
+ "expert_requests": 8763,
370
+ "expert_hits": 6683,
371
+ "hot_rate_pct": 76.2638,
372
+ "cold_rate_pct": 23.736199999999997,
373
+ "evictions": 0,
374
+ "per_layer_hit_min": 73.75,
375
+ "per_layer_hit_max": 78.1003,
376
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
377
+ },
378
+ {
379
+ "level_mb": 16384,
380
+ "level_gib": 16.0,
381
+ "repeat": 2,
382
+ "wall_s": 17.45,
383
+ "exit_code": 0,
384
+ "peak_rss_mb": 15084.1,
385
+ "peak_anon_mb": 14600.3,
386
+ "peak_swap_mb": 0.0,
387
+ "swap_free": true,
388
+ "anon_within_limit": true,
389
+ "limit_enforced_by": "prlimit --data (RLIMIT_DATA)",
390
+ "status": "OK",
391
+ "decode_tok_s": 19.25,
392
+ "decode_tok": 31,
393
+ "prefill_tok_s": 10.3,
394
+ "prefill_tok": 56,
395
+ "final_rss_mb": 14570,
396
+ "selfpeak_rss_mb": 14570,
397
+ "self_swap_mb": 0,
398
+ "rss_anon_mb": 14562,
399
+ "rss_file_mb": 8,
400
+ "rss_total_mb": 14570,
401
+ "arena_gib": 14.62,
402
+ "n_slots": 11012,
403
+ "total_ssd_gb": 3.22,
404
+ "decode_ssd_mb": 219.0,
405
+ "ssd_mb_per_token": 7.1,
406
+ "expert_requests": 8773,
407
+ "expert_hits": 6692,
408
+ "hot_rate_pct": 76.2795,
409
+ "cold_rate_pct": 23.7205,
410
+ "evictions": 0,
411
+ "per_layer_hit_min": 73.75,
412
+ "per_layer_hit_max": 78.1003,
413
+ "assistant": "Alright, the user is asking about how a Mixture-of-Experts (MoE) model routes tokens to expert weights and why paging those weights from"
414
+ }
415
+ ]
416
+ }
results/strict_memory_levels.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 嚴格 RAM 上限實測：Ling-3.0-tiny Q4_K_M / EPQ expert paging
2
+
3
+ ## 條件
4
+
5
+ - Intel Xeon Platinum 8559C，192 host threads；CPU-only。
6
+ - 上層 cgroup：`memory.max=104000000000`、`memory.swap.max=2936848384`。
7
+ - ctx=256、ubatch=32、threads=4、temperature=0、生成 32 tokens；每級兩輪。
8
+ - 每輪前以 `posix_fadvise(DONTNEED)` 驅逐模型檔 page cache。
9
+ - `prlimit --data=<level bytes>` 設定匿名記憶體硬上限；build 關閉 `GGML_CPU_REPACK` 和 AMX。
10
+ - anon 峰值由 `/proc/<pid>/smaps_rollup` 的 `AnonHWM` 量測；RSS 也包含不受 RLIMIT_DATA 管制的 file-backed mappings。
11
+
12
+ ## 結果
13
+
14
+ | RAM 上限 | 結果 | arena (GiB) | slots | anon 峰值 (MB) | 餘量 (MB) | RSS 峰值 (MB) | decode tok/s | SSD MiB/tok | hot % | swap |
15
+ |---:|:---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
16
+ | 1 GiB | PASS | 0.30 | 225 | 421 | 603 | 905 | 8.23 / 8.27 | 199.7 / 199.8 | 16.76 / 16.73 | 0 |
17
+ | 2 GiB | PASS | 1.30 | 979 | 1,412 | 636 | 1,895 | 12.01 / 11.75 | 73.8 / 73.8 | 49.47 / 49.48 | 0 |
18
+ | 4 GiB | PASS | 3.30 | 2,486 | 3,404 | 692 | 3,888 | 15.83 / 16.92 | 24.8 / 24.8 | 66.74 / 66.74 | 0 |
19
+ | 8 GiB | PASS | 7.01 | 5,279 | 7,064 | 1,128 | 7,548 | 17.72 / 19.57 | 7.1 / 7.1 | 76.28 / 76.28 | 0 |
20
+ | 16 GiB | PASS | 14.62 | 11,012 / 11,013 | 14,600 | 1,784 | 15,084 | 19.23 / 19.25 | 7.2 / 7.1 | 76.26 / 76.28 | 0 |
21
+
22
+ 各雙值欄依序為 repeat 1 / repeat 2。anon 峰值是本次 RLIMIT_DATA 驗證欄位；file-backed RSS 另外計入總 RSS。8 GiB 與 16 GiB 的 hot rate 和 SSD MiB/token 幾乎相同，在本次 workload 下增加至 16 GiB 沒帶來可見的 paging 收益。吞吐受共享主機負載影響。
23
+
24
+ 原始逐輪數據：[`strict_memory_levels.json`](strict_memory_levels.json)。benchmark 命令與參數記錄於 JSON config；完整可重跑腳本為 [`../sdq-ling/strict_bench.py`](../sdq-ling/strict_bench.py)。
results/verify_dequant.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ blk.2.ffn_gate_exps.weight E0 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
2
+ blk.2.ffn_gate_exps.weight E17 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
3
+ blk.2.ffn_gate_exps.weight E127 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
4
+ blk.2.ffn_up_exps.weight E0 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
5
+ blk.2.ffn_up_exps.weight E17 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
6
+ blk.2.ffn_up_exps.weight E127 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
7
+ blk.2.ffn_down_exps.weight E0 type=q6_K n=512: max|ggml - numpy| = 0 PASS
8
+ blk.2.ffn_down_exps.weight E17 type=q6_K n=512: max|ggml - numpy| = 0 PASS
9
+ blk.2.ffn_down_exps.weight E127 type=q6_K n=512: max|ggml - numpy| = 0 PASS
10
+ blk.5.ffn_gate_exps.weight E0 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
11
+ blk.5.ffn_gate_exps.weight E17 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
12
+ blk.5.ffn_gate_exps.weight E127 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
13
+ blk.5.ffn_down_exps.weight E0 type=q6_K n=512: max|ggml - numpy| = 0 PASS
14
+ blk.5.ffn_down_exps.weight E17 type=q6_K n=512: max|ggml - numpy| = 0 PASS
15
+ blk.5.ffn_down_exps.weight E127 type=q6_K n=512: max|ggml - numpy| = 0 PASS
16
+ blk.20.ffn_gate_exps.weight E0 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
17
+ blk.20.ffn_gate_exps.weight E17 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
18
+ blk.20.ffn_gate_exps.weight E127 type=q4_K n=1536: max|ggml - numpy| = 0 PASS
19
+ blk.23.ffn_down_exps.weight E0 type=q6_K n=512: max|ggml - numpy| = 0 PASS
20
+ blk.23.ffn_down_exps.weight E17 type=q6_K n=512: max|ggml - numpy| = 0 PASS
21
+ blk.23.ffn_down_exps.weight E127 type=q6_K n=512: max|ggml - numpy| = 0 PASS
22
+
23
+ ALL PASS
results/verify_moe_layer.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ layer 20: n_tok=1 n_expert_used=8 n_expert=128 n_embd=1536 n_ff=512
2
+ distinct experts: 8
3
+ max|got-ref| = 0.0261884 max|ref| = 1.9054 max_rel = 0.0137443 corr = 0.99985581
4
+ PASS
results/verify_selftest.txt ADDED
The diff for this file is too large to render. See raw diff
 
results/verify_selftest_numpy.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ arch=bailingmoe3 n_expert=128 n_expert_used=8 n_embd=1536 n_ff_exp=512
2
+ L20 E17: gate=q4_K up=q4_K down=q6_K | max_rel_err=0.0228 corr=0.999685 | PASS
3
+
4
+ ALL PASS