Text Generation
llama-cpp-python
GGUF
llama.cpp
Mixture of Experts
ssd-offload
smallthinker
expert-paging
low-ram
Instructions to use HelloSun/SmallThinker4b_EPQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use HelloSun/SmallThinker4b_EPQ with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="HelloSun/SmallThinker4b_EPQ", filename="{{GGUF_FILE}}", )output = llm( "Once upon a time,", max_tokens=512, echo=True ) print(output)
- Notebooks
- Google Colab
- Kaggle
Auto Upload Agent commited on
Commit ·
f279c31
1
Parent(s): 833ef4d
v10: model card 加上完整實測數據(記憶體/速度/SSD/權重組成);修 pager expert 總量高估 7% 的 bug
Browse files- AGENTS.md +1 -1
- README.md +232 -22
- STATUS.md +11 -8
- patches/0001-st-expert-pager.patch +18 -9
- validate/ab-test.json +516 -315
- validate/last-run.json +16 -16
AGENTS.md
CHANGED
|
@@ -35,7 +35,7 @@ SmallThinker-4B-A0.6B-Instruct(Q4_K_M,2508 MiB,MoE 32 層 × 32 experts)
|
|
| 35 |
不該影響任何一個 token 的數值。
|
| 36 |
|
| 37 |
實測成果(`validate/ab-test.json`,`all_passed: true`):
|
| 38 |
-
peak RSS 從上游的 **2.
|
| 39 |
swap 全程 0,冷權重真的從 SSD 重讀 2.3 GiB,輸出四種設定逐字相同。
|
| 40 |
|
| 41 |
## 已知的坑(不要重蹈)
|
|
|
|
| 35 |
不該影響任何一個 token 的數值。
|
| 36 |
|
| 37 |
實測成果(`validate/ab-test.json`,`all_passed: true`):
|
| 38 |
+
peak RSS 從上游的 **2.565 GiB** 降到 **0.596 GiB**(預算 256 MiB,−76.8%),
|
| 39 |
swap 全程 0,冷權重真的從 SSD 重讀 2.3 GiB,輸出四種設定逐字相同。
|
| 40 |
|
| 41 |
## 已知的坑(不要重蹈)
|
README.md
CHANGED
|
@@ -1,37 +1,247 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# SmallThinker-4B-A0.6B — 熱參數在 RAM、冷參數在 SSD
|
| 2 |
|
| 3 |
-
|
| 4 |
-
`SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf`(2
|
| 5 |
-
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
```bash
|
| 9 |
-
./llama_server.sh
|
| 10 |
-
./llama_server.sh --plan
|
| 11 |
-
./llama_server.sh --verify
|
| 12 |
-
ST_RAM_BUDGET_MB=
|
|
|
|
| 13 |
```
|
| 14 |
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
-
|
| 18 |
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
| --- | --- | --- | --- | --- |
|
| 21 |
-
|
|
| 22 |
-
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## 檔案
|
| 28 |
|
| 29 |
| 檔案 | 用途 |
|
| 30 |
| --- | --- |
|
| 31 |
-
| `llama_server.sh` | 一鍵啟動(自動抓 llama.cpp / 套 patch / 編譯 / 下載權重) |
|
| 32 |
-
| `patches/` | llama.cpp 改動
|
| 33 |
-
| `tools/
|
| 34 |
-
| `tools/
|
| 35 |
-
| `tools/
|
| 36 |
-
| `
|
| 37 |
-
| `
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: llama-cpp-python
|
| 4 |
+
tags:
|
| 5 |
+
- gguf
|
| 6 |
+
- llama.cpp
|
| 7 |
+
- moe
|
| 8 |
+
- ssd-offload
|
| 9 |
+
- smallthinker
|
| 10 |
+
- expert-paging
|
| 11 |
+
- low-ram
|
| 12 |
+
base_model:
|
| 13 |
+
- Tiiny/SmallThinker-4BA0.6B-Instruct-GGUF
|
| 14 |
+
pipeline_tag: text-generation
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
# SmallThinker-4B-A0.6B — 熱參數在 RAM、冷參數在 SSD
|
| 18 |
|
| 19 |
+
在 llama.cpp 上跑 [`Tiiny/SmallThinker-4BA0.6B-Instruct-GGUF`](https://huggingface.co/Tiiny/SmallThinker-4BA0.6B-Instruct-GGUF)
|
| 20 |
+
的 `SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf`(2,630,212,704 bytes = 2508 MiB),
|
| 21 |
+
**expert 級 SSD 分頁**:權重永遠是 SSD 上的 file-backed `mmap`,RAM 只留熱的
|
| 22 |
+
expert 權重,用不到的 expert 會被明確丟回 SSD。
|
| 23 |
+
|
| 24 |
+
**計算路徑完全沒動過** —— 仍然是 llama.cpp 自己的 `mul_mat_id`,
|
| 25 |
+
所以輸出與上游**逐字元相同**。這是本專案唯一不妥協的正確性條件。
|
| 26 |
|
| 27 |
```bash
|
| 28 |
+
./llama_server.sh # port 8080
|
| 29 |
+
./llama_server.sh --plan # 只印推導出來的參數
|
| 30 |
+
./llama_server.sh --verify # 啟動 + 打一次 chat + 量記憶體
|
| 31 |
+
ST_RAM_BUDGET_MB=256 ./llama_server.sh # 手動壓低 RAM 預算
|
| 32 |
+
ST_PAGER=0 ./llama_server.sh # 關閉分頁,回到上游行為
|
| 33 |
```
|
| 34 |
|
| 35 |
+
---
|
| 36 |
+
|
| 37 |
+
## 實測結果
|
| 38 |
+
|
| 39 |
+
本機:16 vCPU、**無 GPU**、檔案系統 **overlayfs**、2026-10-07。
|
| 40 |
+
完整證據:[`validate/ab-test.json`](validate/ab-test.json)(`all_passed: true`)、
|
| 41 |
+
[`validate/last-run.json`](validate/last-run.json)。
|
| 42 |
+
|
| 43 |
+
### 記憶體 vs 分頁預算
|
| 44 |
+
|
| 45 |
+
同一份 `llama-cli`、同一個 prompt、同一個 seed(`--seed 42`)、ctx 1024、8 threads:
|
| 46 |
+
|
| 47 |
+
| 設定 | peak 總 RSS | 匿名 | 檔案對映 | swap | SSD 重讀 | 命中率 | 淘汰次數 |
|
| 48 |
+
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 49 |
+
| `ST_PAGER=0`(上游) | **2.565 GiB** | 0.103 | 2.559 | **0** | — | — | — |
|
| 50 |
+
| 預算 1024 MiB | **1.301 GiB** | 0.103 | 1.300 | **0** | 2317 MiB | 57.4% | 2,894 |
|
| 51 |
+
| 預算 512 MiB | **0.833 GiB** | 0.103 | 0.836 | **0** | 2316 MiB | 55.1% | 4,572 |
|
| 52 |
+
| 預算 256 MiB | **0.596 GiB** | 0.103 | 0.592 | **0** | 2320 MiB | 53.7% | 5,625 |
|
| 53 |
+
| 預算 128 MiB | **0.661 GiB** | 0.103 | 0.657 | **0** | 2317 MiB | 54.1% | 5,298 |
|
| 54 |
|
| 55 |
+
**總 RSS 從 2.565 GiB 降到 0.596 GiB(−76.8%),swap 全程 0。**
|
| 56 |
|
| 57 |
+
> 量的是**總 RSS**(含 file-backed mmap 的權重頁),不是只有匿名記憶體 ——
|
| 58 |
+
> 權重頁會算進 RSS 且真的佔用系統 RAM。只看匿名記憶體會嚴重低估(本實測 0.103 GiB)。
|
| 59 |
+
> 峰值由**模型載入**決定,不是 prefill:實測 prompt 2 token 與 211 token 的峰值幾乎一樣。
|
| 60 |
+
|
| 61 |
+
`預算 128 MiB` 反而比 `256 MiB` 高一點:預算已逼近 hot window 內必須保留的下限
|
| 62 |
+
(`ST_HOT_TOKENS=8` × 32 層 × 4 experts/層),再壓沒有意義,命中率也沒比較好。
|
| 63 |
+
|
| 64 |
+
### 速度代價
|
| 65 |
+
|
| 66 |
+
| 設定 | prefill | decode | 耗時 |
|
| 67 |
+
| --- | --- | --- | --- |
|
| 68 |
+
| `ST_PAGER=0`(上游,權重全在 RAM) | 178.7 t/s | **55.4 t/s** | 9.06 s |
|
| 69 |
+
| 預算 1024 MiB | 7.8 t/s | **22.9 t/s** | 7.85 s |
|
| 70 |
+
| 預算 512 MiB | 7.9 t/s | **21.6 t/s** | 7.67 s |
|
| 71 |
+
| 預算 256 MiB | 8.1 t/s | **19.7 t/s** | 7.88 s |
|
| 72 |
+
| 預算 128 MiB | 7.9 t/s | **20.9 t/s** | 8.13 s |
|
| 73 |
+
|
| 74 |
+
**用約 2.4~2.8× 的 decode 速度換 3~4.3× 的記憶體。** 這是 demand paging 的必然代價:
|
| 75 |
+
被丟掉的 expert 下次要用時必須重新讀 SSD。
|
| 76 |
+
|
| 77 |
+
### 本機 SSD 讀取能力([`validate/io-probe.json`](validate/io-probe.json))
|
| 78 |
+
|
| 79 |
+
| threads | 1 | 2 | 4 | 8 |
|
| 80 |
| --- | --- | --- | --- | --- |
|
| 81 |
+
| 循序讀取 | 337 MB/s | 667 MB/s | 1328 MB/s | 2558 MB/s |
|
| 82 |
+
|
| 83 |
+
4K 隨機讀 ≈ 1767 IOPS。**換機器務必用 `tools/io_probe.py` 重測**,
|
| 84 |
+
decode 速度幾乎完全由它決定。
|
| 85 |
+
|
| 86 |
+
### 端到端(`llama_server.sh --verify`,`ST_RAM_BUDGET_MB=512`)
|
| 87 |
+
|
| 88 |
+
| 項目 | 值 |
|
| 89 |
+
| --- | --- |
|
| 90 |
+
| peak 總 RSS | 1.090 GiB |
|
| 91 |
+
| VmSwap | **0** |
|
| 92 |
+
| expert 常駐 / 預算 | **510.3 / 512.0 MiB** ✅ |
|
| 93 |
+
| 分頁命中率 | 41.2% |
|
| 94 |
+
| `read_bytes`(整個行程) | 2.44 GiB |
|
| 95 |
+
| chat 輸出 | `A **MoE (Model Parallelism) layer** enables a neural network to` |
|
| 96 |
+
|
| 97 |
+
1.090 GiB 的組成:非 expert 權重 410 MiB(常駐)+ expert 分頁 510 MiB
|
| 98 |
+
+ KV / compute buffer / 執行檔(llama-server 開 4 個 slot)。
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
+
## 模型事實(全部從 GGUF metadata 實測,不是抄設定檔)
|
| 103 |
+
|
| 104 |
+
| 項目 | 值 |
|
| 105 |
+
| --- | --- |
|
| 106 |
+
| arch | `smallthinker` |
|
| 107 |
+
| 層數 | 32 |
|
| 108 |
+
| hidden | 1536 |
|
| 109 |
+
| experts / 每 token 用 | 32 / 4 |
|
| 110 |
+
| expert FFN | 768 |
|
| 111 |
+
| attention heads / KV heads | 12 / 2 |
|
| 112 |
+
| key_len / val_len | 128 / 128 |
|
| 113 |
+
| context_length(模型上限) | 32768 |
|
| 114 |
+
| 量化 | Q4_K_M |
|
| 115 |
+
| 檔案 | 2,630,212,704 bytes(2508 MiB) |
|
| 116 |
+
|
| 117 |
+
### 權重組成
|
| 118 |
+
|
| 119 |
+
| | 位元組 | 佔檔案 |
|
| 120 |
+
| --- | --- | --- |
|
| 121 |
+
| **expert 權重(1024 個)** | 2,194,145,280(**2092.5 MiB**) | **83.4%** |
|
| 122 |
+
| 其餘(embedding / attention / norms / router) | 430,065,024(410.2 MiB) | 16.6% |
|
| 123 |
+
|
| 124 |
+
expert 佔 83.4% —— 所以「熱在 RAM、冷在 SSD」這件事基本上就是
|
| 125 |
+
「expert 權重怎麼放」。
|
| 126 |
+
|
| 127 |
+
**每個 expert 是三段連續位元組**(`blk.N.ffn_{gate,up,down}_exps.weight`,
|
| 128 |
+
expert 維度是最外層 `ne[2]`,所以單一 expert 是一段連續區間):
|
| 129 |
+
|
| 130 |
+
| 段 | shape | layer 0 每個 expert |
|
| 131 |
+
| --- | --- | --- |
|
| 132 |
+
| gate | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 133 |
+
| up | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 134 |
+
| down | [768, 1536, 32] | 967,680 B(**Q6_K**) |
|
| 135 |
+
| 合計 | | 2.19 MiB |
|
| 136 |
|
| 137 |
+
⚠️ **`down_exps` 的量化型別逐層不同**:32 層裡有 **16 層是 Q6_K、16 層是 Q4_K**
|
| 138 |
+
(層號 0,1,2,3,6,9,12,15,18,21,24,27,28,29,30,31 是 Q6_K)。
|
| 139 |
+
所以「一個 expert 多大」必須**逐層算**:Q6_K 層 70.03 MiB/層、Q4_K 層 60.75 MiB/層,
|
| 140 |
+
平均 2.04 MiB/個。拿 layer 0 當全模型常數會把總量高估 148 MiB(7%)。
|
| 141 |
+
|
| 142 |
+
---
|
| 143 |
+
|
| 144 |
+
## 機制
|
| 145 |
+
|
| 146 |
+
```
|
| 147 |
+
llama.cpp 正常載入(權重留在 SSD 的 file-backed mmap)
|
| 148 |
+
│
|
| 149 |
+
├─ 每層 top-k 選完 expert 之後,插入一個 ggml custom op
|
| 150 |
+
│ ├─ 讀出這一層這批 token 用到哪些 expert,標成「熱」
|
| 151 |
+
│ └─ 若 resident 超過預算 → 用 LFU→LRU 順序淘汰最冷的
|
| 152 |
+
│ a. madvise(MADV_DONTNEED) 拿掉本行程 PTE → RSS 下降
|
| 153 |
+
│ b. posix_fadvise(DONTNEED) 丟掉 page cache → 下次真的讀 SSD
|
| 154 |
+
│
|
| 155 |
+
└─ 計算本身:完全不動,仍是上游 ggml_mul_mat_id
|
| 156 |
+
```
|
| 157 |
+
|
| 158 |
+
淘汰的兩步驟**順序不能換**:
|
| 159 |
+
只做 (a) → RSS 下降但 `read_bytes` 不變(頁還在 page cache);
|
| 160 |
+
只做 (b) → page cache 下降但行程 RSS 不變(VMA 還映射著)。
|
| 161 |
+
(核心的 `invalidate_mapping_pages()` 會跳過仍被 VMA 映射的頁,所以必須先 (a)。)
|
| 162 |
+
|
| 163 |
+
### 與 `HelloSun/sddqwen35a3b_v01` 的取捨
|
| 164 |
+
|
| 165 |
+
| | sddqwen35a3b_v01 | 本專案 |
|
| 166 |
+
| --- | --- | --- |
|
| 167 |
+
| 權重怎麼進 RAM | `pread` 進自己配置的 arena | 留在 llama.cpp 自己的 mmap |
|
| 168 |
+
| MoE 計算 | **自訂 ggml op**,自己重寫 Q4_K 點積 | **不動**,用上游 `mul_mat_id` |
|
| 169 |
+
| 輸出正確性 | 要另外驗證 kernel 寫對沒有 | 逐字等於上游,結構上不可能錯 |
|
| 170 |
+
| SSD 讀取 | 預取可與計算重疊 | demand paging(碰到缺頁才讀) |
|
| 171 |
+
| 程式碼量 | ~2800 行 | ~550 行 |
|
| 172 |
+
|
| 173 |
+
**放棄 I/O 預取,換取「計算路徑完全沒動過」的保證。**
|
| 174 |
+
|
| 175 |
+
---
|
| 176 |
+
|
| 177 |
+
## 環境變數
|
| 178 |
+
|
| 179 |
+
| 變數 | 預設 | 說明 |
|
| 180 |
+
| --- | --- | --- |
|
| 181 |
+
| `ST_PAGER` | `1` | `0` = 關閉分頁(上游行為,做 A/B 用) |
|
| 182 |
+
| `ST_RAM_BUDGET_MB` | 自動偵測 | expert 權重可以常駐 RAM 的上限 |
|
| 183 |
+
| `ST_RESERVE_MB` | `0` | KV + compute 預留,從預算扣掉 |
|
| 184 |
+
| `ST_ARENA_MB` | `0` | 直接指定 expert 常駐量(有值優先) |
|
| 185 |
+
| `ST_HOT_TOKENS` | `8` | 最近 N 個 tick 用過的 expert 不淘汰 |
|
| 186 |
+
| `ST_VERBOSE` | `0` | 印每次淘汰 + 實際 RSS |
|
| 187 |
+
| `ST_STATS_FILE` | — | 行程退出時把統計寫成 JSON(`SIGUSR2` 也會寫) |
|
| 188 |
+
| `ST_EVICT_MODE` | `both` | 診斷用:`both`/`madvise`/`fadvise`/`none` |
|
| 189 |
+
|
| 190 |
+
## ⚠️ 必看的坑
|
| 191 |
+
|
| 192 |
+
1. **`-DGGML_CPU_REPACK=OFF` 是硬性要求**(`llama_server.sh` 已自動加上)。
|
| 193 |
+
開著時 ggml 把 Q4_K 轉成 repack 格式放進**匿名**緩衝區,權重就離開 mmap。
|
| 194 |
+
這時 `madvise(MADV_DONTNEED)` 不是「讀 SSD」,而是**把那塊記憶體歸零**
|
| 195 |
+
→ 輸出變成 `papers bases abstract…` 之類的亂碼,但行程看起來完全正常、
|
| 196 |
+
tok/s 只掉一點。patch 裡也有 `in_map` 檢查會擋下並印出 buffer 型別。
|
| 197 |
+
2. **分頁開著時一定要關掉 `MAP_POPULATE`**,否則整個 2.45 GiB 會在
|
| 198 |
+
「載入模型」那一瞬間全部 fault 進 page cache,峰值立刻爆掉 ——
|
| 199 |
+
而且分頁統計看起來完全正常(sweep 有跑、帳面 resident 也對),光看統計抓不到。
|
| 200 |
+
|
| 201 |
+
完整清單(每條都有實測證據)在 [`STATUS.md`](STATUS.md) 的「踩過的坑」表格。
|
| 202 |
|
| 203 |
## 檔案
|
| 204 |
|
| 205 |
| 檔案 | 用途 |
|
| 206 |
| --- | --- |
|
| 207 |
+
| [`llama_server.sh`](llama_server.sh) | 一鍵啟動(自動抓 llama.cpp / 套 patch / 編譯 / 下載權重) |
|
| 208 |
+
| [`patches/0001-st-expert-pager.patch`](patches/0001-st-expert-pager.patch) | llama.cpp 改動 |
|
| 209 |
+
| [`tools/ab_test.py`](tools/ab_test.py) | A/B 驗證(RSS / swap / SSD 讀取 / 輸出比對) |
|
| 210 |
+
| [`tools/verify_run.py`](tools/verify_run.py) | 對 llama-server 打 chat 並量記憶體 |
|
| 211 |
+
| [`tools/io_probe.py`](tools/io_probe.py) | 實測本機 SSD 讀取能力(**換機器必跑**) |
|
| 212 |
+
| [`tools/drop_model_cache.py`](tools/drop_model_cache.py) | 量測前丟掉模型檔的 page cache |
|
| 213 |
+
| [`STATUS.md`](STATUS.md) | **進度真相來源** |
|
| 214 |
+
| [`AGENTS.md`](AGENTS.md) | 續作指引(假設 agent 崩潰後只看這兩個檔案) |
|
| 215 |
+
|
| 216 |
+
## 重現
|
| 217 |
+
|
| 218 |
+
```bash
|
| 219 |
+
git clone https://huggingface.co/HelloSun/SmallThinker4b
|
| 220 |
+
cd SmallThinker4b
|
| 221 |
+
|
| 222 |
+
# 端到端驗證
|
| 223 |
+
ST_RAM_BUDGET_MB=512 ./llama_server.sh --verify
|
| 224 |
+
|
| 225 |
+
# A/B 驗證(baseline vs 各種預算)
|
| 226 |
+
python3 tools/ab_test.py --bin <path>/llama-cli --model <path>/model.gguf \
|
| 227 |
+
--budget 1024 --budget 512 --budget 256 --out validate/ab-test.json
|
| 228 |
+
|
| 229 |
+
# 本機 SSD 讀取能力
|
| 230 |
+
python3 tools/io_probe.py --model <path>/model.gguf --out validate/io-probe.json
|
| 231 |
+
```
|
| 232 |
+
|
| 233 |
+
## 進度與續作
|
| 234 |
+
|
| 235 |
+
工作成果保存在這個 repo(`./sync.sh "改了什麼"` 即可上傳)。
|
| 236 |
+
**假設 agent 崩潰**:clone 這個 repo → 讀 `STATUS.md` → 讀 `AGENTS.md` → 接著做。
|
| 237 |
+
本機很不穩,**每改完程式並編譯過就上傳,不要累積**。
|
| 238 |
+
|
| 239 |
+
尚未做(見 `STATUS.md` 的「下一步」):I/O 預取(讓 SSD 讀取與計算重疊)、
|
| 240 |
+
KV cache 也放 SSD、限制 prefill 批次大小以壓低 prefill 階段峰值。
|
| 241 |
+
|
| 242 |
+
## 授權
|
| 243 |
+
|
| 244 |
+
本 repo 的程式碼(`llama_server.sh` / `patches/` / `tools/`)沿用
|
| 245 |
+
llama.cpp 的 MIT。模型權重屬於
|
| 246 |
+
[`Tiiny/SmallThinker-4BA0.6B-Instruct-GGUF`](https://huggingface.co/Tiiny/SmallThinker-4BA0.6B-Instruct-GGUF)
|
| 247 |
+
(Apache-2.0),本 repo **不含**權重。
|
STATUS.md
CHANGED
|
@@ -23,7 +23,7 @@ file-backed `mmap`;RAM 只留「熱的」expert 權重 + KV + compute buffer
|
|
| 23 |
| context_length(模型上限) | 32768 |
|
| 24 |
| 量化 | Q4_K_M |
|
| 25 |
|
| 26 |
-
**expert 權重總量
|
| 27 |
這件事基本上就是「expert 權重怎麼放」。
|
| 28 |
|
| 29 |
每個 expert 是三段連續位元組(`blk.N.ffn_{gate,up,down}_exps.weight`,
|
|
@@ -34,9 +34,12 @@ expert 維度是最外層):
|
|
| 34 |
| gate | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 35 |
| up | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 36 |
| down | [768, 1536, 32] | 967,680 B(Q6_K) |
|
| 37 |
-
| **合計** | | **2,294,784 B
|
| 38 |
|
| 39 |
-
共 32 × 32 = 1024 個 expert。
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
⚠️ **三段的量化型別逐層可能不同**(實測有些層的 `down_exps` 是 Q4_K 而非 Q6_K,
|
| 42 |
`nb[2` 就不同)。所以「每個 expert 多大」必須**逐 slot 算**,
|
|
@@ -55,11 +58,11 @@ expert 維度是最外層):
|
|
| 55 |
|
| 56 |
| 設定 | peak 總 RSS | swap | SSD 重讀 | 命中率 | 淘汰次數 | 輸出逐字相同 |
|
| 57 |
| --- | --- | --- | --- | --- | --- | --- |
|
| 58 |
-
| `ST_PAGER=0`(上游行為) | **2.
|
| 59 |
| 預算 1024 MiB | **1.301 GiB** | 0 | 2317 MiB | 57.4% | 2894 | ✅ |
|
| 60 |
-
| 預算 512 MiB | **0.833 GiB** | 0 |
|
| 61 |
-
| 預算 256 MiB | **0.
|
| 62 |
-
| 預算 128 MiB | **0.
|
| 63 |
|
| 64 |
同一份 `llama-cli`、同一個 prompt、同一個 seed,四種設定的輸出**完全相同**:
|
| 65 |
|
|
@@ -76,7 +79,7 @@ expert 維度是最外層):
|
|
| 76 |
|
| 77 |
| 項目 | 值 |
|
| 78 |
| --- | --- |
|
| 79 |
-
| peak 總 RSS | **1.
|
| 80 |
| VmSwap | **0** |
|
| 81 |
| expert 常駐 / 預算 | **510.3 / 512.0 MiB** ✅ |
|
| 82 |
| 命中率 | 41.2% |
|
|
|
|
| 23 |
| context_length(模型上限) | 32768 |
|
| 24 |
| 量化 | Q4_K_M |
|
| 25 |
|
| 26 |
+
**expert 權重總量 2092.5 MiB / 2508 MiB = 83.4%。** 也就是說「熱在 RAM、冷在 SSD」
|
| 27 |
這件事基本上就是「expert 權重怎麼放」。
|
| 28 |
|
| 29 |
每個 expert 是三段連續位元組(`blk.N.ffn_{gate,up,down}_exps.weight`,
|
|
|
|
| 34 |
| gate | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 35 |
| up | [1536, 768, 32] | 663,552 B(Q4_K) |
|
| 36 |
| down | [768, 1536, 32] | 967,680 B(Q6_K) |
|
| 37 |
+
| **合計** | | **2,294,784 B = 2.19 MiB** |
|
| 38 |
|
| 39 |
+
共 32 × 32 = 1024 個 expert,整體平均 2.04 MiB/個(因為 16 層的 down 是 Q4_K)。
|
| 40 |
+
|
| 41 |
+
逐層實測(`tools` 之外,用 gguf-py 直接算):Q6_K down 的層 70.03 MiB/層、
|
| 42 |
+
Q4_K down 的層 60.75 MiB/層;Q6_K 的層號是 0,1,2,3,6,9,12,15,18,21,24,27,28,29,30,31。
|
| 43 |
|
| 44 |
⚠️ **三段的量化型別逐層可能不同**(實測有些層的 `down_exps` 是 Q4_K 而非 Q6_K,
|
| 45 |
`nb[2` 就不同)。所以「每個 expert 多大」必須**逐 slot 算**,
|
|
|
|
| 58 |
|
| 59 |
| 設定 | peak 總 RSS | swap | SSD 重讀 | 命中率 | 淘汰次數 | 輸出逐字相同 |
|
| 60 |
| --- | --- | --- | --- | --- | --- | --- |
|
| 61 |
+
| `ST_PAGER=0`(上游行為) | **2.565 GiB** | 0 | — | — | — | 基準 |
|
| 62 |
| 預算 1024 MiB | **1.301 GiB** | 0 | 2317 MiB | 57.4% | 2894 | ✅ |
|
| 63 |
+
| 預算 512 MiB | **0.833 GiB** | 0 | 2316 MiB | 55.1% | 4572 | ✅ |
|
| 64 |
+
| 預算 256 MiB | **0.596 GiB** | 0 | 2320 MiB | 53.7% | 5625 | ✅ |
|
| 65 |
+
| 預算 128 MiB | **0.661 GiB** | 0 | 2317 MiB | 54.1% | 5298 | ✅ |
|
| 66 |
|
| 67 |
同一份 `llama-cli`、同一個 prompt、同一個 seed,四種設定的輸出**完全相同**:
|
| 68 |
|
|
|
|
| 79 |
|
| 80 |
| 項目 | 值 |
|
| 81 |
| --- | --- |
|
| 82 |
+
| peak 總 RSS | **1.090 GiB**(非 expert 權重 410 MiB + expert 分頁 510 MiB + KV/compute/sampler) |
|
| 83 |
| VmSwap | **0** |
|
| 84 |
| expert 常駐 / 預算 | **510.3 / 512.0 MiB** ✅ |
|
| 85 |
| 命中率 | 41.2% |
|
patches/0001-st-expert-pager.patch
CHANGED
|
@@ -126,10 +126,10 @@ index 1f3b80b08..c7b77476e 100644
|
|
| 126 |
|
| 127 |
diff --git a/src/st_pager.cpp b/src/st_pager.cpp
|
| 128 |
new file mode 100644
|
| 129 |
-
index 000000000..
|
| 130 |
--- /dev/null
|
| 131 |
+++ b/src/st_pager.cpp
|
| 132 |
-
@@ -0,0 +1,
|
| 133 |
+// ST expert pager —— 熱 expert 留在 RAM,冷 expert 丟回 SSD
|
| 134 |
+//
|
| 135 |
+// 設計取捨(為什麼不照抄 sddqwen35a3b_v01 的 pread arena):
|
|
@@ -409,8 +409,13 @@ index 000000000..9cdbc0068
|
|
| 409 |
+ if (il == 0) {
|
| 410 |
+ n_expert = (int) ne2;
|
| 411 |
+ g_part_bytes[p] = nb2;
|
| 412 |
-
+
|
| 413 |
-
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 414 |
+ // 只要求 expert **數量**一致。三段的量化型別可以逐層不同
|
| 415 |
+ // (實測 down_exps 有些層是 Q6_K、有些是 Q4_K),
|
| 416 |
+ // 所以位元組數逐 slot 存,不拿 layer 0 當基準。
|
|
@@ -427,7 +432,7 @@ index 000000000..9cdbc0068
|
|
| 427 |
+ g_n_expert = n_expert;
|
| 428 |
+ g_slot_bytes = slot_bytes;
|
| 429 |
+ g_n_layer = (int) layers.size();
|
| 430 |
-
+ const size_t total_expert_bytes = slot_bytes * (size_t) g_n_expert
|
| 431 |
+
|
| 432 |
+ g_slots.assign((size_t) g_n_layer * g_n_expert, slot{});
|
| 433 |
+ for (int il = 0; il < g_n_layer; il++) {
|
|
@@ -470,11 +475,15 @@ index 000000000..9cdbc0068
|
|
| 470 |
+ }
|
| 471 |
+
|
| 472 |
+ fprintf(stderr,
|
| 473 |
-
+ "[st] expert pager 就緒:%d 層 × %d experts = %d 個,"
|
| 474 |
-
+ "
|
| 475 |
-
+ "
|
|
|
|
| 476 |
+ g_n_layer, g_n_expert, g_n_layer * g_n_expert,
|
| 477 |
-
+
|
|
|
|
|
|
|
|
|
|
| 478 |
+ (double) total_expert_bytes / (1024.0 * 1024.0),
|
| 479 |
+ (double) g_budget / (1024.0 * 1024.0), g_hot_tokens);
|
| 480 |
+ return true;
|
|
|
|
| 126 |
|
| 127 |
diff --git a/src/st_pager.cpp b/src/st_pager.cpp
|
| 128 |
new file mode 100644
|
| 129 |
+
index 000000000..12da88c88
|
| 130 |
--- /dev/null
|
| 131 |
+++ b/src/st_pager.cpp
|
| 132 |
+
@@ -0,0 +1,588 @@
|
| 133 |
+// ST expert pager —— 熱 expert 留在 RAM,冷 expert 丟回 SSD
|
| 134 |
+//
|
| 135 |
+// 設計取捨(為什麼不照抄 sddqwen35a3b_v01 的 pread arena):
|
|
|
|
| 409 |
+ if (il == 0) {
|
| 410 |
+ n_expert = (int) ne2;
|
| 411 |
+ g_part_bytes[p] = nb2;
|
| 412 |
+
+ }
|
| 413 |
+
+ // **逐層**累加,不能用 layer 0 的大小代表全部:
|
| 414 |
+
+ // 實測這個模型有 16 層的 down_exps 是 Q6_K、16 層是 Q4_K,
|
| 415 |
+
+ // 用 layer 0 會把總量高估 148 MiB(2092 → 2241 MiB,差 7%)。
|
| 416 |
+
+ slot_bytes += nb2;
|
| 417 |
+
+
|
| 418 |
+
+ if ((int) ne2 != n_expert) {
|
| 419 |
+ // 只要求 expert **數量**一致。三段的量化型別可以逐層不同
|
| 420 |
+ // (實測 down_exps 有些層是 Q6_K、有些是 Q4_K),
|
| 421 |
+ // 所以位元組數逐 slot 存,不拿 layer 0 當基準。
|
|
|
|
| 432 |
+ g_n_expert = n_expert;
|
| 433 |
+ g_slot_bytes = slot_bytes;
|
| 434 |
+ g_n_layer = (int) layers.size();
|
| 435 |
+
+ const size_t total_expert_bytes = slot_bytes * (size_t) g_n_expert;
|
| 436 |
+
|
| 437 |
+ g_slots.assign((size_t) g_n_layer * g_n_expert, slot{});
|
| 438 |
+ for (int il = 0; il < g_n_layer; il++) {
|
|
|
|
| 475 |
+ }
|
| 476 |
+
|
| 477 |
+ fprintf(stderr,
|
| 478 |
+
+ "[st] expert pager 就緒:%d 層 × %d experts = %d 個 expert,"
|
| 479 |
+
+ "expert 權重總量 %.1f MiB(平均每個 %.2f MiB;逐層量化型別不同,"
|
| 480 |
+
+ "以 layer 0 為例是 gate %zu + up %zu + down %zu = %.2f MiB),"
|
| 481 |
+
+ "預算 %.1f MiB,hot window %d tick\n",
|
| 482 |
+ g_n_layer, g_n_expert, g_n_layer * g_n_expert,
|
| 483 |
+
+ (double) total_expert_bytes / (1024.0 * 1024.0),
|
| 484 |
+
+ (double) total_expert_bytes / (double) (g_n_layer * g_n_expert) / (1024.0 * 1024.0),
|
| 485 |
+
+ g_part_bytes[0], g_part_bytes[1], g_part_bytes[2],
|
| 486 |
+
+ (double) (g_part_bytes[0] + g_part_bytes[1] + g_part_bytes[2]) / (1024.0 * 1024.0),
|
| 487 |
+ (double) total_expert_bytes / (1024.0 * 1024.0),
|
| 488 |
+ (double) g_budget / (1024.0 * 1024.0), g_hot_tokens);
|
| 489 |
+ return true;
|
validate/ab-test.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"when": "2026-10-07T16:
|
| 3 |
"model": "/root/work/models/SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf",
|
| 4 |
"model_size_mib": 2508,
|
| 5 |
"prompt": "Hello, who are you?",
|
|
@@ -12,183 +12,376 @@
|
|
| 12 |
"ST_PAGER": "0"
|
| 13 |
},
|
| 14 |
"rc": 0,
|
| 15 |
-
"seconds":
|
| 16 |
"peak": {
|
| 17 |
-
"total_rss_gb": 2.
|
| 18 |
"anon_rss_gb": 0.1033,
|
| 19 |
-
"file_rss_gb": 2.
|
| 20 |
"peak_swap_gb": 0.0,
|
| 21 |
-
"hwm_rss_gb": 2.
|
| 22 |
},
|
| 23 |
"decode": {
|
| 24 |
-
"total_rss_gb": 2.
|
| 25 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 26 |
},
|
| 27 |
"rss_timeline_mib": [
|
| 28 |
[
|
| 29 |
-
0.
|
| 30 |
-
0.
|
| 31 |
],
|
| 32 |
[
|
| 33 |
-
0.
|
| 34 |
-
|
| 35 |
],
|
| 36 |
[
|
| 37 |
-
0.
|
| 38 |
-
|
| 39 |
],
|
| 40 |
[
|
| 41 |
-
0.
|
| 42 |
-
|
| 43 |
],
|
| 44 |
[
|
| 45 |
0.825,
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
],
|
| 48 |
[
|
| 49 |
-
1.
|
| 50 |
-
|
| 51 |
],
|
| 52 |
[
|
| 53 |
-
1.
|
| 54 |
-
|
| 55 |
],
|
| 56 |
[
|
| 57 |
-
1.
|
| 58 |
-
|
| 59 |
],
|
| 60 |
[
|
| 61 |
-
|
| 62 |
-
|
| 63 |
],
|
| 64 |
[
|
| 65 |
-
|
| 66 |
-
|
| 67 |
],
|
| 68 |
[
|
| 69 |
-
2.
|
| 70 |
-
|
| 71 |
],
|
| 72 |
[
|
| 73 |
-
2.
|
| 74 |
-
|
| 75 |
],
|
| 76 |
[
|
| 77 |
-
2.
|
| 78 |
-
|
| 79 |
],
|
| 80 |
[
|
| 81 |
-
|
| 82 |
-
|
| 83 |
],
|
| 84 |
[
|
| 85 |
-
|
| 86 |
-
|
| 87 |
],
|
| 88 |
[
|
| 89 |
-
3.
|
| 90 |
-
|
| 91 |
],
|
| 92 |
[
|
| 93 |
-
3.
|
| 94 |
-
|
| 95 |
],
|
| 96 |
[
|
| 97 |
-
|
| 98 |
-
|
| 99 |
],
|
| 100 |
[
|
| 101 |
-
|
| 102 |
-
|
| 103 |
],
|
| 104 |
[
|
| 105 |
-
4.
|
| 106 |
-
|
| 107 |
],
|
| 108 |
[
|
| 109 |
-
4.
|
| 110 |
-
|
| 111 |
],
|
| 112 |
[
|
| 113 |
-
4.
|
| 114 |
-
|
| 115 |
],
|
| 116 |
[
|
| 117 |
-
|
| 118 |
-
|
| 119 |
],
|
| 120 |
[
|
| 121 |
-
|
| 122 |
-
|
| 123 |
],
|
| 124 |
[
|
| 125 |
-
5.
|
| 126 |
-
|
| 127 |
],
|
| 128 |
[
|
| 129 |
-
5.
|
| 130 |
-
|
| 131 |
],
|
| 132 |
[
|
| 133 |
-
|
| 134 |
-
|
| 135 |
],
|
| 136 |
[
|
| 137 |
-
|
| 138 |
-
|
| 139 |
],
|
| 140 |
[
|
| 141 |
-
6.
|
| 142 |
-
|
| 143 |
],
|
| 144 |
[
|
| 145 |
-
6.
|
| 146 |
-
|
| 147 |
],
|
| 148 |
[
|
| 149 |
-
6.
|
| 150 |
-
|
| 151 |
],
|
| 152 |
[
|
| 153 |
-
|
| 154 |
-
|
| 155 |
],
|
| 156 |
[
|
| 157 |
-
7.
|
| 158 |
-
|
| 159 |
],
|
| 160 |
[
|
| 161 |
-
7.
|
| 162 |
-
|
| 163 |
],
|
| 164 |
[
|
| 165 |
-
7.
|
| 166 |
-
|
| 167 |
],
|
| 168 |
[
|
| 169 |
-
|
| 170 |
-
|
| 171 |
],
|
| 172 |
[
|
| 173 |
-
8.
|
| 174 |
-
|
| 175 |
],
|
| 176 |
[
|
| 177 |
-
8.
|
| 178 |
-
|
| 179 |
],
|
| 180 |
[
|
| 181 |
-
8.
|
| 182 |
-
|
| 183 |
]
|
| 184 |
],
|
| 185 |
"io_delta": {},
|
| 186 |
-
"timing_line": "[ Prompt:
|
| 187 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 188 |
"output_matches_baseline": true,
|
| 189 |
"swap_zero": true,
|
| 190 |
"rc_ok": true
|
| 191 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
"budget512mb": {
|
| 193 |
"env": {
|
| 194 |
"ST_PAGER": "1",
|
|
@@ -196,154 +389,158 @@
|
|
| 196 |
"ST_RESERVE_MB": "200"
|
| 197 |
},
|
| 198 |
"rc": 0,
|
| 199 |
-
"seconds": 7.
|
| 200 |
"peak": {
|
| 201 |
-
"total_rss_gb": 0.
|
| 202 |
"anon_rss_gb": 0.103,
|
| 203 |
-
"file_rss_gb": 0.
|
| 204 |
"peak_swap_gb": 0.0,
|
| 205 |
-
"hwm_rss_gb": 0.
|
| 206 |
},
|
| 207 |
"decode": {
|
| 208 |
-
"total_rss_gb": 0.
|
| 209 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 210 |
},
|
| 211 |
"rss_timeline_mib": [
|
| 212 |
[
|
| 213 |
0.0,
|
| 214 |
-
0.
|
| 215 |
],
|
| 216 |
[
|
| 217 |
-
0.
|
| 218 |
-
75.
|
| 219 |
],
|
| 220 |
[
|
| 221 |
-
0.
|
| 222 |
-
|
| 223 |
],
|
| 224 |
[
|
| 225 |
-
0.
|
| 226 |
-
|
| 227 |
],
|
| 228 |
[
|
| 229 |
-
0.
|
| 230 |
-
|
| 231 |
],
|
| 232 |
[
|
| 233 |
-
1.
|
| 234 |
-
|
| 235 |
],
|
| 236 |
[
|
| 237 |
-
1.
|
| 238 |
-
|
| 239 |
],
|
| 240 |
[
|
| 241 |
-
1.
|
| 242 |
-
|
| 243 |
],
|
| 244 |
[
|
| 245 |
-
1.
|
| 246 |
-
|
| 247 |
],
|
| 248 |
[
|
| 249 |
-
1.
|
| 250 |
-
|
| 251 |
],
|
| 252 |
[
|
| 253 |
-
2.
|
| 254 |
-
|
| 255 |
],
|
| 256 |
[
|
| 257 |
-
2.
|
| 258 |
-
|
| 259 |
],
|
| 260 |
[
|
| 261 |
-
2.
|
| 262 |
-
|
| 263 |
],
|
| 264 |
[
|
| 265 |
-
2.
|
| 266 |
-
|
| 267 |
],
|
| 268 |
[
|
| 269 |
-
3.
|
| 270 |
-
|
| 271 |
],
|
| 272 |
[
|
| 273 |
-
3.
|
| 274 |
-
714.
|
| 275 |
],
|
| 276 |
[
|
| 277 |
-
3.
|
| 278 |
-
|
| 279 |
],
|
| 280 |
[
|
| 281 |
-
3.
|
| 282 |
-
|
| 283 |
],
|
| 284 |
[
|
| 285 |
-
|
| 286 |
-
|
| 287 |
],
|
| 288 |
[
|
| 289 |
-
4.
|
| 290 |
-
|
| 291 |
],
|
| 292 |
[
|
| 293 |
-
4.
|
| 294 |
-
|
| 295 |
],
|
| 296 |
[
|
| 297 |
-
4.
|
| 298 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 299 |
],
|
| 300 |
[
|
| 301 |
-
5.
|
| 302 |
-
|
| 303 |
],
|
| 304 |
[
|
| 305 |
-
5.
|
| 306 |
-
|
| 307 |
],
|
| 308 |
[
|
| 309 |
-
5.
|
| 310 |
-
|
| 311 |
],
|
| 312 |
[
|
| 313 |
-
5.
|
| 314 |
-
|
| 315 |
],
|
| 316 |
[
|
| 317 |
-
5.
|
| 318 |
-
|
| 319 |
],
|
| 320 |
[
|
| 321 |
-
6.
|
| 322 |
-
|
| 323 |
],
|
| 324 |
[
|
| 325 |
-
6.
|
| 326 |
-
|
| 327 |
],
|
| 328 |
[
|
| 329 |
-
6.
|
| 330 |
-
|
| 331 |
],
|
| 332 |
[
|
| 333 |
-
6.
|
| 334 |
-
|
| 335 |
],
|
| 336 |
[
|
| 337 |
-
7.
|
| 338 |
-
|
| 339 |
],
|
| 340 |
[
|
| 341 |
-
7.
|
| 342 |
-
852.
|
| 343 |
]
|
| 344 |
],
|
| 345 |
"io_delta": {},
|
| 346 |
-
"timing_line": "[ Prompt: 7.9 t/s | Generation:
|
| 347 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 348 |
"pager": {
|
| 349 |
"routing_calls": 8448,
|
|
@@ -356,17 +553,17 @@
|
|
| 356 |
"resident_bytes": 327075840,
|
| 357 |
"budget_bytes": 327155712,
|
| 358 |
"hit_rate": 0.5514,
|
| 359 |
-
"proc_self_read_bytes":
|
| 360 |
"proc_self_rchar": 11947223
|
| 361 |
},
|
| 362 |
"output_matches_baseline": true,
|
| 363 |
"swap_zero": true,
|
| 364 |
"rc_ok": true,
|
| 365 |
-
"rss_vs_baseline": -1.
|
| 366 |
-
"decode_rss_vs_baseline": -1.
|
| 367 |
"decode_rss_lower_than_baseline": true,
|
| 368 |
"rss_lower_than_baseline": true,
|
| 369 |
-
"ssd_reads_mib":
|
| 370 |
"cold_weights_really_read_from_ssd": true,
|
| 371 |
"pager_hit_rate": 0.5514,
|
| 372 |
"pager_evictions": 4572,
|
|
@@ -381,16 +578,16 @@
|
|
| 381 |
"ST_RESERVE_MB": "200"
|
| 382 |
},
|
| 383 |
"rc": 0,
|
| 384 |
-
"seconds": 7.
|
| 385 |
"peak": {
|
| 386 |
-
"total_rss_gb": 0.
|
| 387 |
"anon_rss_gb": 0.1029,
|
| 388 |
-
"file_rss_gb": 0.
|
| 389 |
"peak_swap_gb": 0.0,
|
| 390 |
-
"hwm_rss_gb": 0.
|
| 391 |
},
|
| 392 |
"decode": {
|
| 393 |
-
"total_rss_gb": 0.
|
| 394 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 395 |
},
|
| 396 |
"rss_timeline_mib": [
|
|
@@ -399,148 +596,144 @@
|
|
| 399 |
0.4
|
| 400 |
],
|
| 401 |
[
|
| 402 |
-
0.
|
| 403 |
-
|
| 404 |
],
|
| 405 |
[
|
| 406 |
-
0.
|
| 407 |
-
71.
|
| 408 |
],
|
| 409 |
[
|
| 410 |
-
0.
|
| 411 |
-
|
| 412 |
],
|
| 413 |
[
|
| 414 |
-
0.
|
| 415 |
-
|
| 416 |
],
|
| 417 |
[
|
| 418 |
-
1.
|
| 419 |
-
|
| 420 |
-
],
|
| 421 |
-
[
|
| 422 |
-
1.25,
|
| 423 |
-
224.4
|
| 424 |
],
|
| 425 |
[
|
| 426 |
-
1.
|
| 427 |
-
|
| 428 |
],
|
| 429 |
[
|
| 430 |
-
1.
|
| 431 |
-
|
| 432 |
],
|
| 433 |
[
|
| 434 |
-
1.
|
| 435 |
-
|
| 436 |
],
|
| 437 |
[
|
| 438 |
-
|
| 439 |
-
|
| 440 |
],
|
| 441 |
[
|
| 442 |
-
2.
|
| 443 |
-
|
| 444 |
],
|
| 445 |
[
|
| 446 |
-
2.
|
| 447 |
-
|
| 448 |
],
|
| 449 |
[
|
| 450 |
-
2.
|
| 451 |
-
|
| 452 |
],
|
| 453 |
[
|
| 454 |
-
2.
|
| 455 |
-
|
| 456 |
],
|
| 457 |
[
|
| 458 |
-
3.
|
| 459 |
-
|
| 460 |
],
|
| 461 |
[
|
| 462 |
-
3.
|
| 463 |
-
|
| 464 |
],
|
| 465 |
[
|
| 466 |
-
3.
|
| 467 |
-
|
| 468 |
],
|
| 469 |
[
|
| 470 |
-
3.
|
| 471 |
-
|
| 472 |
],
|
| 473 |
[
|
| 474 |
-
|
| 475 |
-
|
| 476 |
],
|
| 477 |
[
|
| 478 |
-
4.
|
| 479 |
-
|
| 480 |
],
|
| 481 |
[
|
| 482 |
-
4.
|
| 483 |
-
|
| 484 |
],
|
| 485 |
[
|
| 486 |
-
4.
|
| 487 |
-
|
| 488 |
],
|
| 489 |
[
|
| 490 |
-
4.
|
| 491 |
-
|
| 492 |
],
|
| 493 |
[
|
| 494 |
-
5.
|
| 495 |
-
|
| 496 |
],
|
| 497 |
[
|
| 498 |
-
5.
|
| 499 |
-
|
| 500 |
],
|
| 501 |
[
|
| 502 |
-
5.
|
| 503 |
-
|
| 504 |
],
|
| 505 |
[
|
| 506 |
-
5.
|
| 507 |
-
|
| 508 |
],
|
| 509 |
[
|
| 510 |
-
6.
|
| 511 |
-
|
| 512 |
],
|
| 513 |
[
|
| 514 |
-
6.
|
| 515 |
-
|
| 516 |
],
|
| 517 |
[
|
| 518 |
-
6.
|
| 519 |
-
|
| 520 |
],
|
| 521 |
[
|
| 522 |
-
6.
|
| 523 |
-
|
| 524 |
],
|
| 525 |
[
|
| 526 |
-
|
| 527 |
-
|
| 528 |
],
|
| 529 |
[
|
| 530 |
-
7.
|
| 531 |
-
|
| 532 |
],
|
| 533 |
[
|
| 534 |
-
7.
|
| 535 |
-
|
| 536 |
],
|
| 537 |
[
|
| 538 |
-
7.
|
| 539 |
-
|
| 540 |
]
|
| 541 |
],
|
| 542 |
"io_delta": {},
|
| 543 |
-
"timing_line": "[ Prompt: 8.1 t/s | Generation:
|
| 544 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 545 |
"pager": {
|
| 546 |
"routing_calls": 8448,
|
|
@@ -553,17 +746,17 @@
|
|
| 553 |
"resident_bytes": 57701376,
|
| 554 |
"budget_bytes": 58720256,
|
| 555 |
"hit_rate": 0.537,
|
| 556 |
-
"proc_self_read_bytes":
|
| 557 |
"proc_self_rchar": 11947223
|
| 558 |
},
|
| 559 |
"output_matches_baseline": true,
|
| 560 |
"swap_zero": true,
|
| 561 |
"rc_ok": true,
|
| 562 |
-
"rss_vs_baseline": -1.
|
| 563 |
-
"decode_rss_vs_baseline": -1.
|
| 564 |
"decode_rss_lower_than_baseline": true,
|
| 565 |
"rss_lower_than_baseline": true,
|
| 566 |
-
"ssd_reads_mib":
|
| 567 |
"cold_weights_really_read_from_ssd": true,
|
| 568 |
"pager_hit_rate": 0.537,
|
| 569 |
"pager_evictions": 5625,
|
|
@@ -578,16 +771,16 @@
|
|
| 578 |
"ST_RESERVE_MB": "200"
|
| 579 |
},
|
| 580 |
"rc": 0,
|
| 581 |
-
"seconds":
|
| 582 |
"peak": {
|
| 583 |
-
"total_rss_gb": 0.
|
| 584 |
-
"anon_rss_gb": 0.
|
| 585 |
-
"file_rss_gb": 0.
|
| 586 |
"peak_swap_gb": 0.0,
|
| 587 |
-
"hwm_rss_gb": 0.
|
| 588 |
},
|
| 589 |
"decode": {
|
| 590 |
-
"total_rss_gb": 0.
|
| 591 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 592 |
},
|
| 593 |
"rss_timeline_mib": [
|
|
@@ -596,136 +789,144 @@
|
|
| 596 |
0.4
|
| 597 |
],
|
| 598 |
[
|
| 599 |
-
0.
|
| 600 |
-
|
| 601 |
],
|
| 602 |
[
|
| 603 |
-
0.
|
| 604 |
-
71.
|
| 605 |
],
|
| 606 |
[
|
| 607 |
-
0.
|
| 608 |
-
|
| 609 |
],
|
| 610 |
[
|
| 611 |
-
0.
|
| 612 |
-
|
| 613 |
],
|
| 614 |
[
|
| 615 |
-
1.
|
| 616 |
-
|
| 617 |
],
|
| 618 |
[
|
| 619 |
-
1.
|
| 620 |
-
|
| 621 |
],
|
| 622 |
[
|
| 623 |
-
1.
|
| 624 |
-
|
| 625 |
],
|
| 626 |
[
|
| 627 |
-
1.
|
| 628 |
-
|
| 629 |
],
|
| 630 |
[
|
| 631 |
-
1.
|
| 632 |
-
|
| 633 |
],
|
| 634 |
[
|
| 635 |
-
2.
|
| 636 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 637 |
],
|
| 638 |
[
|
| 639 |
-
2.
|
| 640 |
-
|
| 641 |
],
|
| 642 |
[
|
| 643 |
-
|
| 644 |
-
|
| 645 |
],
|
| 646 |
[
|
| 647 |
-
|
| 648 |
-
|
| 649 |
],
|
| 650 |
[
|
| 651 |
-
3.
|
| 652 |
-
|
| 653 |
],
|
| 654 |
[
|
| 655 |
-
3.
|
| 656 |
-
|
| 657 |
],
|
| 658 |
[
|
| 659 |
-
|
| 660 |
-
|
| 661 |
],
|
| 662 |
[
|
| 663 |
-
|
| 664 |
-
|
| 665 |
],
|
| 666 |
[
|
| 667 |
-
4.
|
| 668 |
-
|
| 669 |
],
|
| 670 |
[
|
| 671 |
-
4.
|
| 672 |
-
|
| 673 |
],
|
| 674 |
[
|
| 675 |
-
|
| 676 |
-
|
| 677 |
],
|
| 678 |
[
|
| 679 |
-
|
| 680 |
-
|
| 681 |
],
|
| 682 |
[
|
| 683 |
-
|
| 684 |
-
|
| 685 |
],
|
| 686 |
[
|
| 687 |
-
5.
|
| 688 |
-
|
| 689 |
],
|
| 690 |
[
|
| 691 |
-
|
| 692 |
-
|
| 693 |
],
|
| 694 |
[
|
| 695 |
-
|
| 696 |
-
|
| 697 |
],
|
| 698 |
[
|
| 699 |
-
|
| 700 |
-
|
| 701 |
],
|
| 702 |
[
|
| 703 |
-
6.
|
| 704 |
-
|
| 705 |
],
|
| 706 |
[
|
| 707 |
-
6.
|
| 708 |
-
|
| 709 |
],
|
| 710 |
[
|
| 711 |
-
|
| 712 |
-
|
| 713 |
],
|
| 714 |
[
|
| 715 |
-
|
| 716 |
-
|
| 717 |
],
|
| 718 |
[
|
| 719 |
-
7.
|
| 720 |
-
|
| 721 |
],
|
| 722 |
[
|
| 723 |
-
7.
|
| 724 |
-
675.
|
| 725 |
]
|
| 726 |
],
|
| 727 |
"io_delta": {},
|
| 728 |
-
"timing_line": "[ Prompt:
|
| 729 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 730 |
"pager": {
|
| 731 |
"routing_calls": 8448,
|
|
@@ -738,14 +939,14 @@
|
|
| 738 |
"resident_bytes": 132240384,
|
| 739 |
"budget_bytes": 134217728,
|
| 740 |
"hit_rate": 0.5415,
|
| 741 |
-
"proc_self_read_bytes":
|
| 742 |
"proc_self_rchar": 11947223
|
| 743 |
},
|
| 744 |
"output_matches_baseline": true,
|
| 745 |
"swap_zero": true,
|
| 746 |
"rc_ok": true,
|
| 747 |
-
"rss_vs_baseline": -1.
|
| 748 |
-
"decode_rss_vs_baseline": -1.
|
| 749 |
"decode_rss_lower_than_baseline": true,
|
| 750 |
"rss_lower_than_baseline": true,
|
| 751 |
"ssd_reads_mib": 2317.3,
|
|
|
|
| 1 |
{
|
| 2 |
+
"when": "2026-10-07T16:47:19+0200",
|
| 3 |
"model": "/root/work/models/SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf",
|
| 4 |
"model_size_mib": 2508,
|
| 5 |
"prompt": "Hello, who are you?",
|
|
|
|
| 12 |
"ST_PAGER": "0"
|
| 13 |
},
|
| 14 |
"rc": 0,
|
| 15 |
+
"seconds": 9.06,
|
| 16 |
"peak": {
|
| 17 |
+
"total_rss_gb": 2.5648,
|
| 18 |
"anon_rss_gb": 0.1033,
|
| 19 |
+
"file_rss_gb": 2.559,
|
| 20 |
"peak_swap_gb": 0.0,
|
| 21 |
+
"hwm_rss_gb": 2.5648
|
| 22 |
},
|
| 23 |
"decode": {
|
| 24 |
+
"total_rss_gb": 2.5648,
|
| 25 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 26 |
},
|
| 27 |
"rss_timeline_mib": [
|
| 28 |
[
|
| 29 |
+
0.0,
|
| 30 |
+
0.8
|
| 31 |
],
|
| 32 |
[
|
| 33 |
+
0.205,
|
| 34 |
+
76.1
|
| 35 |
],
|
| 36 |
[
|
| 37 |
+
0.41,
|
| 38 |
+
74.4
|
| 39 |
],
|
| 40 |
[
|
| 41 |
+
0.619,
|
| 42 |
+
124.9
|
| 43 |
],
|
| 44 |
[
|
| 45 |
0.825,
|
| 46 |
+
195.9
|
| 47 |
+
],
|
| 48 |
+
[
|
| 49 |
+
1.032,
|
| 50 |
+
263.6
|
| 51 |
+
],
|
| 52 |
+
[
|
| 53 |
+
1.239,
|
| 54 |
+
333.7
|
| 55 |
],
|
| 56 |
[
|
| 57 |
+
1.454,
|
| 58 |
+
402.4
|
| 59 |
],
|
| 60 |
[
|
| 61 |
+
1.664,
|
| 62 |
+
473.7
|
| 63 |
],
|
| 64 |
[
|
| 65 |
+
1.873,
|
| 66 |
+
544.7
|
| 67 |
],
|
| 68 |
[
|
| 69 |
+
2.088,
|
| 70 |
+
616.2
|
| 71 |
],
|
| 72 |
[
|
| 73 |
+
2.299,
|
| 74 |
+
688.0
|
| 75 |
],
|
| 76 |
[
|
| 77 |
+
2.511,
|
| 78 |
+
759.8
|
| 79 |
],
|
| 80 |
[
|
| 81 |
+
2.724,
|
| 82 |
+
830.6
|
| 83 |
],
|
| 84 |
[
|
| 85 |
+
2.938,
|
| 86 |
+
902.1
|
| 87 |
],
|
| 88 |
[
|
| 89 |
+
3.152,
|
| 90 |
+
974.8
|
| 91 |
],
|
| 92 |
[
|
| 93 |
+
3.366,
|
| 94 |
+
1045.4
|
| 95 |
],
|
| 96 |
[
|
| 97 |
+
3.582,
|
| 98 |
+
1116.1
|
| 99 |
],
|
| 100 |
[
|
| 101 |
+
3.803,
|
| 102 |
+
1190.9
|
| 103 |
],
|
| 104 |
[
|
| 105 |
+
4.026,
|
| 106 |
+
1265.4
|
| 107 |
],
|
| 108 |
[
|
| 109 |
+
4.245,
|
| 110 |
+
1338.0
|
| 111 |
],
|
| 112 |
[
|
| 113 |
+
4.465,
|
| 114 |
+
1411.0
|
| 115 |
],
|
| 116 |
[
|
| 117 |
+
4.686,
|
| 118 |
+
1485.0
|
| 119 |
],
|
| 120 |
[
|
| 121 |
+
4.909,
|
| 122 |
+
1560.5
|
| 123 |
],
|
| 124 |
[
|
| 125 |
+
5.135,
|
| 126 |
+
1634.3
|
| 127 |
],
|
| 128 |
[
|
| 129 |
+
5.358,
|
| 130 |
+
1709.6
|
| 131 |
],
|
| 132 |
[
|
| 133 |
+
5.581,
|
| 134 |
+
1784.6
|
| 135 |
],
|
| 136 |
[
|
| 137 |
+
5.811,
|
| 138 |
+
1859.6
|
| 139 |
],
|
| 140 |
[
|
| 141 |
+
6.047,
|
| 142 |
+
1937.9
|
| 143 |
],
|
| 144 |
[
|
| 145 |
+
6.277,
|
| 146 |
+
2014.4
|
| 147 |
],
|
| 148 |
[
|
| 149 |
+
6.506,
|
| 150 |
+
2090.7
|
| 151 |
],
|
| 152 |
[
|
| 153 |
+
6.734,
|
| 154 |
+
2166.2
|
| 155 |
],
|
| 156 |
[
|
| 157 |
+
6.964,
|
| 158 |
+
2243.7
|
| 159 |
],
|
| 160 |
[
|
| 161 |
+
7.196,
|
| 162 |
+
2320.7
|
| 163 |
],
|
| 164 |
[
|
| 165 |
+
7.433,
|
| 166 |
+
2399.8
|
| 167 |
],
|
| 168 |
[
|
| 169 |
+
7.67,
|
| 170 |
+
2477.3
|
| 171 |
],
|
| 172 |
[
|
| 173 |
+
7.907,
|
| 174 |
+
2556.6
|
| 175 |
],
|
| 176 |
[
|
| 177 |
+
8.148,
|
| 178 |
+
2620.6
|
| 179 |
],
|
| 180 |
[
|
| 181 |
+
8.385,
|
| 182 |
+
2626.3
|
| 183 |
],
|
| 184 |
[
|
| 185 |
+
8.634,
|
| 186 |
+
2626.3
|
| 187 |
],
|
| 188 |
[
|
| 189 |
+
8.901,
|
| 190 |
+
2626.3
|
| 191 |
]
|
| 192 |
],
|
| 193 |
"io_delta": {},
|
| 194 |
+
"timing_line": "[ Prompt: 178.7 t/s | Generation: 55.4 t/s ]",
|
| 195 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 196 |
"output_matches_baseline": true,
|
| 197 |
"swap_zero": true,
|
| 198 |
"rc_ok": true
|
| 199 |
},
|
| 200 |
+
"budget1024mb": {
|
| 201 |
+
"env": {
|
| 202 |
+
"ST_PAGER": "1",
|
| 203 |
+
"ST_RAM_BUDGET_MB": "1024",
|
| 204 |
+
"ST_RESERVE_MB": "200"
|
| 205 |
+
},
|
| 206 |
+
"rc": 0,
|
| 207 |
+
"seconds": 7.85,
|
| 208 |
+
"peak": {
|
| 209 |
+
"total_rss_gb": 1.3009,
|
| 210 |
+
"anon_rss_gb": 0.103,
|
| 211 |
+
"file_rss_gb": 1.2995,
|
| 212 |
+
"peak_swap_gb": 0.0,
|
| 213 |
+
"hwm_rss_gb": 1.3009
|
| 214 |
+
},
|
| 215 |
+
"decode": {
|
| 216 |
+
"total_rss_gb": 1.3009,
|
| 217 |
+
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 218 |
+
},
|
| 219 |
+
"rss_timeline_mib": [
|
| 220 |
+
[
|
| 221 |
+
0.0,
|
| 222 |
+
0.5
|
| 223 |
+
],
|
| 224 |
+
[
|
| 225 |
+
0.205,
|
| 226 |
+
72.8
|
| 227 |
+
],
|
| 228 |
+
[
|
| 229 |
+
0.409,
|
| 230 |
+
71.1
|
| 231 |
+
],
|
| 232 |
+
[
|
| 233 |
+
0.613,
|
| 234 |
+
115.4
|
| 235 |
+
],
|
| 236 |
+
[
|
| 237 |
+
0.822,
|
| 238 |
+
172.7
|
| 239 |
+
],
|
| 240 |
+
[
|
| 241 |
+
1.043,
|
| 242 |
+
229.1
|
| 243 |
+
],
|
| 244 |
+
[
|
| 245 |
+
1.257,
|
| 246 |
+
289.1
|
| 247 |
+
],
|
| 248 |
+
[
|
| 249 |
+
1.471,
|
| 250 |
+
348.1
|
| 251 |
+
],
|
| 252 |
+
[
|
| 253 |
+
1.685,
|
| 254 |
+
405.3
|
| 255 |
+
],
|
| 256 |
+
[
|
| 257 |
+
1.902,
|
| 258 |
+
465.1
|
| 259 |
+
],
|
| 260 |
+
[
|
| 261 |
+
2.122,
|
| 262 |
+
557.5
|
| 263 |
+
],
|
| 264 |
+
[
|
| 265 |
+
2.345,
|
| 266 |
+
688.3
|
| 267 |
+
],
|
| 268 |
+
[
|
| 269 |
+
2.576,
|
| 270 |
+
742.5
|
| 271 |
+
],
|
| 272 |
+
[
|
| 273 |
+
2.811,
|
| 274 |
+
825.9
|
| 275 |
+
],
|
| 276 |
+
[
|
| 277 |
+
3.057,
|
| 278 |
+
912.6
|
| 279 |
+
],
|
| 280 |
+
[
|
| 281 |
+
3.3,
|
| 282 |
+
996.3
|
| 283 |
+
],
|
| 284 |
+
[
|
| 285 |
+
3.547,
|
| 286 |
+
1089.1
|
| 287 |
+
],
|
| 288 |
+
[
|
| 289 |
+
3.797,
|
| 290 |
+
1180.5
|
| 291 |
+
],
|
| 292 |
+
[
|
| 293 |
+
4.064,
|
| 294 |
+
1194.8
|
| 295 |
+
],
|
| 296 |
+
[
|
| 297 |
+
4.315,
|
| 298 |
+
1209.3
|
| 299 |
+
],
|
| 300 |
+
[
|
| 301 |
+
4.574,
|
| 302 |
+
1236.3
|
| 303 |
+
],
|
| 304 |
+
[
|
| 305 |
+
4.83,
|
| 306 |
+
1246.3
|
| 307 |
+
],
|
| 308 |
+
[
|
| 309 |
+
5.082,
|
| 310 |
+
1253.1
|
| 311 |
+
],
|
| 312 |
+
[
|
| 313 |
+
5.338,
|
| 314 |
+
1213.9
|
| 315 |
+
],
|
| 316 |
+
[
|
| 317 |
+
5.628,
|
| 318 |
+
1224.6
|
| 319 |
+
],
|
| 320 |
+
[
|
| 321 |
+
5.887,
|
| 322 |
+
1226.6
|
| 323 |
+
],
|
| 324 |
+
[
|
| 325 |
+
6.15,
|
| 326 |
+
1220.7
|
| 327 |
+
],
|
| 328 |
+
[
|
| 329 |
+
6.413,
|
| 330 |
+
1252.2
|
| 331 |
+
],
|
| 332 |
+
[
|
| 333 |
+
6.678,
|
| 334 |
+
1297.8
|
| 335 |
+
],
|
| 336 |
+
[
|
| 337 |
+
6.942,
|
| 338 |
+
1322.2
|
| 339 |
+
],
|
| 340 |
+
[
|
| 341 |
+
7.205,
|
| 342 |
+
1320.0
|
| 343 |
+
],
|
| 344 |
+
[
|
| 345 |
+
7.474,
|
| 346 |
+
1320.2
|
| 347 |
+
],
|
| 348 |
+
[
|
| 349 |
+
7.738,
|
| 350 |
+
1324.9
|
| 351 |
+
]
|
| 352 |
+
],
|
| 353 |
+
"io_delta": {},
|
| 354 |
+
"timing_line": "[ Prompt: 7.8 t/s | Generation: 22.9 t/s ]",
|
| 355 |
+
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 356 |
+
"pager": {
|
| 357 |
+
"routing_calls": 8448,
|
| 358 |
+
"expert_touches": 64544,
|
| 359 |
+
"hits": 37015,
|
| 360 |
+
"misses": 2273,
|
| 361 |
+
"evictions": 2894,
|
| 362 |
+
"evicted_bytes": 6197382144,
|
| 363 |
+
"sweeps": 864,
|
| 364 |
+
"resident_bytes": 863972352,
|
| 365 |
+
"budget_bytes": 864026624,
|
| 366 |
+
"hit_rate": 0.5735,
|
| 367 |
+
"proc_self_read_bytes": 2429124608,
|
| 368 |
+
"proc_self_rchar": 11947223
|
| 369 |
+
},
|
| 370 |
+
"output_matches_baseline": true,
|
| 371 |
+
"swap_zero": true,
|
| 372 |
+
"rc_ok": true,
|
| 373 |
+
"rss_vs_baseline": -1.2639,
|
| 374 |
+
"decode_rss_vs_baseline": -1.2639,
|
| 375 |
+
"decode_rss_lower_than_baseline": true,
|
| 376 |
+
"rss_lower_than_baseline": true,
|
| 377 |
+
"ssd_reads_mib": 2316.6,
|
| 378 |
+
"cold_weights_really_read_from_ssd": true,
|
| 379 |
+
"pager_hit_rate": 0.5735,
|
| 380 |
+
"pager_evictions": 2894,
|
| 381 |
+
"pager_misses": 2273,
|
| 382 |
+
"pager_resident_mib": 823.9,
|
| 383 |
+
"pager_budget_mib": 824.0
|
| 384 |
+
},
|
| 385 |
"budget512mb": {
|
| 386 |
"env": {
|
| 387 |
"ST_PAGER": "1",
|
|
|
|
| 389 |
"ST_RESERVE_MB": "200"
|
| 390 |
},
|
| 391 |
"rc": 0,
|
| 392 |
+
"seconds": 7.67,
|
| 393 |
"peak": {
|
| 394 |
+
"total_rss_gb": 0.8327,
|
| 395 |
"anon_rss_gb": 0.103,
|
| 396 |
+
"file_rss_gb": 0.8363,
|
| 397 |
"peak_swap_gb": 0.0,
|
| 398 |
+
"hwm_rss_gb": 0.8327
|
| 399 |
},
|
| 400 |
"decode": {
|
| 401 |
+
"total_rss_gb": 0.8327,
|
| 402 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 403 |
},
|
| 404 |
"rss_timeline_mib": [
|
| 405 |
[
|
| 406 |
0.0,
|
| 407 |
+
0.9
|
| 408 |
],
|
| 409 |
[
|
| 410 |
+
0.206,
|
| 411 |
+
75.8
|
| 412 |
],
|
| 413 |
[
|
| 414 |
+
0.413,
|
| 415 |
+
72.4
|
| 416 |
],
|
| 417 |
[
|
| 418 |
+
0.62,
|
| 419 |
+
136.1
|
| 420 |
],
|
| 421 |
[
|
| 422 |
+
0.828,
|
| 423 |
+
193.1
|
| 424 |
],
|
| 425 |
[
|
| 426 |
+
1.041,
|
| 427 |
+
253.0
|
| 428 |
],
|
| 429 |
[
|
| 430 |
+
1.257,
|
| 431 |
+
312.0
|
| 432 |
],
|
| 433 |
[
|
| 434 |
+
1.472,
|
| 435 |
+
375.3
|
| 436 |
],
|
| 437 |
[
|
| 438 |
+
1.687,
|
| 439 |
+
436.7
|
| 440 |
],
|
| 441 |
[
|
| 442 |
+
1.904,
|
| 443 |
+
507.1
|
| 444 |
],
|
| 445 |
[
|
| 446 |
+
2.127,
|
| 447 |
+
638.1
|
| 448 |
],
|
| 449 |
[
|
| 450 |
+
2.352,
|
| 451 |
+
726.1
|
| 452 |
],
|
| 453 |
[
|
| 454 |
+
2.576,
|
| 455 |
+
699.4
|
| 456 |
],
|
| 457 |
[
|
| 458 |
+
2.803,
|
| 459 |
+
739.9
|
| 460 |
],
|
| 461 |
[
|
| 462 |
+
3.041,
|
| 463 |
+
727.5
|
| 464 |
],
|
| 465 |
[
|
| 466 |
+
3.263,
|
| 467 |
+
714.7
|
| 468 |
],
|
| 469 |
[
|
| 470 |
+
3.487,
|
| 471 |
+
703.7
|
| 472 |
],
|
| 473 |
[
|
| 474 |
+
3.71,
|
| 475 |
+
744.8
|
| 476 |
],
|
| 477 |
[
|
| 478 |
+
3.941,
|
| 479 |
+
726.0
|
| 480 |
],
|
| 481 |
[
|
| 482 |
+
4.161,
|
| 483 |
+
725.3
|
| 484 |
],
|
| 485 |
[
|
| 486 |
+
4.383,
|
| 487 |
+
733.5
|
| 488 |
],
|
| 489 |
[
|
| 490 |
+
4.606,
|
| 491 |
+
733.0
|
| 492 |
+
],
|
| 493 |
+
[
|
| 494 |
+
4.827,
|
| 495 |
+
736.0
|
| 496 |
],
|
| 497 |
[
|
| 498 |
+
5.048,
|
| 499 |
+
725.0
|
| 500 |
],
|
| 501 |
[
|
| 502 |
+
5.271,
|
| 503 |
+
719.1
|
| 504 |
],
|
| 505 |
[
|
| 506 |
+
5.494,
|
| 507 |
+
717.1
|
| 508 |
],
|
| 509 |
[
|
| 510 |
+
5.722,
|
| 511 |
+
694.0
|
| 512 |
],
|
| 513 |
[
|
| 514 |
+
5.945,
|
| 515 |
+
736.3
|
| 516 |
],
|
| 517 |
[
|
| 518 |
+
6.18,
|
| 519 |
+
761.5
|
| 520 |
],
|
| 521 |
[
|
| 522 |
+
6.422,
|
| 523 |
+
815.8
|
| 524 |
],
|
| 525 |
[
|
| 526 |
+
6.663,
|
| 527 |
+
836.8
|
| 528 |
],
|
| 529 |
[
|
| 530 |
+
6.93,
|
| 531 |
+
840.2
|
| 532 |
],
|
| 533 |
[
|
| 534 |
+
7.184,
|
| 535 |
+
844.3
|
| 536 |
],
|
| 537 |
[
|
| 538 |
+
7.443,
|
| 539 |
+
852.4
|
| 540 |
]
|
| 541 |
],
|
| 542 |
"io_delta": {},
|
| 543 |
+
"timing_line": "[ Prompt: 7.9 t/s | Generation: 21.6 t/s ]",
|
| 544 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 545 |
"pager": {
|
| 546 |
"routing_calls": 8448,
|
|
|
|
| 553 |
"resident_bytes": 327075840,
|
| 554 |
"budget_bytes": 327155712,
|
| 555 |
"hit_rate": 0.5514,
|
| 556 |
+
"proc_self_read_bytes": 2428149760,
|
| 557 |
"proc_self_rchar": 11947223
|
| 558 |
},
|
| 559 |
"output_matches_baseline": true,
|
| 560 |
"swap_zero": true,
|
| 561 |
"rc_ok": true,
|
| 562 |
+
"rss_vs_baseline": -1.7321,
|
| 563 |
+
"decode_rss_vs_baseline": -1.7321,
|
| 564 |
"decode_rss_lower_than_baseline": true,
|
| 565 |
"rss_lower_than_baseline": true,
|
| 566 |
+
"ssd_reads_mib": 2315.7,
|
| 567 |
"cold_weights_really_read_from_ssd": true,
|
| 568 |
"pager_hit_rate": 0.5514,
|
| 569 |
"pager_evictions": 4572,
|
|
|
|
| 578 |
"ST_RESERVE_MB": "200"
|
| 579 |
},
|
| 580 |
"rc": 0,
|
| 581 |
+
"seconds": 7.88,
|
| 582 |
"peak": {
|
| 583 |
+
"total_rss_gb": 0.5956,
|
| 584 |
"anon_rss_gb": 0.1029,
|
| 585 |
+
"file_rss_gb": 0.5922,
|
| 586 |
"peak_swap_gb": 0.0,
|
| 587 |
+
"hwm_rss_gb": 0.5956
|
| 588 |
},
|
| 589 |
"decode": {
|
| 590 |
+
"total_rss_gb": 0.5956,
|
| 591 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 592 |
},
|
| 593 |
"rss_timeline_mib": [
|
|
|
|
| 596 |
0.4
|
| 597 |
],
|
| 598 |
[
|
| 599 |
+
0.207,
|
| 600 |
+
72.9
|
| 601 |
],
|
| 602 |
[
|
| 603 |
+
0.413,
|
| 604 |
+
71.1
|
| 605 |
],
|
| 606 |
[
|
| 607 |
+
0.624,
|
| 608 |
+
128.7
|
| 609 |
],
|
| 610 |
[
|
| 611 |
+
0.835,
|
| 612 |
+
189.3
|
| 613 |
],
|
| 614 |
[
|
| 615 |
+
1.047,
|
| 616 |
+
212.1
|
|
|
|
|
|
|
|
|
|
|
|
|
| 617 |
],
|
| 618 |
[
|
| 619 |
+
1.259,
|
| 620 |
+
228.5
|
| 621 |
],
|
| 622 |
[
|
| 623 |
+
1.474,
|
| 624 |
+
246.9
|
| 625 |
],
|
| 626 |
[
|
| 627 |
+
1.69,
|
| 628 |
+
267.0
|
| 629 |
],
|
| 630 |
[
|
| 631 |
+
1.907,
|
| 632 |
+
291.7
|
| 633 |
],
|
| 634 |
[
|
| 635 |
+
2.128,
|
| 636 |
+
388.4
|
| 637 |
],
|
| 638 |
[
|
| 639 |
+
2.352,
|
| 640 |
+
482.7
|
| 641 |
],
|
| 642 |
[
|
| 643 |
+
2.577,
|
| 644 |
+
490.7
|
| 645 |
],
|
| 646 |
[
|
| 647 |
+
2.802,
|
| 648 |
+
467.5
|
| 649 |
],
|
| 650 |
[
|
| 651 |
+
3.024,
|
| 652 |
+
448.6
|
| 653 |
],
|
| 654 |
[
|
| 655 |
+
3.25,
|
| 656 |
+
437.3
|
| 657 |
],
|
| 658 |
[
|
| 659 |
+
3.489,
|
| 660 |
+
482.6
|
| 661 |
],
|
| 662 |
[
|
| 663 |
+
3.719,
|
| 664 |
+
491.9
|
| 665 |
],
|
| 666 |
[
|
| 667 |
+
3.952,
|
| 668 |
+
489.6
|
| 669 |
],
|
| 670 |
[
|
| 671 |
+
4.184,
|
| 672 |
+
481.9
|
| 673 |
],
|
| 674 |
[
|
| 675 |
+
4.417,
|
| 676 |
+
451.6
|
| 677 |
],
|
| 678 |
[
|
| 679 |
+
4.652,
|
| 680 |
+
458.8
|
| 681 |
],
|
| 682 |
[
|
| 683 |
+
4.887,
|
| 684 |
+
455.6
|
| 685 |
],
|
| 686 |
[
|
| 687 |
+
5.122,
|
| 688 |
+
453.0
|
| 689 |
],
|
| 690 |
[
|
| 691 |
+
5.36,
|
| 692 |
+
450.3
|
| 693 |
],
|
| 694 |
[
|
| 695 |
+
5.6,
|
| 696 |
+
459.5
|
| 697 |
],
|
| 698 |
[
|
| 699 |
+
5.834,
|
| 700 |
+
447.4
|
| 701 |
],
|
| 702 |
[
|
| 703 |
+
6.068,
|
| 704 |
+
496.0
|
| 705 |
],
|
| 706 |
[
|
| 707 |
+
6.301,
|
| 708 |
+
517.2
|
| 709 |
],
|
| 710 |
[
|
| 711 |
+
6.543,
|
| 712 |
+
560.1
|
| 713 |
],
|
| 714 |
[
|
| 715 |
+
6.785,
|
| 716 |
+
580.2
|
| 717 |
],
|
| 718 |
[
|
| 719 |
+
7.024,
|
| 720 |
+
593.8
|
| 721 |
],
|
| 722 |
[
|
| 723 |
+
7.263,
|
| 724 |
+
593.4
|
| 725 |
],
|
| 726 |
[
|
| 727 |
+
7.504,
|
| 728 |
+
603.1
|
| 729 |
],
|
| 730 |
[
|
| 731 |
+
7.743,
|
| 732 |
+
606.7
|
| 733 |
]
|
| 734 |
],
|
| 735 |
"io_delta": {},
|
| 736 |
+
"timing_line": "[ Prompt: 8.1 t/s | Generation: 19.7 t/s ]",
|
| 737 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 738 |
"pager": {
|
| 739 |
"routing_calls": 8448,
|
|
|
|
| 746 |
"resident_bytes": 57701376,
|
| 747 |
"budget_bytes": 58720256,
|
| 748 |
"hit_rate": 0.537,
|
| 749 |
+
"proc_self_read_bytes": 2433163264,
|
| 750 |
"proc_self_rchar": 11947223
|
| 751 |
},
|
| 752 |
"output_matches_baseline": true,
|
| 753 |
"swap_zero": true,
|
| 754 |
"rc_ok": true,
|
| 755 |
+
"rss_vs_baseline": -1.9692,
|
| 756 |
+
"decode_rss_vs_baseline": -1.9692,
|
| 757 |
"decode_rss_lower_than_baseline": true,
|
| 758 |
"rss_lower_than_baseline": true,
|
| 759 |
+
"ssd_reads_mib": 2320.4,
|
| 760 |
"cold_weights_really_read_from_ssd": true,
|
| 761 |
"pager_hit_rate": 0.537,
|
| 762 |
"pager_evictions": 5625,
|
|
|
|
| 771 |
"ST_RESERVE_MB": "200"
|
| 772 |
},
|
| 773 |
"rc": 0,
|
| 774 |
+
"seconds": 8.13,
|
| 775 |
"peak": {
|
| 776 |
+
"total_rss_gb": 0.6614,
|
| 777 |
+
"anon_rss_gb": 0.103,
|
| 778 |
+
"file_rss_gb": 0.657,
|
| 779 |
"peak_swap_gb": 0.0,
|
| 780 |
+
"hwm_rss_gb": 0.6614
|
| 781 |
},
|
| 782 |
"decode": {
|
| 783 |
+
"total_rss_gb": 0.6614,
|
| 784 |
"note": "prefill 後的峰值;prefill 會摸遍全部 expert,所以 peak 不代表 decode"
|
| 785 |
},
|
| 786 |
"rss_timeline_mib": [
|
|
|
|
| 789 |
0.4
|
| 790 |
],
|
| 791 |
[
|
| 792 |
+
0.207,
|
| 793 |
+
65.7
|
| 794 |
],
|
| 795 |
[
|
| 796 |
+
0.415,
|
| 797 |
+
71.2
|
| 798 |
],
|
| 799 |
[
|
| 800 |
+
0.624,
|
| 801 |
+
76.7
|
| 802 |
],
|
| 803 |
[
|
| 804 |
+
0.834,
|
| 805 |
+
137.4
|
| 806 |
],
|
| 807 |
[
|
| 808 |
+
1.048,
|
| 809 |
+
194.3
|
| 810 |
],
|
| 811 |
[
|
| 812 |
+
1.264,
|
| 813 |
+
251.2
|
| 814 |
],
|
| 815 |
[
|
| 816 |
+
1.482,
|
| 817 |
+
300.9
|
| 818 |
],
|
| 819 |
[
|
| 820 |
+
1.703,
|
| 821 |
+
322.4
|
| 822 |
],
|
| 823 |
[
|
| 824 |
+
1.928,
|
| 825 |
+
340.0
|
| 826 |
],
|
| 827 |
[
|
| 828 |
+
2.151,
|
| 829 |
+
363.3
|
| 830 |
+
],
|
| 831 |
+
[
|
| 832 |
+
2.383,
|
| 833 |
+
508.0
|
| 834 |
+
],
|
| 835 |
+
[
|
| 836 |
+
2.619,
|
| 837 |
+
553.7
|
| 838 |
],
|
| 839 |
[
|
| 840 |
+
2.857,
|
| 841 |
+
519.9
|
| 842 |
],
|
| 843 |
[
|
| 844 |
+
3.112,
|
| 845 |
+
507.3
|
| 846 |
],
|
| 847 |
[
|
| 848 |
+
3.361,
|
| 849 |
+
553.1
|
| 850 |
],
|
| 851 |
[
|
| 852 |
+
3.608,
|
| 853 |
+
541.7
|
| 854 |
],
|
| 855 |
[
|
| 856 |
+
3.856,
|
| 857 |
+
547.3
|
| 858 |
],
|
| 859 |
[
|
| 860 |
+
4.107,
|
| 861 |
+
542.7
|
| 862 |
],
|
| 863 |
[
|
| 864 |
+
4.358,
|
| 865 |
+
544.0
|
| 866 |
],
|
| 867 |
[
|
| 868 |
+
4.599,
|
| 869 |
+
548.5
|
| 870 |
],
|
| 871 |
[
|
| 872 |
+
4.839,
|
| 873 |
+
562.8
|
| 874 |
],
|
| 875 |
[
|
| 876 |
+
5.082,
|
| 877 |
+
518.0
|
| 878 |
],
|
| 879 |
[
|
| 880 |
+
5.32,
|
| 881 |
+
519.3
|
| 882 |
],
|
| 883 |
[
|
| 884 |
+
5.556,
|
| 885 |
+
529.6
|
| 886 |
],
|
| 887 |
[
|
| 888 |
+
5.794,
|
| 889 |
+
524.4
|
| 890 |
],
|
| 891 |
[
|
| 892 |
+
6.033,
|
| 893 |
+
521.3
|
| 894 |
],
|
| 895 |
[
|
| 896 |
+
6.271,
|
| 897 |
+
564.9
|
| 898 |
],
|
| 899 |
[
|
| 900 |
+
6.509,
|
| 901 |
+
547.8
|
| 902 |
],
|
| 903 |
[
|
| 904 |
+
6.746,
|
| 905 |
+
623.4
|
| 906 |
],
|
| 907 |
[
|
| 908 |
+
6.992,
|
| 909 |
+
643.7
|
| 910 |
],
|
| 911 |
[
|
| 912 |
+
7.234,
|
| 913 |
+
656.7
|
| 914 |
],
|
| 915 |
[
|
| 916 |
+
7.475,
|
| 917 |
+
667.5
|
| 918 |
],
|
| 919 |
[
|
| 920 |
+
7.716,
|
| 921 |
+
670.3
|
| 922 |
],
|
| 923 |
[
|
| 924 |
+
7.955,
|
| 925 |
+
675.6
|
| 926 |
]
|
| 927 |
],
|
| 928 |
"io_delta": {},
|
| 929 |
+
"timing_line": "[ Prompt: 7.9 t/s | Generation: 20.9 t/s ]",
|
| 930 |
"output": "Hello! 😊 I'm DeepSeek-R1, your friendly AI assistant. I'm here to help with all kinds of questions, learning, and creative tasks",
|
| 931 |
"pager": {
|
| 932 |
"routing_calls": 8448,
|
|
|
|
| 939 |
"resident_bytes": 132240384,
|
| 940 |
"budget_bytes": 134217728,
|
| 941 |
"hit_rate": 0.5415,
|
| 942 |
+
"proc_self_read_bytes": 2429915136,
|
| 943 |
"proc_self_rchar": 11947223
|
| 944 |
},
|
| 945 |
"output_matches_baseline": true,
|
| 946 |
"swap_zero": true,
|
| 947 |
"rc_ok": true,
|
| 948 |
+
"rss_vs_baseline": -1.9034,
|
| 949 |
+
"decode_rss_vs_baseline": -1.9034,
|
| 950 |
"decode_rss_lower_than_baseline": true,
|
| 951 |
"rss_lower_than_baseline": true,
|
| 952 |
"ssd_reads_mib": 2317.3,
|
validate/last-run.json
CHANGED
|
@@ -1,40 +1,40 @@
|
|
| 1 |
{
|
| 2 |
-
"when": "2026-10-07T16:
|
| 3 |
"model": "/root/work/models/SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf",
|
| 4 |
"port": 8080,
|
| 5 |
-
"pid":
|
| 6 |
"ram_budget_mb": 512,
|
| 7 |
"arena_mb": -1875,
|
| 8 |
"expert_slots": 252,
|
| 9 |
"chat": {
|
| 10 |
"prompt_tokens": 37,
|
| 11 |
"completion_tokens": 16,
|
| 12 |
-
"seconds": 3.
|
| 13 |
-
"tok_per_s": 4.
|
| 14 |
"content": "A **MoE (Model Parallelism) layer** enables a neural network to",
|
| 15 |
"timings": {
|
| 16 |
"cache_n": 0,
|
| 17 |
"prompt_n": 37,
|
| 18 |
-
"prompt_ms":
|
| 19 |
-
"prompt_per_token_ms":
|
| 20 |
-
"prompt_per_second":
|
| 21 |
"predicted_n": 16,
|
| 22 |
-
"predicted_ms":
|
| 23 |
-
"predicted_per_token_ms":
|
| 24 |
-
"predicted_per_second":
|
| 25 |
}
|
| 26 |
},
|
| 27 |
"peak": {
|
| 28 |
-
"total_rss_gb": 1.
|
| 29 |
"anon_rss_gb": 0.1931,
|
| 30 |
-
"file_rss_gb": 1.
|
| 31 |
"peak_swap_gb": 0.0,
|
| 32 |
-
"hwm_rss_gb": 1.
|
| 33 |
},
|
| 34 |
"io_delta": {
|
| 35 |
-
"read_bytes":
|
| 36 |
"rchar": 0,
|
| 37 |
-
"write_bytes":
|
| 38 |
},
|
| 39 |
"pager": {
|
| 40 |
"routing_calls": 4608,
|
|
@@ -47,7 +47,7 @@
|
|
| 47 |
"resident_bytes": 535071744,
|
| 48 |
"budget_bytes": 536870912,
|
| 49 |
"hit_rate": 0.4122,
|
| 50 |
-
"proc_self_read_bytes":
|
| 51 |
"proc_self_rchar": 11946335
|
| 52 |
},
|
| 53 |
"pager_expert_resident_mib": 510.3,
|
|
|
|
| 1 |
{
|
| 2 |
+
"when": "2026-10-07T16:47:29+0200",
|
| 3 |
"model": "/root/work/models/SmallThinker-4B-A0.6B-Instruct.Q4_K.gguf",
|
| 4 |
"port": 8080,
|
| 5 |
+
"pid": 38517,
|
| 6 |
"ram_budget_mb": 512,
|
| 7 |
"arena_mb": -1875,
|
| 8 |
"expert_slots": 252,
|
| 9 |
"chat": {
|
| 10 |
"prompt_tokens": 37,
|
| 11 |
"completion_tokens": 16,
|
| 12 |
+
"seconds": 3.89,
|
| 13 |
+
"tok_per_s": 4.1168,
|
| 14 |
"content": "A **MoE (Model Parallelism) layer** enables a neural network to",
|
| 15 |
"timings": {
|
| 16 |
"cache_n": 0,
|
| 17 |
"prompt_n": 37,
|
| 18 |
+
"prompt_ms": 3122.416,
|
| 19 |
+
"prompt_per_token_ms": 84.38962162162163,
|
| 20 |
+
"prompt_per_second": 11.849798361268965,
|
| 21 |
"predicted_n": 16,
|
| 22 |
+
"predicted_ms": 744.758,
|
| 23 |
+
"predicted_per_token_ms": 49.650533333333335,
|
| 24 |
+
"predicted_per_second": 20.140770559027224
|
| 25 |
}
|
| 26 |
},
|
| 27 |
"peak": {
|
| 28 |
+
"total_rss_gb": 1.0901,
|
| 29 |
"anon_rss_gb": 0.1931,
|
| 30 |
+
"file_rss_gb": 1.0845,
|
| 31 |
"peak_swap_gb": 0.0,
|
| 32 |
+
"hwm_rss_gb": 1.0901
|
| 33 |
},
|
| 34 |
"io_delta": {
|
| 35 |
+
"read_bytes": 1160712192,
|
| 36 |
"rchar": 0,
|
| 37 |
+
"write_bytes": 4096
|
| 38 |
},
|
| 39 |
"pager": {
|
| 40 |
"routing_calls": 4608,
|
|
|
|
| 47 |
"resident_bytes": 535071744,
|
| 48 |
"budget_bytes": 536870912,
|
| 49 |
"hit_rate": 0.4122,
|
| 50 |
+
"proc_self_read_bytes": 2439696384,
|
| 51 |
"proc_self_rchar": 11946335
|
| 52 |
},
|
| 53 |
"pager_expert_resident_mib": 510.3,
|