onwx 1.3.1: image encoder weights kept bf16 (1.8 -> 0.9 GB), OpenBLAS 8 threads; chat, window and benchmark fixes

CHANGELOG.md

@@ -5,6 +5,13 @@
 日付は日本時間です。各版は Hugging Face の `ryugyosoft/onwx` にタグ（`v0.2.0` 以降）付きで残っています。
 onwx ウィンドウは、新しい版があるとこのファイルから変更点を表示します（`## v…` の見出しで区切っています）。
 
+## v1.3.1 (2026-10-31)
+- 画像を読むエンコーダー（Qwen3.5）の重みを bf16 のまま持つようにしました（最初の画像のあと、約 1.8 GB → 約 0.9 GB）。メモリ 24 GB 未満の PC では、画像を読み終えたら手放します（onw と同じ。`ONWX_KEEP_VISION=1` / `0` で指定もできます）
+- サーバーの numpy が CPU のスレッド数ぶん確保していたメモリ（コミット済み、約 750 MB）を約 250 MB にしました（使う量は変わりません）
+- チャット画面：回答中の会話を履歴から開き直すと、直前の質問と途中の回答が消えることがあったのを直しました。上部の速度表示は、新しい会話・別の会話・モデルの切り替えで消えるようにしました。狭いウィンドウでヘッダーが崩れていたのを直しました
+- ウィンドウ：設定を保存した直後に、古い値が一瞬表示に戻ることがあったのを直しました
+- ベンチマーク：コピーする条件（生成トークン数・回数）を、測ったときのものにしました。2 回目以降が失敗しても、それまでの結果を残します。最良値は、いちばん速かった回のトークン数・再利用数で出します。モデルの読み込み中に開いても、読み込みが終われば使えます。考えるモデルでは、コンテキストに収まるかの判定に思考の分も入れました
+
 ## v1.3.0 (2026-10-31)
 - 検証ページを **ベンチマーク**（`/bench`）に変えました。128・1K・2K・4K（8K も選べます）トークンのプロンプトで、プロンプトの読み込み（プリフィル）の速さと、そのあとの生成（デコード）の速さを表にします。プロンプトはそのモデル自身のトークナイザーで長さを合わせ、毎回違う一文で始めるので前の会話は再利用されません。「結果をコピー（Markdown）」で、アプリ・モデル・CPU・OS と一緒に表をコピーできます（記事などに）。トレイとウィンドウの「検証ページ」も「ベンチマーク」になり、前のアドレス（`/check`）はベンチマークへ移ります
 - チャット画面の上部に、直前の回答のプリフィル速度と生成速度を出すようにしました

CHANGELOG_en.md

@@ -5,6 +5,13 @@
 Dates are Japan time. Every version is kept on Hugging Face `ryugyosoft/onwx` with a tag (from `v0.2.0` on).
 The onwx window shows what changed from this file when there is a newer version (sections start at `## v…`).
 
+## v1.3.1 (2026-10-31)
+- The image encoder (Qwen3.5) keeps its weights in bf16 (after the first image about 1.8 GB -> about 0.9 GB); on PCs with less than 24 GB it is let go after each image (as in onw; `ONWX_KEEP_VISION=1` / `0` decides instead)
+- The server's numpy reserved memory per CPU thread (about 750 MB committed); now about 250 MB (what it uses is unchanged)
+- Chat page: reopening the conversation that is being answered could lose its last question and the partial answer: fixed. The speeds at the top go away with a new conversation, another conversation or another model. The header no longer breaks up in narrow windows
+- Window: right after a save the old values could flash back for a moment: fixed
+- Benchmark: the copied conditions (tokens generated, runs) are the measured run's; a failed later run keeps the results before it; the best result shows the fastest run's own token and reuse counts; opened while the model loads, it works once the model is in; for thinking models the context check counts the thinking too
+
 ## v1.3.0 (2026-10-31)
 - The check page is now a **benchmark** (`/bench`): prompts of 128, 1K, 2K and 4K tokens (8K can be added), with how fast the prompt is read (prefill) and how fast the answer is then generated (decode). The prompts are cut to length by the model's own tokenizer and start with a different sentence every time, so nothing of an earlier conversation is reused. "Copy the results (Markdown)" copies the table with the app, model, CPU and OS (for articles and posts). The tray's and the window's "check page" entries open the benchmark, and the old address (`/check`) leads there
 - The chat page shows the last answer's prefill and decode speed at the top

onwx/__init__.py

@@ -2,6 +2,6 @@
 
 (A working name: the product's name is set here and in the install scripts.)
 """
-__version__ = "1.3.0"
+__version__ = "1.3.1"
 NAME = "onwx"
 TITLE = "onwx (AMD Ryzen AI NPU)"

onwx/app.py

@@ -666,6 +666,9 @@ def _start_server(cfg):
     if int(cfg["context"]) != 1024:
         args += ["--context", str(cfg["context"])]
     env = {**os.environ, "PYTHONUTF8": "1"}
+    # [bugs 10-31] numpy's OpenBLAS reserves a buffer per CPU thread at import (~750 MB committed with 24-32 threads,
+    # ~30 MB used); 8 threads keep the CPU-side matrix work (image encoders) fast
+    env.setdefault("OPENBLAS_NUM_THREADS", "8")
     if cfg.get("api_key"):
         env["ONWX_API_KEY"] = cfg["api_key"]
     if cfg.get("think_default"):                      # API requests that say nothing about thinking: think

onwx/chat.py

@@ -76,11 +76,13 @@ class ChatEngine:
         self._drop = sorted(specials - set(_tools.MARKERS), key=len, reverse=True)   # ... when tool markup is kept
         self._tool_stop = {i for i in (self.tok.convert_tokens_to_ids(t) for t in _tools.STOP_AFTER)
                            if isinstance(i, int) and i != self.tok.unk_token_id}
-        # images (Qwen3.5 with its vision tower): the tower runs on the CPU, loaded at the first image (~1.6 GB) and
-        # kept while the model is (ONWX_KEEP_VISION=0 frees it after each image)
+        # images (Qwen3.5 with its vision tower): the tower runs on the CPU, loaded at the first image (~0.9 GB, its
+        # weights bf16) and kept while the model is on PCs with 24 GB or more ([bugs 10-31] as onw; smaller ones let
+        # it go after each image); ONWX_KEEP_VISION=1 / 0 decides instead
         self.vision = self.model.meta.get("vision")
         self._venc, self._grids = None, []
-        self.keep_vision = os.environ.get("ONWX_KEEP_VISION", "1") != "0"
+        kv, ram = os.environ.get("ONWX_KEEP_VISION", ""), total_ram_gb()
+        self.keep_vision = kv == "1" if kv in ("0", "1") else (ram is None or ram >= 24)
         # end-of-turn ids: text config, full config and generation_config (Gemma lists <eos>, <turn|>, ...)
         self.eos = {self.tok.eos_token_id}
         cands = [self.model.cfg.get("eos_token_id")]

onwx/qwen_vision.py

@@ -70,15 +70,16 @@ class QwenVision:
         """d: the model folder (vision.npz); cfg: config.json's vision_config"""
         self.cfg = cfg
         z = np.load(os.path.join(d, "vision.npz"))
-        self.W = {k: ((z[k].astype(np.uint32) << 16).view(np.float32) if z[k].dtype == np.uint16 else z[k].astype(np.float32))
-                  for k in z.files}
+        # [bugs 10-31] kept as stored (bf16 as uint16: half of float32's memory), widened when a weight is used
+        self.W = {k: (z[k] if z[k].dtype == np.uint16 else z[k].astype(np.float32)) for k in z.files}
         self.D, self.nh = cfg["hidden_size"], cfg["num_heads"]
         self.hd = self.D // self.nh
         self.side = int(cfg["num_position_embeddings"] ** 0.5)
         self.m = cfg["spatial_merge_size"]
 
     def _w(self, n):
-        return self.W[VP + n]
+        a = self.W[VP + n]
+        return (a.astype(np.uint32) << 16).view(np.float32) if a.dtype == np.uint16 else a
 
     def _tables(self, grid):
         """position embeddings [N, D] and rotary cos / sin [N, hd] of the patches (merge-window order)"""

onwx/web/bench.html

@@ -68,7 +68,7 @@ td.big { font-weight: 700; }
 <script>
 translate(document.body);
 const $ = id => document.getElementById(id);
-let INFO = {}, KEY = new URLSearchParams(location.search).get("key") || "", RESULTS = [];
+let INFO = {}, KEY = new URLSearchParams(location.search).get("key") || "", RESULTS = [], LAST = {};
 
 const esc = s => String(s ?? "").replace(/[&<>"]/g, c => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c]));
 const fmt = (x, d) => (x == null || !isFinite(x)) ? "-" : Number(x).toLocaleString(undefined, { minimumFractionDigits: d, maximumFractionDigits: d });
@@ -116,7 +116,7 @@ async function run() {
   const lens = [...document.querySelectorAll(".len")].filter(c => c.checked).map(c => +c.value);
   const gen = +$("gen").value, reps = +$("reps").value;
   const ctx = INFO.context || INFO.max_context || 0;
-  $("run").disabled = true; $("copy").disabled = true; RESULTS = []; show();
+  $("run").disabled = true; $("copy").disabled = true; RESULTS = []; LAST = { gen, reps }; show();
   try {
     $("status").textContent = T("慣らし運転…");
     await api(await prompt(64, 0), 8);
@@ -124,16 +124,16 @@ async function run() {
   let k = 0;
   for (const n of lens) {
     const label = T("{0} トークン", n.toLocaleString());
-    if (ctx && n + gen + 64 > ctx) { RESULTS.push({ label, err: T("コンテキスト長（{0}）を超えるので飛ばしました", ctx) }); show(); continue; }
+    if (ctx && n + gen + (INFO.always_thinks ? 1024 : 0) + 64 > ctx) { RESULTS.push({ label, err: T("コンテキスト長（{0}）を超えるので飛ばしました", ctx) }); show(); continue; }
     let best = null;
     for (let i = 0; i < reps; i++) {
       $("status").textContent = T("{0}（{1}/{2}回目）…", label, i + 1, reps);
       try {
         const m = measure(await api(await prompt(n, ++k), gen));
-        best = best ? { ...best, prefill_s: Math.min(best.prefill_s, m.prefill_s),
-                        prefill_tps: Math.max(best.prefill_tps || 0, m.prefill_tps || 0),
-                        decode_tps: Math.max(best.decode_tps || 0, m.decode_tps || 0) } : m;
-      } catch (e) { best = { err: e.message }; break; }
+        const dec = Math.max(best?.decode_tps || 0, m.decode_tps || 0);
+        if (!best || (m.prefill_tps || 0) > (best.prefill_tps || 0)) best = { ...m };
+        best.decode_tps = dec || null;
+      } catch (e) { if (!best) best = { err: e.message }; break; }   // (a run that came through stays)
     }
     RESULTS.push({ label: best.err ? label : T("{0} トークン", best.prompt.toLocaleString()), ...best }); show();
   }
@@ -142,7 +142,7 @@ async function run() {
 
 function markdown() {
   const env = [[T("アプリ"), INFO.app], [T("モデル"), INFO.model], [T("デバイス"), Object.values(INFO.devices || {}).join(", ")],
-               [T("CPU"), INFO.cpu], [T("OS"), INFO.os], [T("条件"), T("温度 0・思考オフ・生成 {0} トークン・{1} 回の最良値", $("gen").value, $("reps").value)],
+               [T("CPU"), INFO.cpu], [T("OS"), INFO.os], [T("条件"), T("温度 0・思考オフ・生成 {0} トークン・{1} 回の最良値", LAST.gen, LAST.reps)],
                [T("測定日"), new Date().toLocaleDateString()]].filter(([, v]) => v);
   const head = `| ${T("プロンプト")} | ${T("プリフィル")} | ${T("プリフィル速度")} | ${T("生成速度")} |\n|---|---:|---:|---:|`;
   const body = RESULTS.filter(r => !r.err).map(r =>
@@ -157,17 +157,21 @@ $("copy").onclick = async () => {
           document.execCommand("copy"); a.remove(); $("status").textContent = T("コピーしました"); }
 };
 
+function loadInfo() {
 fetch("/api/info").then(r => r.json()).then(i => {
+  if (!i.model) { setTimeout(loadInfo, 2000); return; }   // (the model is still loading)
   INFO = i;
   document.title = $("h1").textContent = T("{0} ベンチマーク", (i.app || "").split(" ")[0] || "LLM");
   const rows = [[T("アプリ"), i.app], [T("モデル名"), i.model], [T("デバイス"), Object.values(i.devices || {}).join(", ")],
-    [T("CPU"), i.cpu], [T("OS"), i.os], [T("コンテキスト長"), T("{0} トークン", i.context || i.max_context)],
+    [T("CPU"), i.cpu], [T("OS"), i.os], [T("コンテキスト長"), (i.context || i.max_context) && T("{0} トークン", i.context || i.max_context)],
     [T("先読み（MTP・ドラフト）"), i.mtp ? T("有効") : T("なし")]].filter(([, v]) => v);
   $("info").innerHTML = rows.map(([k, v]) => `<tr><td>${esc(k)}</td><td>${esc(v)}</td></tr>`).join("");
   $("base").textContent = location.origin + "/v1"; $("mid").textContent = i.model;
   const ctx = i.context || i.max_context || 0;
   for (const c of document.querySelectorAll(".len")) if (ctx && +c.value + 512 > ctx) { c.checked = false; c.disabled = true; }
-}).catch(() => { $("info").innerHTML = `<tr><td class='bad'>${T("サーバーに接続できません")}</td></tr>`; });
+}).catch(() => { $("info").innerHTML = `<tr><td class='bad'>${T("サーバーに接続できません")}</td></tr>`; setTimeout(loadInfo, 3000); });
+}
+loadInfo();
 $("run").onclick = run;
 </script>
 </body>

onwx/web/index.html

@@ -21,8 +21,11 @@
   html, body { height: 100%; margin: 0; }
   body { background: var(--bg); color: var(--text); font: 15px/1.6 system-ui, -apple-system, "Segoe UI", "Hiragino Sans", "Noto Sans JP", sans-serif; display: flex; flex-direction: column; }
   header { display: flex; align-items: center; gap: 12px; padding: 10px 16px; border-bottom: 1px solid var(--border); background: var(--panel); }
-  header h1 { font-size: 16px; margin: 0; font-weight: 600; }
-  .badge { font-size: 12px; padding: 2px 8px; border-radius: 999px; border: 1px solid var(--border); color: var(--muted); }
+  header h1 { font-size: 16px; margin: 0; font-weight: 600; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+  .badge { font-size: 12px; padding: 2px 8px; border-radius: 999px; border: 1px solid var(--border); color: var(--muted); white-space: nowrap; }
+  @media (max-width: 600px) { #speed { display: none !important; } }
+  header button { white-space: nowrap; flex-shrink: 0; }
+  #dev { min-width: 0; overflow: hidden; text-overflow: ellipsis; }
   .badge.ok { color: var(--accent); border-color: var(--accent); }
   header .spacer { flex: 1; }
   button { font: inherit; cursor: pointer; border-radius: 8px; border: 1px solid var(--border); background: var(--panel); color: var(--text); padding: 6px 12px; }
@@ -226,6 +229,7 @@ function loadInfo() {   // the server's model and settings: tried again until it
     const id = i.model || i.title || "", first = !MODEL;
     if (MODEL && id === MODEL) return;     // the same model: its settings (and the user's changes) stay
     MODEL = id;
+    clearSpeed();
     if (i.title) { $("title").textContent = i.title; document.title = i.title + " on NPU"; }
     VISION = i.vision !== false; $("attach").style.display = VISION ? "" : "none";
     if (!VISION) { pending = []; showThumbs(); if (first && $("empty")) $("empty").textContent = T("メッセージを入力してください。"); }
@@ -398,8 +402,10 @@ document.addEventListener("dragleave", () => document.body.classList.remove("dra
 document.addEventListener("drop", e => { e.preventDefault(); document.body.classList.remove("drag"); addFiles(e.dataTransfer.files); });
 $("input").addEventListener("input", e => { e.target.style.height = "auto"; e.target.style.height = e.target.scrollHeight + "px"; });
 $("input").addEventListener("keydown", e => { if (e.key === "Enter" && !e.shiftKey && !e.isComposing) { e.preventDefault(); if (!controller) send(); } });   // (not while an answer is written)
+function clearSpeed() { const b = document.getElementById("speed"); if (b) b.style.display = "none"; }
 function newChat() {
   if (controller) controller.abort();
+  clearSpeed();
   history.length = 0; pending = []; showThumbs(); chat = { id: null, created: 0, title: "", msgs: [] };
   $("log").innerHTML = `<div class="empty" id="empty">${T("新しい会話です。")}</div>`;
   listChats();
@@ -486,7 +492,9 @@ async function listChats() {
   }
 }
 async function openChat(id) {
+  if (id === chat.id) return;          // (the open one - an answer may still be coming in)
   if (controller) controller.abort();
+  clearSpeed();
   const r = await fetch("/api/chats/" + id); if (!r.ok) return listChats();
   const c = await r.json();
   chat = { id: c.id, created: c.created, title: c.title, model: c.model, msgs: c.messages || [] };

onwx/web/manage.html

@@ -189,14 +189,14 @@
     <div class="card">
       <div class="row">
         <span class="muted">使うモデル</span>
-        <select class="grow" id="pick"></select>
+        <select class="grow" id="pick" aria-label="モデル"></select>
         <button class="primary" id="b-load" onclick="this.dataset.act === 'unload' ? post('unload') : loadPicked()">ロード</button>
       </div>
       <div class="muted small" style="margin-top:5px">一度に 1 モデル。別のモデルを選んで「切り替え」を押すと、今のモデルをアンロードしてから読み込みます。</div>
     </div>
     <div class="card set" id="prefcard">
       <div class="row"><b>このモデルの既定</b><span class="muted small" id="pf-model"></span></div>
-      <div class="opt" id="pf-think-row"><span class="k">思考</span><input type="range" id="pf-think" min="0" max="3" step="1" style="width:220px"><b id="pf-think-label" class="small"></b>
+      <div class="opt" id="pf-think-row"><span class="k">思考</span><input type="range" id="pf-think" aria-label="思考" min="0" max="3" step="1" style="width:220px"><b id="pf-think-label" class="small"></b>
         <span class="desc" id="pf-think-desc"></span></div>
       <div class="opt"><span class="k">生成設定</span><select id="pf-sampling" style="width:360px;max-width:100%"><option value="greedy">貪欲（温度 0。毎回同じ答え）</option><option value="recommended">モデルの推奨値</option><option value="custom">自分で指定</option></select></div>
       <div class="opt" id="pf-custom"><span class="k"></span>
@@ -481,7 +481,7 @@
     if ($("pf-think-row").style.display === "none") delete prefs.think;
     const r = await post("model_prefs", { key, prefs });
     if (r.error) return;
-    prefDirty = false; $("pf-saved").textContent = T("保存しました（すぐ反映されます）");
+    STALE = REQ; prefDirty = false; $("pf-saved").textContent = T("保存しました（すぐ反映されます）");
   }
   // row buttons carry the model key in data-key (any folder name, quotes too) - one handler for the table and the menu
   function btn(act, key, label, cls, off) {
@@ -558,7 +558,7 @@
     const oldLang = S.config.lang || "auto";
     const r = await post("settings", body);
     if (r.error) return;
-    setDirty(false);
+    STALE = REQ; setDirty(false);
     $("saved").textContent = T("保存しました");
     if (r.restart_needed && await ask(T("設定は保存しました。サーバーを新しい設定で今すぐ再起動しますか？（キャンセルしても、次にモデルを読み込むときから反映されます）"))) await post("restart");
     if (oldLang !== body.lang) location.reload();          // the page is served in the new language
@@ -635,12 +635,16 @@
   if (justUpdated) setTimeout(() => toast(T("v{0} に更新しました", justUpdated)), 400);
 
   async function refresh() {
+    const n = ++REQ;
     try {
       const r = await fetch("/api/state?token=" + encodeURIComponent(TOKEN), {cache: "no-store"});
       if (r.status === 403) { $("env").textContent = T("この画面は onwx のウィンドウかトレイから開いてください"); return; }
-      S = await r.json(); render();
+      const j = await r.json();
+      if (n <= STALE) return;              // (asked before the last save: shows the old values)
+      S = j; render();
     } catch (e) { $("hpill").className = "pill danger"; $("hpill").textContent = T("接続なし"); }
   }
+  let REQ = 0, STALE = 0;               // state polls / the last one started before a save
   refresh(); setInterval(refresh, 1000);
 </script>
 </body>