onwx 0.6.1: faster reading of what agents add after a long context (Hermes run 989 -> 778 s), tool calls written inside the thinking, API max output tokens setting (default 8192), an anchor checkpoint at user messages, clearer XRT diagnosis on Linux

README.md

@@ -45,6 +45,7 @@ onwx の MTP（温度 0 のときの先読み）を使った値は、生成の
 - 512 トークンの回答をプロンプトから書き終えるまでの実効速度（512 ÷ 合計時間）は、128〜32K のどの長さでも onwx が上です（MTP なしでも）。
 - プロンプトの読み込みは 2K〜8K で onwx が速く（1K はほぼ同じ。4K で FLM 8.7 秒、onwx 7.7 秒）、256〜512 と 16K〜32K では FLM が少し速いです。CPU も使う設定では 16K まで onwx が速く、32K でほぼ同じです（FLM 74 秒、onwx 74 秒）。
 - 生成はどの長さでも onwx が 3〜4 割速く、MTP を使うと 8K までの文脈で約 14〜18 tok/s です（24K を超えると MTP は自動で止まります）。
+- **エージェントでも速い**：Hermes Agent のコーディング作業（バグ修正、29 回のやり取り、文脈 1.2 万〜2.9 万トークン）で記録した同じリクエストを流すと、全体で onwx 0.6.1 が 778 秒、FLM が 874 秒でした（0.6.0 は 989 秒。長い文脈の後ろに足す読み込みを 0.6.1 で速くしました）。
 - Ryzen AI 9 HX 370 でも傾向は同じでした（onwx 0.5.0。32K のプロンプト：FLM 379 tok/s、onwx 339 tok/s、生成 FLM 6.0 tok/s、onwx 8.9 tok/s）。
 
 ## インストール（コマンドを 1 行貼るだけ）
@@ -77,7 +78,7 @@ curl -fsSL https://huggingface.co/ryugyosoft/onwx/resolve/main/install.sh | bash
   Ryzen AI Max シリーズ（Strix Halo、例：Ryzen AI Max+ 395）
 - **Windows 11** と AMD の NPU ドライバー（32.0.203 以降で確認。Windows Update か PC メーカーのドライバーで入ります）。
   タスクマネージャーの「パフォーマンス」に NPU が出ていれば大丈夫です
-- Ubuntu では NPU ドライバー（amdxdna）と XRT（Python バインディング pyxrt 付き）
+- Ubuntu では NPU ドライバー（amdxdna）に加えて、AMD の NPU ランタイム XRT（Python バインディング pyxrt 付き）が必要です。カーネルのドライバーだけでは動きません。AMD の [xdna-driver](https://github.com/amd/xdna-driver) の手順で XRT と NPU プラグインを入れてください（`/opt/xilinx/xrt` に入ります）。足りないものがあると、onwx のウィンドウに理由が出ます
 - メモリ 16 GB 以上（Ternary Bonsai 2 27B で約 8.8 GB を使います）
 - 空き容量：アプリ約 0.5 GB ＋ モデル 7.6 GB
 - インターネット接続（最初のダウンロードのとき）
@@ -122,6 +123,7 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx      # 端末でチャット
 `/v1/chat/completions`（ストリーミング可）、`/v1/models`、`/v1/messages`（Anthropic 形式）、`/v1/responses` に対応しています。
 思考モードは `"chat_template_kwargs": {"enable_thinking": true}` か `"reasoning_effort"` で切り替えます。
 設定で LAN からの接続を許可すると、他の PC からも使えます（API キーの設定をおすすめします）。
+アプリが `max_tokens` を送らないときの回答の長さの上限は、設定の「API の最大出力トークン」で変えられます（既定 8192）。
 
 ## ライセンス
 

README_en.md

@@ -33,6 +33,7 @@ match. onwx with its MTP drafts (temperature 0) is shown apart, as a dashed line
 - A 512-token answer, start to finish (512 / total time): onwx is ahead at every length from 128 to 32K, even without MTP.
 - Prompt reading: onwx is faster from 2K to 8K (1K about the same; 4K: FLM 8.7 s, onwx 7.7 s); FLM is a little faster at 256-512 and 16K-32K. With the CPU setting onwx is faster up to 16K and about the same at 32K (FLM 74 s, onwx 74 s).
 - Generation: onwx is 30-40% faster at every length; with MTP about 14-18 tok/s up to 8K of context (past 24K MTP turns itself off).
+- **Fast for agents too**: the same recorded requests of a Hermes Agent coding task (a bug fix, 29 requests, 12K-29K tokens of context) took 778 s on onwx 0.6.1 and 874 s on FLM (0.6.0: 989 s - 0.6.1 reads what is added after a long context faster).
 - The Ryzen AI 9 HX 370 shows the same picture (onwx 0.5.0; a 32K prompt: FLM 379 tok/s, onwx 339 tok/s; generation FLM 6.0, onwx 8.9 tok/s).
 
 ## Install (paste one line)
@@ -61,7 +62,7 @@ curl -fsSL https://huggingface.co/ryugyosoft/onwx/resolve/main/install.sh | bash
   series (Strix Halo, e.g. Ryzen AI Max+ 395)
 - **Windows 11** with AMD's NPU driver (tested with 32.0.203; it comes with Windows Update or the PC maker's drivers) -
   if Task Manager's Performance tab shows an NPU, you are set
-- On Ubuntu: the NPU driver (amdxdna) and XRT with its Python bindings (pyxrt)
+- On Ubuntu: besides the NPU driver (amdxdna), AMD's NPU runtime XRT with its Python bindings (pyxrt) - the kernel driver alone is not enough. Install XRT and its NPU plugin as AMD's [xdna-driver](https://github.com/amd/xdna-driver) describes (it goes to `/opt/xilinx/xrt`); if something is missing, the onwx window says what
 - 16 GB of RAM or more (Ternary Bonsai 2 27B takes about 8.8 GB)
 - Disk: about 0.5 GB for the app + 7.6 GB for the model
 
@@ -104,7 +105,7 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx      # chat in the terminal
 
 `/v1/chat/completions` (streaming too), `/v1/models`, `/v1/messages` (Anthropic format) and `/v1/responses`.
 Thinking: `"chat_template_kwargs": {"enable_thinking": true}` or `"reasoning_effort"`. Allow LAN connections in the
-settings to use it from other PCs (set an API key then).
+settings to use it from other PCs (set an API key then). The longest answer when an app sends no `max_tokens` is the setting "API max output tokens" (default 8192).
 
 ## License
 

onwx/__init__.py

@@ -2,6 +2,6 @@
 
 (A working name: the product's name is set here and in the install scripts.)
 """
-__version__ = "0.6.0"
+__version__ = "0.6.1"
 NAME = "onwx"
 TITLE = "onwx (AMD Ryzen AI NPU)"

onwx/app.py

@@ -40,7 +40,7 @@ MEMORY = {
 STREAMABLE = {}                      # (no model streams weights from the SSD)
 DEFAULTS = {"model": "", "port": 8100, "lan": False, "api_key": "", "context": 3840,
             "autostart_server": True, "login_start": False, "models_dir": "", "model_folders": [], "update_check": True,
-            "think_default": False, "lang": "auto", "cpu_glue": False,
+            "think_default": False, "lang": "auto", "cpu_glue": False, "api_max_tokens": 8192,
             "api_sampling": "greedy", "api_temperature": 0.7, "api_top_p": 0.95, "api_top_k": 20, "api_presence": 0.0,
             "model_prefs": {}}
 CONTEXTS = ["1024", "2048", "3840", "8192", "16384", "32768", "65536", "131072", "262144"]   # (models built before the
@@ -639,6 +639,7 @@ def _start_server(cfg):
         env["ONWX_THINK_DEFAULT"] = "1"
     if cfg.get("cpu_glue"):                           # [0.5] the prefill glue ops on the CPU (opt-in)
         env["ONWX_CPU_GLUE"] = "1"
+    env["ONWX_MAX_TOKENS"] = str(int(cfg.get("api_max_tokens", 8192)))   # [0.6.1] requests without max_tokens
     env["ONWX_MODEL_KEY"] = cfg["model"]               # the server reads this model's defaults (model_prefs) itself
     # API requests that send no sampling values (agent apps like OpenCode): greedy, the model's recommendation or set
     if cfg.get("api_sampling") == "recommended":

onwx/chat.py

@@ -97,6 +97,7 @@ class ChatEngine:
         self.device = "NPU"
         self.pld = False
         self.checkpoint = None                 # (token ids, model snapshot) at the end of the last conversation
+        self.anchor = None                     # (the same at the end of its last user message)
         m = self.model                         # warm up (a request's first inference is slow)
         for rep in range(2):
             m.reset()
@@ -138,6 +139,10 @@ class ChatEngine:
         if "<think>" in raw:
             pre, _, rest = raw.partition("<think>")
             reasoning, closed, post = rest.partition("</think>")
+            tc = reasoning.find("<tool_call>") if markup else -1
+            if tc >= 0:                                          # a call written inside the thought (Qwen3.5 does it
+                reasoning, post, closed = reasoning[:tc], reasoning[tc:] + post, True   # after tool results, never
+                                                                 # closing it): the thought ends where the call starts
             if not closed:                                       # still thinking (hold back a half-written tag)
                 for k in range(len("</think>") - 1, 0, -1):
                     if reasoning.endswith("</think>"[:k]):
@@ -428,14 +433,21 @@ class ChatEngine:
             pos, delta = self._positions(ids)
         key = lambda upto: (ids[:upto], [h for s0, e0, h in spans if s0 < upto])
         # prefix reuse: state kept at the end of the previous conversation (before its generation prompt)
-        ck, reuse = self.checkpoint, 0
-        if ck is not None and len(ck[0][0]) <= B and key(len(ck[0][0])) == ck[0]:
-            reuse = len(ck[0][0])
-            self.model.restore(ck[1])
+        # or, failing that, at the end of the conversation's last user message (the anchor): agents add a user message
+        # after tool turns (Hermes: "continue"), and Qwen's template then drops the thinking of the turns since the
+        # last user message - the prompt differs from there on, not from the start
+        reuse = 0
+        for ck in (self.checkpoint, self.anchor):
+            if ck is not None and len(ck[0][0]) <= B and key(len(ck[0][0])) == ck[0]:
+                reuse = len(ck[0][0])
+                self.model.restore(ck[1])
+                if ck is not self.checkpoint:       # (the latest one's rows above the anchor get overwritten)
+                    self.checkpoint = None
+                break
         else:
             self.model.reset()
-            self.checkpoint = None              # (its KV rows are overwritten now: a prefill failing part-way would
-                                                # leave a checkpoint over another conversation's rows)
+            self.checkpoint = self.anchor = None    # (their KV rows are overwritten now: a prefill failing part-way
+                                                    # would leave a checkpoint over another conversation's rows)
         t0 = time.time()
         t_vis = 0.0
         if images and any(e0 > reuse for s0, e0, _ in spans):
@@ -454,11 +466,32 @@ class ChatEngine:
         self.model.feeding = True                   # ([MTP] the MTP layer's rows go along with the prompt)
         w = int(os.environ.get("ONWX_MTP_ROWS", "128") or 0)   # ([MTP] rows of the last w prompt tokens only (~3.5 ms a row); 0: all)
         self.model.mtp_from = max(0, n - w) if w > 0 else 0
+        user = bool(messages) and messages[-1].get("role") == "user"
+        al = self.model.chunk_align() if hasattr(self.model, "chunk_align") and embeds is None and pos is None else 0
         try:
-            if B > reuse:
-                self._prefill(ids, reuse, B, need_logits=False, embeds=embeds, pos=pos)
-            self.checkpoint = (key(B), self.model.snapshot())
-            lg = self._prefill(ids, B, n, embeds=embeds, pos=pos)
+            if al:
+                # [0.6.1] checkpoints at multiples of the chunks' start unit (256): a chunk can only start there, and
+                # one that starts elsewhere has up to 255 tokens to read on the old programs first (seconds deep in a
+                # long cache). Each request reads from the checkpoint to its end in one go (re-reading what the last
+                # one added - a padded chunk costs the same); the checkpoint moves up once that passes 1024 tokens,
+                # or at a new user message (the anchor)
+                A = B // al * al
+                if A > reuse and (user or n - reuse > 1024):
+                    self._prefill(ids, reuse, A, need_logits=False)
+                    self.checkpoint = (key(A), self.model.snapshot())
+                    if user:
+                        self.anchor = self.checkpoint
+                    reuse_to = A
+                else:
+                    reuse_to = reuse
+                lg = self._prefill(ids, reuse_to, n)
+            else:
+                if B > reuse:
+                    self._prefill(ids, reuse, B, need_logits=False, embeds=embeds, pos=pos)
+                self.checkpoint = (key(B), self.model.snapshot())
+                if user:
+                    self.anchor = self.checkpoint   # (a new user message: B is its end)
+                lg = self._prefill(ids, B, n, embeds=embeds, pos=pos)
             self.model.delta = delta
         except Cancelled:
             yield {"finish_reason": "stop", "prompt_tokens": n, "cached_tokens": reuse, "completion_tokens": 0,

onwx/i18n.json

@@ -1,5 +1,8 @@
 {
 "en": {
+"API の最大出力トークン": "API max output tokens",
+"文脈の残りまで": "Up to the rest of the context",
+"アプリが上限を指定しないときの、1 回の回答の長さの上限です（エージェントがファイルを書くときなどに効きます）。大きくしても使用メモリは増えません。": "The longest single answer when an app sets no limit (matters when agents write files). A larger value uses no more memory.",
 "CPU も使ってプロンプトの読み込みを速くする（消費電力が増えます）": "Use the CPU too, to read prompts faster (uses more power)",
 "長いプロンプトほど効きます。ノート PC ではファンの音が大きくなることがあります。": "Helps most with long prompts. On a laptop the fan may get louder.",
 " + 画像 {0}秒": " + image {0} s",
@@ -363,6 +366,9 @@
 "日本語に強い 8B を int4 の重みで、全部 NPU で動かします（国立情報学研究所の llm-jp-4.1-8b-thinking。考えてから答えるモデルです。生成は Ryzen AI 9 HX 370 で約 11 tok/s、1,000 トークン以上のプロンプトの読み込みは Ryzen AI Max+ 395 で約 300〜430 tok/s）。文脈は最大 65536 トークン": "An 8B strong in Japanese, with int4 weights, run entirely on the NPU (NII's llm-jp-4.1-8b-thinking, a model that thinks before it answers; generation about 11 tok/s on a Ryzen AI 9 HX 370, prompts of 1,000+ tokens read at about 300-430 tok/s on a Ryzen AI Max+ 395). Context up to 65536 tokens"
 },
 "zh": {
+"API の最大出力トークン": "API 最大输出 token 数",
+"文脈の残りまで": "直到上下文用完",
+"アプリが上限を指定しないときの、1 回の回答の長さの上限です（エージェントがファイルを書くときなどに効きます）。大きくしても使用メモリは増えません。": "应用未指定上限时，单次回答长度的上限（智能体写文件时会用到）。调大不会增加内存占用。",
 "CPU も使ってプロンプトの読み込みを速くする（消費電力が増えます）": "同时使用 CPU 加快读取提示词（耗电增加）",
 "長いプロンプトほど効きます。ノート PC ではファンの音が大きくなることがあります。": "提示词越长效果越明显。笔记本电脑的风扇可能会更响。",
 " + 画像 {0}秒": " + 图像 {0} 秒",

onwx/manager.py

@@ -457,10 +457,15 @@ class Manager:
                     cfg[k] = min(max(cast(float(new[k])), lo), hi)
         except (TypeError, ValueError):
             return {"error": T("生成設定の値が不正です")}
+        if "api_max_tokens" in new:                            # [0.6.1] 0: up to the end of the context
+            try:
+                cfg["api_max_tokens"] = max(0, int(new["api_max_tokens"]))
+            except (TypeError, ValueError):
+                return {"error": T("生成設定の値が不正です")}
         if ctx != cfg["context"]:
             cfg["context_chosen"] = True                       # (the old 1024 default moves up; a choice stays)
         cfg["port"], cfg["context"] = port, ctx
-        server_keys = ("lan", "api_key", "port", "context", "think_default", "cpu_glue",
+        server_keys = ("lan", "api_key", "port", "context", "think_default", "cpu_glue", "api_max_tokens",
                        "api_sampling", "api_temperature", "api_top_p", "api_top_k", "api_presence")
         app.save_config(cfg)
         i18n.current(cfg)                                  # messages switch language right away

onwx/npu_pfx.py

@@ -279,8 +279,14 @@ class BigPrefill:
         self.CAP = min(caps)
         kr = max(int(json.load(open(os.path.join(self.attn_dirs(M)[-1], "case.json"))).get("kround", 8)) for M in self.MS)
         self.ALIGN = self.RA = 4 * kr                 # (a chunk starts at a multiple of ALIGN; RA: the cache read past it)
+        # (pfa5 needs the 256: a chunk at a 128-offset start reads its first rows wrong - KL 0.01-0.2 against the
+        # aligned run, 2026-10-15)
+        self.tcost = len(self.attn_dirs(max(self.MS))) == 2     # [0.6.1] (pfa5: Qwen3.5 9B / 4B - costs measured)
         self.cost = cfg.get("cost", {"1024": 2.5, "2048": 4.3})
         self.old_tps = float(cfg.get("old_tps", 95.0))
+        # [0.6.1] the costs grow with the cache extent T: a chunk's attention reads the cache once for the chunk
+        # (Max+ 395: a 1024 chunk 2.1-2.5 + 0.029 s per 1K of T), the old programs once a block (Bonsai.old_cost)
+        self.chunk_k = float(cfg.get("chunk_k", 0.028)) if self.tcost else 0.0
         self.D = D = dims_of(b)
         self.H, self.ND, self.NO = D["H"], D["ND"], D.get("NO", 1)
         self.bz = D.get("model") == "bonsai"
@@ -467,10 +473,13 @@ class BigPrefill:
             self.dev.sync(q["aux"][0], True)
             q["aux_T0"] = T0
 
-    def plan(self, n):
-        """how many of n tokens the next chunk takes (0: the old programs are faster for all n): the cheapest split by
-        the measured costs (xdna.json "pfx" "cost": s a chunk of M, "old_tps": the old programs' tokens / s)"""
-        cost = {int(k): v for k, v in self.cost.items()}
+    def plan(self, n, T=0):
+        """how many of n tokens at cache position T the next chunk takes (0: the old programs are faster for all n):
+        the cheapest split by the measured costs (xdna.json "pfx" "cost": s a chunk of M, "old_tps": the old programs'
+        tokens / s; both grow with T: "old_k", "chunk_k")"""
+        kT = T / 1024.0
+        cost = {int(k): v + self.chunk_k * int(k) / 1024.0 * kT for k, v in self.cost.items()}
+        old = (lambda r: self.b.old_cost(r, T)) if self.tcost else (lambda r: r / self.old_tps)
         memo = {}
 
         def f(r):
@@ -478,7 +487,7 @@ class BigPrefill:
                 return 0.0, 0
             if r in memo:
                 return memo[r]
-            c, take = r / self.old_tps, 0
+            c, take = old(r), 0
             for M in self.MS:
                 v = cost[M] + f(r - M)[0]
                 if v < c:

onwx/server.py

@@ -407,7 +407,12 @@ def gen_args(body):
         raise HTTPException(400, "n > 1 is not supported (one NPU pipeline)")
     stop = body.get("stop") or []
     mx = body.get("max_completion_tokens", body.get("max_tokens"))
-    mx = 512 if mx is None else _num({"max_tokens": mx}, "max_tokens", 512, int)
+    # [0.6.1] a request without max_tokens: the setting (ONWX_MAX_TOKENS; 0 = up to the end of the context - the
+    # engine stops there anyway); agents send none and write whole files in one answer
+    if mx is None:
+        mx = int(os.environ.get("ONWX_MAX_TOKENS", "8192") or 0) or (1 << 30)
+    else:
+        mx = _num({"max_tokens": mx}, "max_tokens", 512, int)
     if mx < 1:
         raise HTTPException(400, "max_tokens must be at least 1")
     think = (body.get("chat_template_kwargs") or {}).get("enable_thinking")

onwx/web/manage.html

@@ -240,6 +240,8 @@
       <div class="opt"><span class="k">API キー（任意）</span><input type="password" id="f-api_key" placeholder="未設定なら何を入れても通ります" style="width:260px"></div>
       <div class="opt"><span class="k">コンテキスト長</span><select id="f-context" style="width:110px"><option>1024</option><option>2048</option><option>3840</option><option>8192</option><option>16384</option><option>32768</option><option>65536</option><option>131072</option><option>262144</option></select> <span id="ctxnote" class="small muted"></span>
         <span class="desc">会話で使える最大の長さ（モデルの上限まで。長いほどメモリを使い、長い会話ほど遅くなります）。</span></div>
+      <div class="opt"><span class="k">API の最大出力トークン</span><select id="f-api_max_tokens" style="width:150px"><option value="2048">2048</option><option value="4096">4096</option><option value="8192">8192</option><option value="16384">16384</option><option value="32768">32768</option><option value="0">文脈の残りまで</option></select>
+        <span class="desc">アプリが上限を指定しないときの、1 回の回答の長さの上限です（エージェントがファイルを書くときなどに効きます）。大きくしても使用メモリは増えません。</span></div>
       <label class="swrow"><span class="switch"><input type="checkbox" id="f-lan"><i></i></span><span>LAN からの接続を許可する<span class="desc">他の PC から API を使えるようにします。API キーの設定をおすすめします。</span></span></label>
       <label class="swrow"><span class="switch"><input type="checkbox" id="f-cpu_glue"><i></i></span><span>CPU も使ってプロンプトの読み込みを速くする（消費電力が増えます）<span class="desc">長いプロンプトほど効きます。ノート PC ではファンの音が大きくなることがあります。</span></span></label>
     </div>
@@ -423,6 +425,7 @@
       $("f-lang").value = c.lang || "auto";
       for (const k of ["autostart_server", "login_start", "lan", "update_check"]) $("f-" + k).checked = c[k] !== false && !!(c[k] ?? true);
       $("f-cpu_glue").checked = !!c.cpu_glue;             // (off unless chosen)
+      $("f-api_max_tokens").value = String(c.api_max_tokens ?? 8192);
     }
   }
   function noteOf(m) {
@@ -535,7 +538,8 @@
     if (await ask(T("{0}\nを探す場所から外しますか？（フォルダやモデルは削除しません）", d))) post("folders", {remove: d});
   }
   async function saveSettings() {
-    const body = {port: $("f-port").value, context: $("f-context").value, api_key: $("f-api_key").value.trim(), lang: $("f-lang").value};
+    const body = {port: $("f-port").value, context: $("f-context").value, api_key: $("f-api_key").value.trim(), lang: $("f-lang").value,
+                  api_max_tokens: $("f-api_max_tokens").value};
     for (const k of ["autostart_server", "login_start", "lan", "update_check", "cpu_glue"]) body[k] = $("f-" + k).checked;
     const oldLang = S.config.lang || "auto";
     const r = await post("settings", body);

onwx/xdna.py

@@ -52,17 +52,39 @@ def is_xdna_model(d):
 
 # ---------------------------------------------------------------- devices
 PYXRT_DIRS = ["{XILINX_XRT}/python", "/opt/xilinx/xrt/python", "/usr/lib/python3/dist-packages",
-              "/usr/local/lib/python3/dist-packages", "/usr/lib/x86_64-linux-gnu/xrt/python", "/usr/local/xrt/python"]
+              "/usr/local/lib/python3/dist-packages", "/usr/lib/x86_64-linux-gnu/xrt/python", "/usr/local/xrt/python",
+              # [0.6.1] versioned Python folders and other XRT roots (distribution packages differ)
+              "/usr/lib/python3*/dist-packages", "/usr/lib/python3*/site-packages", "/usr/local/lib/python3*/dist-packages",
+              "/usr/local/lib/python3*/site-packages", "/usr/lib64/python3*/site-packages", "/opt/xilinx/xrt/lib/python*",
+              "/usr/lib/xrt/python", "/usr/share/xrt/python", "/opt/amd/xrt/python"]
+XRT_LIBS = ["{XILINX_XRT}/lib", "/opt/xilinx/xrt/lib", "/usr/lib/x86_64-linux-gnu", "/usr/lib64", "/usr/lib",
+            "/usr/local/lib", "/opt/amd/xrt/lib"]
+
+
+def xrt_libs():
+    """[0.6.1] where XRT's runtime library (libxrt_coreutil) is - XRT installed at all?"""
+    import glob
+    xrt = os.environ.get("XILINX_XRT", "/opt/xilinx/xrt")
+    out = []
+    for d in XRT_LIBS:
+        for f in sorted(glob.glob(os.path.join(d.format(XILINX_XRT=xrt), "libxrt_coreutil.so*"))):
+            if f not in out:
+                out.append(f)
+    return out
 
 
 def pyxrt_dirs():
-    """the folders that may hold XRT's Python bindings (its packages put them beside the library, off sys.path)"""
+    """the folders that may hold XRT's Python bindings (its packages put them beside the library, off sys.path):
+    the known ones, and <root>/python beside every XRT library found"""
+    import glob
     xrt = os.environ.get("XILINX_XRT", "/opt/xilinx/xrt")
+    cands = [d.format(XILINX_XRT=xrt) for d in PYXRT_DIRS]
+    cands += [os.path.join(os.path.dirname(os.path.dirname(f)), "python") for f in xrt_libs()]
     out = []
-    for d in PYXRT_DIRS:
-        d = d.format(XILINX_XRT=xrt)
-        if d not in out and os.path.isdir(d):
-            out.append(d)
+    for c in cands:
+        for d in (sorted(glob.glob(c)) if "*" in c else [c]):
+            if d not in out and os.path.isdir(d):
+                out.append(d)
     return out
 
 
@@ -109,10 +131,18 @@ def import_pyxrt():
             f"XRT's Python bindings (pyxrt) were found ({', '.join(found)}) but they are built for Python {want}, and "
             f"onwx runs on Python {here} ({sys.executable}). Install XRT built for Python {here} (AMD xdna-driver: build "
             f"XRT on this PC), or run onwx with Python {want}.")
+    libs = xrt_libs()
+    if not libs:                                       # [0.6.1] say which of the two it is
+        raise RuntimeError(
+            "XRT (AMD's NPU runtime) is not installed - the amdxdna kernel driver alone is not enough. Install XRT and "
+            "its NPU plugin from AMD's xdna-driver (https://github.com/amd/xdna-driver); that XRT has the Python "
+            "bindings onwx needs (/opt/xilinx/xrt/python/pyxrt*.so). Searched for libxrt_coreutil.so in: "
+            f"{', '.join(d.format(XILINX_XRT=os.environ.get('XILINX_XRT', '/opt/xilinx/xrt')) for d in XRT_LIBS)}.")
     raise RuntimeError(
-        "XRT's Python bindings (pyxrt) are missing: install the AMD NPU driver (amdxdna) and XRT with its Python "
-        f"bindings (usually /opt/xilinx/xrt/python), or set XILINX_XRT to the XRT folder. Searched: "
-        f"{', '.join(PYXRT_DIRS).format(XILINX_XRT=os.environ.get('XILINX_XRT', '/opt/xilinx/xrt'))}.")
+        f"XRT is installed ({', '.join(libs)}) but its Python bindings (pyxrt) were not found - distribution packages "
+        "may leave them out. Install XRT built from AMD's xdna-driver (https://github.com/amd/xdna-driver; it puts "
+        "them in /opt/xilinx/xrt/python), or set XILINX_XRT to an XRT folder with python/pyxrt*.so. Searched: "
+        f"{', '.join(pyxrt_dirs()) or ', '.join(PYXRT_DIRS)}.")
 
 
 PF128_CAP = int(os.environ.get("ONWX_PF128_CAP", "62208"))   # [0.5] the 128-token prompt programs' last working cache extent
@@ -438,9 +468,9 @@ class Bonsai:
         res = []
         keep = (lambda r: res.append(np.asarray(r, np.float32).reshape(-1, self.hid))) if rows else (lambda r: None)
         bp = (self._big() if embeds is None and mpos is None and self.rdelta == 0 and not self._big_off else None)
-        while bp is not None and len(toks) - i >= bp.MIN:
-            a = (-self.pos) % bp.ALIGN               # (a chunk starts at a multiple of ALIGN (32 / 256): the rest first)
-            m = bp.plan(len(toks) - i - a)           # (the cheapest split; a short tail: a padded chunk or the old
+        while bp is not None and len(toks) - i >= (32 if bp.tcost else bp.MIN):   # ([0.6.1] short ones too: deep
+            a = (-self.pos) % bp.ALIGN               # in a long cache a padded chunk beats the old programs; plan decides)
+            m = bp.plan(len(toks) - i - a, self.pos + a)   # (the cheapest split; a short tail: a padded chunk or the old
             if m == 0 or self.pos + a + bp.chunk_for(m) + max(32, bp.RA) > self.tmax + self.kv_slack:   # programs, by the measured costs)
                 break
             if self.pos + a + bp.chunk_for(m) > bp.CAP:   # (past the chunk programs' cache extent: the old programs)
@@ -476,7 +506,7 @@ class Bonsai:
                          None if mpos is None else mpos[:, i:i + k])
             keep(p128.rows)
             i += k
-        while self.pf is not None and len(toks) - i >= 2 and self.pos + self.pf.NT <= self.tmax:
+        while self.pf is not None and len(toks) - i >= max(2, self._few()) and self.pos + self.pf.NT <= self.tmax:
             if self.pair and self.pos % self.kq:      # (pair-major: a batch starts at an even position; q8: at a quad)
                 self.step(toks[i], want_logits=False, emb=e(i), mpos=mp(i))
                 if rows:
@@ -502,6 +532,43 @@ class Bonsai:
             return None
         return self.head(h)
 
+    def _old_k(self):
+        """[0.6.1] the old programs' costs at the cache position, s: (a 128-token block or None, a 32-token block, a
+        single step) - each reads the whole cache once (Max+ 395, s + s per 1K of cache: Qwen3.5 9B 0.85 + 0.125,
+        0.33 + 0.067, 0.083 + 0.00075; the 4B (no 128s) 0.25 + 0.067, 0.05 + 0.0008; xdna.json "old_cost" overrides);
+        None: not measured for this model (the old rules)"""
+        p128 = getattr(self, "pf128", None) is not None
+        c = getattr(self, "_oc", False)
+        if c is False:                                # (once)
+            c = self.man.get("old_cost")
+            if c is None and os.path.isdir(os.path.join(self.d, "plans", "pfx", "m1024", "pfa5r1")):
+                c = [0.85, 0.125, 0.33, 0.067, 0.083, 0.00075] if p128 else [None, None, 0.25, 0.067, 0.05, 0.0008]
+            self._oc = c
+        if c is None:
+            return None
+        kT = self.pos / 1024.0
+        return (c[0] + c[1] * kT if p128 and c[0] is not None else None, c[2] + c[3] * kT, c[4] + c[5] * kT)
+
+    def old_cost(self, r, T):
+        """[0.6.1] s for r tokens on the old programs from cache position T (BigPrefill.plan)"""
+        pos, self.pos = self.pos, T
+        try:
+            k = self._old_k()
+        finally:
+            self.pos = pos
+        if k is None:
+            return r / 150.0
+        c128, c32, st = k
+        t = 0.0
+        if c128 is not None:
+            t, r = (r // 128) * c128, r % 128
+        return t + (r // 32) * c32 + min(c32, (r % 32) * st)
+
+    def _few(self):
+        """[0.6.1] below how many tokens single steps beat one dispatch of the 32-token prompt program here"""
+        k = self._old_k()
+        return 2 if k is None else int(k[1] / k[2])
+
     def _big(self):
         """[0.4] the chunked NPU prefill (None: not in this package / disabled / failed to start)"""
         if self.bigpf is None and self._big_ok:
@@ -1120,6 +1187,12 @@ class XdnaModel:
             lg[self.banned] = -np.inf
         return lg[None]
 
+    def chunk_align(self):
+        """[0.6.1] where chunked prompt reading can start (a multiple of this) when its costs are measured (Qwen3.5 9B /
+        4B), else 0 (chat then keeps its checkpoints where the conversation ended)"""
+        bp = self.rt._big()
+        return bp.ALIGN if bp is not None and getattr(bp, "tcost", False) else 0
+
     def reset(self):
         self.rt.reset()
         self._mtp_reset()

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "onwx"
-version = "0.6.0"
+version = "0.6.1"
 description = "LLMs entirely on the AMD Ryzen AI NPU (XDNA2)"
 readme = "README.md"
 license = {text = "Apache-2.0"}