onw 0.10.5: the compile cache prunes itself after a load; attention graphs in onw blob store

CHANGELOG.md

@@ -5,6 +5,15 @@
 日付は日本時間です。各版は Hugging Face の `ryugyosoft/onw` にタグ（`v0.4.2` 以降）付きで残っています。
 onw ウィンドウは、新しい版があるとこのファイルから変更点を表示します（`## v…` の見出しで区切っています）。
 
+## v0.10.5 (2026-09-29)
+- **コンパイルキャッシュの自動掃除**。モデルを読み込んだあと、その読み込みで使わなかったコンパイル結果（別の経路・
+  別のブロックサイズ・以前の版の切り分け方のもの）を `npu_cache` から消します。NPUW を使わない機種では OpenVINO 側の
+  キャッシュ（コンパイル確認の子プロセスや旧版が残していた複製）も空にします。手元では Gemma 4 の 32.6 GB → 8.1 GB、
+  LFM2-8B の 2.0 GB → 0.9 GB。`ONW_CACHE_PRUNE=0` で止められます。
+- アテンションのグラフも onw 自身の blob 置き場（`npu_cache/onw_blobs/attn_*`）に保存するように（OpenVINO のキャッシュを
+  onw のもので占めないため）。コンパイル確認の子プロセスも、onw の blob に書き出すときは OpenVINO のキャッシュに複製を
+  残しません。
+
 ## v0.10.4 (2026-09-29)
 - **Lunar Lake：0.10.3 で NPUW のコンパイルは通るのに読み込みが終わらない（最初の推論でプロセスが落ちる）問題を修正**。
   NPUW の重みバンクはコンパイル後もモデルの .bin を参照し続けるのに、読み込み後に解放していました。.bin はメモリに

CHANGELOG_en.md

@@ -4,6 +4,14 @@
 
 Dates are JST. Every version is on Hugging Face `ryugyosoft/onw`, tagged from `v0.4.2` on.
 
+## v0.10.5 (2026-09-29)
+- **The compile cache cleans itself.** After a model is loaded, the compiled files that load did not use (other
+  routes, other block sizes, earlier versions of the splitting) are removed from `npu_cache`; on machines without NPUW
+  OpenVINO's own cache (duplicates left by the compile probes and older versions) is emptied too. Here: Gemma 4
+  32.6 -> 8.1 GB, LFM2-8B 2.0 -> 0.9 GB. `ONW_CACHE_PRUNE=0` turns it off.
+- The attention graphs go to onw's own blob store as well (`npu_cache/onw_blobs/attn_*`), and a compile probe that
+  exports to onw's store no longer leaves a copy in OpenVINO's cache.
+
 ## v0.10.4 (2026-09-29)
 - **Lunar Lake: with 0.10.3 the NPUW compile finished but the load never did (the process died at the first
   inference).** NPUW's weights bank keeps reading the model's .bin after compiling, and onw released it after loading.

TECHNICAL.md

@@ -81,6 +81,9 @@ NPU は形の決まったグラフしか扱えませんが、MoE はトークン
 で拒否するので、`NPUW_ONLINE_PIPELINE=REP`（繰り返しブロック。部品の中の Sliding 層を折り畳む）でコンパイルし、
 だめなら NONE（1 サブグラフ）、通常コンパイルの順に落ちます。LM ヘッドは NONE。NPUW の重みバンクはコンパイル後も
 モデルの .bin を参照するので、.bin は mmap で持ってモデルと同じ寿命にしています（0.10.4）。
+読み込みが終わると、その読み込みで取り込まなかった `onw_blobs/seg*` を消し、NPUW を使わない機種では OpenVINO 側の
+キャッシュ（`npu_cache/*.blob`。onw の blob は全部自前の置き場にあるので不要）も空にします（0.10.5、`ONW_CACHE_PRUNE=0`
+で無効）。アテンションのグラフと画像エンコーダーは必要になったときにコンパイルするので、掃除の対象外です。
 
 ## Gated DeltaNet（ハイブリッドモデル）を NPU で動かす
 

TECHNICAL_en.md

@@ -80,6 +80,9 @@ NPUW's default online partitioner (and REG / COMPUTE / SPATIAL) rejects the spli
 so they compile with `NPUW_ONLINE_PIPELINE=REP` (repeated blocks: the sliding layers inside a part get folded), falling
 back to NONE (one subgraph) and then a plain compile. The LM head goes NONE. NPUW's weights bank keeps reading the
 model's .bin after compiling, so the .bin is a file-backed mapping that lives as long as the model (0.10.4).
+After a load, the `onw_blobs/seg*` files it did not import are removed, and on machines without NPUW so is OpenVINO's
+own cache (`npu_cache/*.blob`: every blob onw needs is in its own store there) - 0.10.5, `ONW_CACHE_PRUNE=0` disables
+it. The attention graphs and the vision encoder compile on demand and are left alone.
 
 ## Gated DeltaNet (hybrid models) on the NPU
 

onw/__init__.py

@@ -2,4 +2,4 @@
 
 A small engine that runs LLMs entirely on the Intel NPU: static segment graphs + host-managed state.
 """
-__version__ = "0.10.4"
+__version__ = "0.10.5"

onw/kvcache.py

@@ -11,6 +11,7 @@ Layout (measured fastest on NPU 3720): per block one K tensor [kv, C, D] and one
 INT8 in NPU host memory (bound zero-copy); the per-token scales sit outside the matmuls (K's on the scores, V's on the
 probabilities), so each head is a plain 2-D matmul against "weights".
 """
+import io, os
 import numpy as np
 import openvino as ov
 import openvino.opset13 as ops
@@ -77,10 +78,11 @@ class _Layer:
 
 
 class PagedKV:
-    def __init__(self, core, device, layers, capacity, config=None, bits=8):
+    def __init__(self, core, device, layers, capacity, config=None, bits=8, blobs=None):
         """layers: {name: dict(kv, D, Dv, scale, G, owner)} - owner: the layer whose cache it reads (shared KV).
-        bits: 8 (INT8 with per-token scales) or 16 (f16, scales of 1)."""
-        self.core, self.device, self.config = core, device, config or {}
+        bits: 8 (INT8 with per-token scales) or 16 (f16, scales of 1). blobs: (cache dir, machine key) to keep the
+        compiled attention graphs in onw's own blob store (npu_cache/onw_blobs/attn_*) instead of OpenVINO's cache."""
+        self.core, self.device, self.config, self.blobs = core, device, config or {}, blobs
         self.kt = ov.Type.f16 if bits == 16 else ov.Type.i8
         self.capacity = capacity
         self.ctx = core.get_default_context(device) if device == "NPU" else None
@@ -146,7 +148,27 @@ class PagedKV:
         key = (R, C, L.kv, L.D, L.Dv, L.scale, str(self.kt))
         if key not in self.graphs:
             m = attention_graph(R, C, L.kv, L.D, L.Dv, L.scale, self.kt)
-            self.graphs[key] = self.core.compile_model(m, self.device, self.config).create_infer_request()
+            if self.blobs and self.device == "NPU":
+                from .runtime import blob_path, import_blob
+                bp = blob_path(self.blobs[0], m.get_friendly_name() + f"_Dv{L.Dv}", [self.blobs[1], *key])
+                cfg = {k: v for k, v in self.config.items() if k != "CACHE_DIR"}
+                cm = None
+                if os.path.exists(bp):
+                    try:
+                        cm = import_blob(self.core, bp, self.device, cfg)
+                    except RuntimeError:
+                        os.remove(bp)
+                if cm is None:
+                    cm = self.core.compile_model(m, self.device, cfg)
+                    os.makedirs(os.path.dirname(bp), exist_ok=True)
+                    buf = io.BytesIO()
+                    cm.export_model(buf)
+                    with open(bp + ".tmp", "wb") as f:
+                        f.write(buf.getbuffer())
+                    os.replace(bp + ".tmp", bp)
+                self.graphs[key] = cm.create_infer_request()
+            else:
+                self.graphs[key] = self.core.compile_model(m, self.device, self.config).create_infer_request()
         return self.graphs[key]
 
     def attend(self, name, q, pos, S, n, window=None):

onw/runtime.py

@@ -140,6 +140,38 @@ def blob_path(cache_dir, name, parts):
     return os.path.join(cache_dir, "onw_blobs", f"{os.path.splitext(os.path.basename(name))[0]}-{h.hexdigest()[:16]}.blob")
 
 
+USED_BLOBS = set()                                    # onw_blobs files this process imported or wrote (see prune_cache)
+
+
+def prune_cache(cache_dir, plain_only):
+    """Drop compiled files a load did not use: onw_blobs/seg* not in USED_BLOBS (other routes, other block sizes,
+    earlier versions of the splitting), and - when every route is plain, so nothing of ours lives in OpenVINO's own
+    cache - the *.blob entries in the cache root (compile probes and older versions left them). Attention graphs and
+    the vision encoder (compiled later, on demand) are keyed by the machine and left alone."""
+    gone, n, busy = 0, 0, 0
+    stale = []
+    try:
+        blobs = os.path.join(cache_dir, "onw_blobs")
+        stale += [os.path.join(blobs, f) for f in os.listdir(blobs) if f.startswith("seg") or f.endswith(".tmp")]
+        if plain_only:
+            stale += [os.path.join(cache_dir, f) for f in os.listdir(cache_dir) if f.endswith(".blob")]
+    except OSError:
+        return
+    for p in stale:
+        if p in USED_BLOBS or not os.path.isfile(p):
+            continue
+        try:
+            size = os.path.getsize(p)
+            os.chmod(p, 0o666)                         # OpenVINO writes its cache entries read-only
+            os.remove(p)
+            gone, n = gone + size, n + 1
+        except OSError:                                # mapped by another process (a server still running on it)
+            busy += 1
+    if n or busy:
+        print(f"[cache] removed {n} unused compiled files ({gone / 2**30:.1f} GB)"
+              + (f", {busy} in use by another process" if busy else ""), flush=True)
+
+
 def import_blob(core, path, device, cfg):
     """Import a compiled blob we wrote ourselves. Unlike a CACHE_DIR hit - which OpenVINO keeps in memory next to the
     driver's own copy, so it could be exported again - an imported blob is released once the driver has loaded it:
@@ -369,10 +401,15 @@ class SegmentedModel:
         print(f"[segments] compiled {sum(len(v) for v in self.segs.values())} in {time.time()-t0:.1f}s", flush=True)
         self.kv = None
         if kv_dims:
+            cache = (config or {}).get("CACHE_DIR")
+            own = device == "NPU" and cache and os.environ.get("ONW_OWN_BLOBS") != "0"
             self.kv = PagedKV(core, os.environ.get("ONW_KV_DEVICE", device), kv_dims, self.T,
                               {k: v for k, v in (config or {}).items() if k == "CACHE_DIR"},
-                              bits=int(os.environ.get("ONW_KV_BITS", "8")))
+                              bits=int(os.environ.get("ONW_KV_BITS", "8")), blobs=(cache, tried.key) if own else None)
             print(f"[kv] paged INT8 cache for {len(kv_dims)} attention layers, up to {self.T} tokens", flush=True)
+        if device == "NPU" and (config or {}).get("CACHE_DIR") and os.environ.get("ONW_CACHE_PRUNE") != "0":
+            # everything this load did not touch is stale (or belongs to a route / version no longer taken)
+            prune_cache(config["CACHE_DIR"], plain_only=not npuw and os.environ.get("ONW_OWN_BLOBS") != "0")
         self._inputs = {n for lst in self.segs.values() for _, _, names in lst for n in names}
         self.state, owner = {}, {}
         for req, m, names in self.segs[min(self.segs)]:
@@ -432,6 +469,7 @@ class SegmentedModel:
             if own and os.path.exists(bp):
                 try:
                     req = import_blob(core, bp, device, nocache).create_infer_request()
+                    USED_BLOBS.add(bp)
                     tried.set(name, how)
                     return req
                 except RuntimeError as e:
@@ -441,11 +479,13 @@ class SegmentedModel:
             if (how != known or own) and device == "NPU":   # compile out of process first (and, own, export there)
                 tried.set(name, "trying:" + how)       # a crash of this process too leaves a mark to skip it next time
                 tried.save()
-                # the child keeps CACHE_DIR: an existing OpenVINO cache entry makes this fast (it only exports)
-                r = _probe(d, name, cfg, npuw, T, export=bp)
+                # exporting to our store: the child compiles without CACHE_DIR (no second copy in OpenVINO's cache);
+                # otherwise it keeps CACHE_DIR and the parent's compile below is a cache hit
+                r = _probe(d, name, nocache if own else cfg, npuw, T, export=bp)
                 if r is True and own and os.path.exists(bp):
                     try:
                         req = import_blob(core, bp, device, nocache).create_infer_request()
+                        USED_BLOBS.add(bp)
                         if how != known:
                             print(f"[compile] {name}: {how}", flush=True)
                         tried.set(name, how)
@@ -460,7 +500,7 @@ class SegmentedModel:
                     err = RuntimeError(f"{name}: {how}: {r}")
                     continue
             try:
-                req = core.compile_model(model, device, cfg).create_infer_request()
+                req = core.compile_model(model, device, nocache if own else cfg).create_infer_request()
                 if how != known:
                     print(f"[compile] {name}: {how}", flush=True)
                 tried.set(name, how)
@@ -512,6 +552,7 @@ class SegmentedModel:
             if own and os.path.exists(bp):
                 try:
                     req = import_blob(core, bp, device, nocache).create_infer_request()
+                    USED_BLOBS.add(bp)
                     tried.set(pname, how)
                     return req
                 except RuntimeError:
@@ -526,6 +567,7 @@ class SegmentedModel:
                     with open(bp + ".tmp", "wb") as f:
                         f.write(buf.getbuffer())
                     os.replace(bp + ".tmp", bp)
+                    USED_BLOBS.add(bp)
                 if how != known:
                     print(f"[compile] {pname}: {how}", flush=True)
                 tried.set(pname, how)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "onw"
-version = "0.10.4"
+version = "0.10.5"
 description = "Ore no NPU ga konna ni ugoku wake nai - LLMs entirely on the Intel NPU"
 readme = "README.md"
 license = {text = "Apache-2.0"}