HiramKHackenbacker commited on
Commit
1d86775
·
verified ·
1 Parent(s): 81c2abf

docs: correct known-issue mechanism (shape-signature specialization cache, guided affected equally at depth)

Browse files
Files changed (1) hide show
  1. README.md +11 -8
README.md CHANGED
@@ -28,14 +28,17 @@ table as a *static graph input* rather than carrying it in the graph. That table
28
  `ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare
29
  input-arity error naming `ple_table`/`ple_scale`.
30
 
31
- > ⚠️ **Known issue — unguided generation.** On the current reference engine
32
- > implementation, free-form (unguided) decoding exhibits unbounded Metal memory growth:
33
- > approximately **80 MB of process `phys_footprint` per generated token**, on both the
34
- > sequential and pipelined engines, scaling with prompt length. Long generations exhaust
35
- > memory and the process is terminated. **Constrained/guided decoding is unaffected** and
36
- > plateaus at 8–10 GB. A fix is in progress. Until it lands, use these bundles for
37
- > guided/constrained generation only. A fixed token cap is not a reliable mitigation,
38
- > because the safe cap shrinks as the prompt grows.
 
 
 
39
 
40
  ## Provenance
41
 
 
28
  `ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare
29
  input-arity error naming `ple_table`/`ple_scale`.
30
 
31
+ > ⚠️ **Known issue — memory growth on long generations.** The Core AI runtime caches a
32
+ > graph specialization per input-shape signature, and this export makes the sequence
33
+ > length load-bearing: every generated token adds one token to `position_ids` and thus
34
+ > mints a new signature, retaining roughly **80 MB of GPU allocations per generated
35
+ > token** (both engines, guided and unguided alike, at equal context). Workloads whose
36
+ > prompt/output lengths repeat e.g. short constrained extraction rows stay bounded,
37
+ > because previously seen lengths replay from cache at no cost. Long free-form
38
+ > generations grow without bound until the process exits; no in-process mitigation
39
+ > exists (token caps only postpone it, and shape-length reuse is the only stable state).
40
+ > A re-export onto a shape-stable contract (fixed-capacity KV read with an explicit
41
+ > attention mask) is planned and will replace these bundles' decode graphs.
42
 
43
  ## Provenance
44