docs: correct known-issue mechanism (shape-signature specialization cache, guided affected equally at depth)
Browse files
README.md
CHANGED
|
@@ -28,14 +28,17 @@ table as a *static graph input* rather than carrying it in the graph. That table
|
|
| 28 |
`ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare
|
| 29 |
input-arity error naming `ple_table`/`ple_scale`.
|
| 30 |
|
| 31 |
-
> ⚠️ **Known issue —
|
| 32 |
-
>
|
| 33 |
-
>
|
| 34 |
-
>
|
| 35 |
-
>
|
| 36 |
-
>
|
| 37 |
-
>
|
| 38 |
-
>
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Provenance
|
| 41 |
|
|
|
|
| 28 |
`ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare
|
| 29 |
input-arity error naming `ple_table`/`ple_scale`.
|
| 30 |
|
| 31 |
+
> ⚠️ **Known issue — memory growth on long generations.** The Core AI runtime caches a
|
| 32 |
+
> graph specialization per input-shape signature, and this export makes the sequence
|
| 33 |
+
> length load-bearing: every generated token adds one token to `position_ids` and thus
|
| 34 |
+
> mints a new signature, retaining roughly **80 MB of GPU allocations per generated
|
| 35 |
+
> token** (both engines, guided and unguided alike, at equal context). Workloads whose
|
| 36 |
+
> prompt/output lengths repeat — e.g. short constrained extraction rows — stay bounded,
|
| 37 |
+
> because previously seen lengths replay from cache at no cost. Long free-form
|
| 38 |
+
> generations grow without bound until the process exits; no in-process mitigation
|
| 39 |
+
> exists (token caps only postpone it, and shape-length reuse is the only stable state).
|
| 40 |
+
> A re-export onto a shape-stable contract (fixed-capacity KV read with an explicit
|
| 41 |
+
> attention mask) is planned and will replace these bundles' decode graphs.
|
| 42 |
|
| 43 |
## Provenance
|
| 44 |
|