docs/features/rys_layer_duplication.md

# RYS runtime layer duplication (weight-shared "Repeat-Your-Self")

> Status: **DESIGN + PROTOTYPE** (2026-07-12). Parser/expander prototyped and
> validated; edit surface mapped; MVP scoped. No source landed yet.
>
> Child of [MASTER_PLAN.md](../MASTER_PLAN.md). Additive soft-fork feature —
> new load-time flag, per-arch surgical hooks in the graph loop + KV alloc.

## 1. What this is

David Noel Ng's **RYS** ("Repeat Your Self", [rys-ii][rys]) shows that
re-running a contiguous block of *middle* transformer layers — **no weight
changes, no training** — measurably improves a model (the layers duplicated are
the ones operating in the format-agnostic "reasoning" band; encoding/decoding
boundary layers must not be touched). For Qwen3.5-27B the Pareto-optimal blocks
are tiny: `(33,34)` (+1 layer) already captures most of the EQ gain, up through
`(26,34)` (+8) for the best absolute score.

The naive way to ship a RYS variant is a **physically merged GGUF** (mergekit
passthrough) — but that *doubles the duplicated layers' weight bytes* on disk
and in VRAM. dnhkng is instead working with TurboDerp on **pointer-based**
duplication in ExLlamaV3: the repeated layers **share the weight tensors** with
their originals, so *"no additional VRAM is consumed for the parameters
themselves — you only pay extra for the compute time and KV cache of the
additional forward passes."*

**This feature ports that pointer-shared idea to opencoti llamafile, driven
purely by a load-time flag.** No new GGUF, no re-quant, quant-agnostic:

```
llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 26,34   # RYS-XL, +8 layers
llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 33,34   # RYS-S,  +1 layer
```

Stock llama.cpp has **no** runtime layer-repeat; this is a genuine opencoti
differentiator and fits our "additive, load-time" ethos.

## 2. Mechanism: a layer execution plan (`eff → src`)

llama.cpp's per-architecture graph builder runs a residual loop
`for (il = 0; il < n_layer; ++il)` that (a) reads weights from
`model.layers[il]` and (b) uses `il` to index the KV cache, RoPE, and per-layer
hparams. We replace the identity iteration with a **plan**: a vector of *source*
layer indices to execute in order.

- **`layer_plan[eff] = src`** — for each *effective* position `eff`, which
  source layer's weights to run. Default (no flag) = identity `[0,1,…,n_layer-1]`.
- A RYS block `(i,j)` (half-open, matching the article) inserts source indices
  `i…j-1` a second time **right after** their first pass.
- **`n_layer_eff = layer_plan.size()`** — the *effective* layer count. This is
  what the KV cache, RoPE loop, and residual loop use. The GGUF's `n_layer`
  (the *source* weight count) is unchanged — weight loading is untouched.

Forward loop, conceptually:

```cpp
for (int eff = 0; eff < n_layer_eff; ++eff) {
    const int src = layer_plan[eff];
    const auto & layer = model.layers[src];   // WEIGHTS: shared by pointer, zero extra VRAM
    cur = build_norm(inpL, layer.attn_norm, …, eff);
    cur = build_layer_attn(inp, cur, inp_pos, …, /*kv_il=*/eff);   // KV/cache index: eff (distinct slot)
    …
    inpL = cur;
}
```

The duplicated pass reads the *same* weights but writes its *own* K/V at the
same token positions — exactly the RYS semantics.

### 2a. Prototype (validated)

The plan expander is prototyped and unit-tested against every config in the
article's Qwen3.5-27B Pareto table (reference:
`scratchpad/rys_plan.py`; the C++ will mirror it):

| spec       | blocks       | `n_eff` (from 64) | extra | article |
|------------|--------------|-------------------|-------|---------|
| `33,34`    | (33,34)      | 65                | +1    | +1 ✓   |
| `31,34`    | (31,34)      | 67                | +3    | +3 ✓   |
| `30,35`    | (30,35)      | 69                | +5    | +5 ✓   |
| `26,34`    | (26,34)      | 72                | +8    | +8 ✓   |
| `24,35`    | (24,35)      | 75                | +11   | +11 ✓  |
| `29,34`    | (29,34)      | 69                | +5    | +5 ✓   |

`(26,34)` expands to `…,24,25,[26,27,28,29,30,31,32,33],[26,27,28,29,30,31,32,33],34,35,…`
— the seam is the duplicated `26..33`. Overlapping blocks are rejected in the
MVP (disjoint-ascending only; the Pareto winners are all single blocks).

## 3. Parameter grammar

```
--repeat-layers <spec>
```
- `<spec>` = one or more blocks separated by `;`.
- Each block is `i,j` **or** `i-j` = the half-open range `[i,j)` of **source**
  transformer-layer indices to duplicate in place. `i,j` matches the article's
  `(i,j)` notation for copy-paste from the HF configs.
- Examples: `33,34` · `26,34` · `43,45;28,34` (compose two disjoint blocks).
- Validation (fail-fast at load): `0 ≤ i < j ≤ n_transformer_layers`; blocks
  disjoint; the MTP/`nextn_predict_layers` head is **never** duplicable.

Also plumbed through the opencoti TS adapter (`buildServerArgs`) as
`repeatLayers?: string` so opencode configs can set it.

## 4. Memory / cost model

- **Params: zero extra VRAM** — duplicated layers point at the same
  `model.layers[src]` tensors.
- **KV cache: grows by the extra passes** — `n_layer_eff/n_layer`. E.g. `(26,34)`
  on a 64-layer model = +8 → **+12.5%** KV. Modest; the article calls it out
  explicitly. (This is why RYS-S `(33,34)`, +1.5% KV, is the low-overhead pick.)
- **Compute: +one forward pass per duplicated layer** per token — same ratio as
  KV. The win is quality-per-token, paid in tokens/s.

## 5. What is provably unaffected

- **Weight loading** — untouched; `n_layer` (source) drives `create_tensor`.
- **RoPE / positions** — positions come from the batch, not the layer. A source
  layer run at two `eff` positions sees the same token positions; only its KV
  slot differs. No self-extend-style position hacking.
- **Off (no flag)** — `layer_plan` = identity ⇒ byte-identical to stock. This is
  the primary regression gate.

## 6. Composition with opencoti per-layer features

Everything opencoti indexes by layer must consume the **effective** index (or map
`eff→src` for source-keyed attributes). Scoped incrementally:

| feature | keyed by | MVP posture |
|---|---|---|
| plain f16 / scalar-quant KV | `eff` slot | **MVP** — works directly |
| RoPE / positions | batch pos | unaffected |
| iSWA (Gemma sliding/global per layer) | **src** (`is_swa(il)`) | Gemma phase — map `eff→src` |
| DCA (all-KV) | per-layer | later — gate off under `--repeat-layers` first, then make plan-aware |
| rolling-KV (`window_cells[il]`) | `eff` budget | later — size by `n_layer_eff` |
| PolyKV / sparse-attn | per-layer | later |
| MTP / NextN head | excluded | head never duplicated; spec path unchanged |

**Survey nuance (important):** DCA, rolling-KV (`window_cells[il]`), PolyKV, and
sparse-attn are all keyed through `map_layer_ids` / per-**physical**-layer
structs — so they **auto-inherit eff-correctness** the moment the `:927` alloc
loop (§8b#4) is eff-length. The residual risk is *not* the accessors; it's the
**sizing loops that still say `hparams.n_layer`** and would under-allocate for
eff slots: `n_layer_kv()`, `opencoti_compute_resident_window_cells`
(`kv-cache:249/:772`), and the sparse block-sel reserve `8*n_layer_kv`
(`kv-cache:879`). Plus DCA's `is_swa(il)` in `gemma4:217` needs `src`.

**MVP rule:** ship `--repeat-layers` composing only with plain/quant KV on the
dense arch first; **hard-error at load** if combined with DCA / rolling-KV /
sparse / MTP until each sizing loop is made eff-aware (explicit follow-ups). This
prevents silent under-allocation, which would corrupt rather than error.

> **STATUS UPDATE (RYS-12 / #667, 2026-07-13 — SUPERSEDES the MVP rule above):**
> the DCA / rolling-KV-residency / sparse-attn guards are **LIFTED**. The three
> residency sizing loops (`opencoti_compute_auto_gpu_heads_frac` /
> `_resident_window_cells` / `_auto_select_kv_tier`) iterate the EFFECTIVE plan
> (eff→src), the DCA build path maps `eff→src` for `is_swa`, and sparse-attn
> alloc lives inside the eff-iteration loop (authoritative comment:
> `src/llama-context.cpp` ~line 268). RYS now composes with DCA, rolling-KV
> window/spill, sparse-attn, quant-KV, and MTP (draft context runs the plain
> base stack — bug-2171 — target keeps RYS; lossless). RYS × rolling-KV window
> spill was live-validated 2026-07-14 on the qwen35moe hybrid (bug-2172 gate:
> 44 eff layers + 256/40704 window+tail, needle recovered, no assert). The
> remaining hard-errors are: unsupported archs (RYS-4) and malformed/boundary
> plans (parse validation); boundary-band plans get an advisory WARN.

## 7. Per-architecture scope

### Phase 1 — Qwen3.5 / Qwen3.6 **dense** (MVP target)
Uniform layers (no iSWA, uniform head counts/rope) ⇒ `eff→src` is trivial for
everything except KV-slot allocation. `nextn_predict_layers` (MTP head) excluded
from the plan. Target models: Qwen3.5-27B dense, Qwen3.6 dense variants.
Forward loop: `models/qwen35.cpp` (also covers the dense path), `models/qwen3.cpp`.

### Phase 2 — Gemma-4 **dense** 12B and 31B
Adds the iSWA wrinkle: alternating sliding-window / global layers, per-layer
query scaling / attn logit-softcap. A duplicated layer must inherit its **source**
layer's SWA type and scaling ⇒ per-layer hparam lookups map `eff→src`. The KV
cache's SWA/global split (`kv_local_layer`, the iSWA cache) must size by
`n_layer_eff` with the src-derived SWA pattern. Forward loop: `models/gemma4.cpp`.
Targets: `gemma-4-12B-it` dense, `gemma-4-31B-it` dense.

(Gemma-4 A4B is MoE — out of scope for the dense MVP; RYS on MoE is a separate
question about whether per-expert routing survives block re-traversal.)

## 8. Edit surface

> Line-anchored from the source survey (2026-07-12). All paths under
> `vendors/sources/llamafile/llama.cpp/`.

**Governing finding:** in this fork the `il` handed to `build_attn(...)` is used
for BOTH weight lookup (`model.layers[il]` in the model `.cpp`) AND KV-slot
lookup (`map_layer_ids[il]`). RYS **decouples** them: **weights + architecture
props (`is_swa`, rope-freq, `n_head`, `n_rot`) index by `src`; the KV-slot index
handed to `build_attn` is `eff`.** For Qwen **dense** this is a no-op distinction
outside KV (uniform layers, `is_swa` always false), so `build_attn(eff)` is
clean. For Gemma the iSWA sub-cache choice must be `src`-driven (see §8b).

### 8a. C-ABI plumbing caveat (load-bearing)
`struct llama_context_params` (`include/llama.h:336`) crosses the **C ABI** — it
cannot carry a `std::vector`. So:
- `include/llama.h:336` — add a **compact int spec** (`uint32_t repeat_start,
  repeat_end, repeat_count`, or a small CSV string).
- `src/llama-cparams.h:9` — add `std::vector<int> layer_plan;` to `llama_cparams`
  (plain C++ struct, vector OK). Insert near the opencoti extension fields (69-80).
- `src/llama-context.cpp:~133` — **expand** the compact spec into
  `cparams.layer_plan` here (where `hparams.n_layer` is known); default unset =
  identity `0..n_layer-1`; set `n_layer_eff = layer_plan.size()`.

### 8b. Minimal MVP edit set — Qwen dense (plain KV)
| # | site | change | invasiveness |
|---|------|--------|--------------|
| 1 | `llama-cparams.h:9` | add `layer_plan` vector | trivial |
| 2 | `include/llama.h:336` | compact int spec (C-ABI) | trivial |
| 3 | `llama-context.cpp:~133` | expand spec → `layer_plan`; `n_layer_eff` | trivial |
| 4 | `llama-kv-cache.cpp:927` (+ sizing `:165/:278/:451/:478`) | iterate `n_layer_eff`; size each eff slot from `src=plan[eff]` (`n_embd_k_gqa(src)`, `has_kv(src)`); `map_layer_ids[eff]=layers.size()` | moderate |
| 5 | `models/qwen3.cpp:76` (·qwen2.cpp:76·qwen3moe.cpp:86) | loop over `eff`; `src=plan[eff]`; `model.layers[il]→[src]`; `build_attn(…,eff)`; out-ids guard `eff==n_layer_eff-1` | moderate |
| 6 | `common/arg.cpp:~2780` | `add_opt("--repeat-layers")` (mirror `--override-kv` CSV) | trivial |
| 7 | `common/common.cpp:~1660` | map `common_params` range → `llama_context_params` spec | trivial |

`qwen35.cpp:169` — the plan is built over `[0, n_transformer_layers)` where
`n_transformer_layers = n_layer - nextn_predict_layers` (`:167`), so the **MTP
head is already excluded** by construction; never duplicate a `nextn` block.

**KV precedent:** `llama-kv-cache.cpp` already tolerates a non-identity
`map_layer_ids` — the `il_share` path (`:946-963`, bug-858 dual-ctx MTP) and
`il_reuse` (`:1265-1283`, SWA reuse). Both *share* physical slots; RYS is the
inverse (allocate **more, distinct** slots), so no new indirection type is
needed — just an **eff-length domain** on the `:927` alloc loop. The 30+ read
accessors (`get_k`/`cpy_k` via `mli_at_checked(map_layer_ids, il, …)`) are
unchanged as long as the graph passes `eff` and every `eff` has a map entry.

### 8c. Additional edits — Gemma-4 dense (iSWA), the invasive part
| site | change |
|------|--------|
| `models/gemma4.cpp:194` | loop rewrite splitting `src`/`eff`: `is_swa(src)`, `get_rope_freq_base/scale(cparams,src)` (`:201-202`), `n_rot(src)`, `layers[src].rope_freqs`, `f_attention_scale` by `src`; feed `eff` only to `build_attn` (`:269-283`) + KV routing |
| `gemma4.cpp:164` iSWA dual-cache | `build_attn_inp_kv_iswa()` + kv-cache filter/reuse must assign each **eff** slot to base-vs-swa sub-cache by `is_swa(plan[eff])` — pass a `plan`-aware filter into the iSWA cache ctor (the invasive step: it currently reads `swa_layers[il]`) |
| `llama-model.cpp:1929/1933/1937` | `get_rope_freq_base/scale/factors` only ever called with `src` (they branch on `is_swa`) |

### 8d. hparams helpers (index by `src`)
`llama-hparams.cpp`: `is_swa(il)` `:208`, `n_head(il)` `:30`, `n_head_kv(il)`
`:38`, `n_rot(il)` `:65`, `n_embd_k_gqa/v_gqa(il)` `:103/:109`, `has_kv(il)`
`:231`, `n_layer_kv()` `:250`, `swa_layers[]` (`hparams.h:141`). No signature
changes — the rule is "call these with `src` in the loop." RoPE positions come
from the batch (`llama-graph.cpp:161`), not the layer, so a `src` layer run at
two `eff` positions is correct with no position hacking.

## 9. Validation plan

Correctness is *mechanism* correctness (the plan re-traverses exactly the
intended layers) — RYS quality is the user's block choice, not ours to prove.

1. **Off = identity** — `--repeat-layers` absent ⇒ logits byte-identical to
   stock (teacher-forced, real_frac=0). Primary regression gate.
2. **Plan structural** — boot log dumps `n_layer_eff` + the `eff→src` plan;
   assert it matches the spec (e.g. `(26,34)` → 72 layers, seam at 34/42).
3. **Deterministic divergence** — `(33,34)` vs base produces a *deterministic,
   non-degenerate* logit change (greedy, same GGUF) — proves the duplicated pass
   actually runs and feeds forward, not a no-op.
4. **KV integrity** — niah retrieval stays 100 at a short ctx with a duplicated
   block (the extra KV slots are wired correctly; no cross-slot aliasing).
5. **Known-win spot check** — reproduce the article's *direction*: a small EQ/math
   probe should move in the reported direction for `(33,34)` / `(26,34)` on
   Qwen3.5-27B (sanity that we duplicated the reasoning band, not garbage).
6. **Gemma iSWA** — the src-derived SWA pattern is honored (duplicated global
   layer stays global); niah 100 + coherent decode.

Correctness always via logit-equiv / niah — never greedy-needle-as-proof.

## 10. Risks / caveats

- **KV growth is real** (§4) — surface it in the boot log; the TS adapter should
  account for it in VRAM budgeting.
- **iSWA mapping** — feared to be the main Gemma risk, but **verified correct**
  (see §10a): a middle SWA layer duplicates coherently, so the dual-cache eff→src
  wiring is sound. What actually breaks is *boundary* layers, independent of SWA.
- **MoE out of scope** for dense MVP — but the iSWA path was validated on **both**
  dense (31B, 60 layers) and MoE (A4B, 26B-A4B-128e) with identical behaviour.
- **Upstream syncability** — new flag + per-arch loop hooks tagged
  `opencoti-hook: rys-layer-dup` and registered in `UPSTREAM_SYNC.md`; the
  `layer_plan` indirection is small and localized to keep future bumps cheap.

## 10a. Boundary-layer fragility (bug-2164) — a MODEL property, not an engine bug

Duplicating the **first or last** transformer layers reliably produces incoherent
output. This is the well-documented franken-merge / passthrough self-merge
fragility (mergekit `passthrough`, SOLAR depth-upscaling all DROP the first/last
*m* layers), NOT an engine defect. The engine duplicates *any* layer faithfully.

**Decisive 2×2** (2026-07-13, native `google_gemma-4-26B-A4B-it-Q4_K_M`, 3090,
CHAT endpoint — the raw `/completion` greedy path is INVALID on this thinking
model: its answer lands in `reasoning_content`, so even RYS-off looks like garbage
there). Garbage is determined **entirely by boundary-ness, independent of
SWA/global**:

| | SWA layer | GLOBAL layer |
|---|---|---|
| **middle** (L4, L10 / L5, L17) | coherent ✓ | coherent ✓ |
| **boundary** (L0, L28 / L29) | garbage ✗ | garbage ✗ |

- Gemma-4 A4B coherent band ≈ **L4..L24**; L0–L3 (early) and L25–L29 (late) break.
- Qwen3-8B has a wider tolerant band — only its **last** layer (L35) breaks.
- Every attention output + final logit stays **finite** (no NaN) even under the
  garbage config → finite-but-wrong = model-level, not a numeric/cache corruption.

**Guidance: duplicate MIDDLE layers.** The engine emits a boot-time **advisory
WARNING** (`llama-context.cpp`, RYS engage block) whenever a plan duplicates a
layer within the first/last `max(3, n_transformer/8)`; it warns but does not block
(the exact usable band is model-dependent). Correctness gate = coherence smoke via
the CHAT endpoint (RYS intentionally changes the model, so logit-equivalence vs
baseline is the wrong bar). Historical note: an earlier pass mis-labelled this an
"SWA sub-cache engine bug" by testing only L0 (which is *both* SWA *and* first) on
the noisy raw-greedy path — see [[project_rys_swa_dup_garbage]].

## 10b. RYS × MTP self-spec — the draft context must NOT inherit the plan (bug-2171, 0124)

RYS is a **target-only** capability. With RYS active *and* self-speculation on
(`--spec-type draft-mtp`, Qwen NextN or the gemma4 assistant), the MTP draft
`llama_context` is built from `common_context_params_to_llama(params_base)`, so it
inherited the target's `params.repeat_layers` — and its `cparams.layer_plan` got
RYS-duplicated too. But the NextN head lives at the canonical `il = n_transformer`
(`n_layer − nextn_predict_layers`), which the RYS eff→src map never contains, so
draft-context init aborts with `cpy_k: map_layer_ids MISS il=n_transformer` →
`failed to create MTP context`. Fix (`0124`, `src/llama-context.cpp`): the RYS
plan-expansion block gates the spec on context type —
`rys_spec = (cparams.ctx_type == LLAMA_CONTEXT_TYPE_MTP) ? nullptr : params.repeat_layers`
(mirrored in `sched_reserve()`), so the **draft runs the plain base stack** while the
**target keeps RYS**. This is **lossless**: greedy self-spec verifies every drafted
token against the target, so the drafter's stack depth only moves acceptance/tps.
In self-spec the NextN head consumes the target's *already-RYS* hidden state via
shared memory — running the drafter plain is the intended composition, not a
limitation.

## 11. References

- [dnhkng, *LLM Neuroanatomy II*][rys] — the RYS-II study, Pareto table, pointer-based note ("The Models").
- [dnhkng/RYS on GitHub](https://github.com/dnhkng/RYS) — scanner, probes, model-builder config grammar.
- mergekit `passthrough` — the physical-merge convention `(i,j)` mirrors.

[rys]: https://dnhkng.github.io/posts/rys-ii/