fbjr commited on
Commit
63ab11f
·
verified ·
1 Parent(s): b1566dc

Refresh README: link GitHub repo; add L20 directional-hub / RAW=Turbo / refiner-diffuse / rebalance-lever findings; clarify solo probes are diagnostic (attention-mixed slot, not a clean layer)

Browse files
Files changed (1) hide show
  1. README.md +39 -25
README.md CHANGED
@@ -15,43 +15,57 @@ tags:
15
 
16
  # Krea 2 Projector Explorations
17
 
18
- Interpretability artifacts for Krea 2's text conditioning. Krea 2's text encoder is Qwen3-VL-4B, and the
19
- DiT doesn't use a single hidden state — it takes **12 selected encoder hidden-state layers**
20
- `[2,5,8,11,14,17,20,23,26,29,32,35]` (Krea's `select_layers`), concatenates them, and combines them with a
21
- **learned `Linear(12 → 1)` projector** (`txtfusion.projector`, `[1,12]`). That matrix is the model's own
22
- per-layer importance — identical in Raw and Turbo:
 
 
 
23
 
24
  | layer | L2 | L5 | L8 | L11 | L14 | L17 | L20 | L23 | L26 | L29 | L32 | L35 |
25
  |-------|----|----|----|----|----|----|----|----|----|----|----|----|
26
  | w | -0.05 | -0.16 | +0.37 | +0.50 | +0.71 | +0.39 | +0.40 | **-1.44** | -0.51 | -0.89 | -0.61 | +0.11 |
27
 
28
- The deeper layers (L23–L32) carry the most weight. (Terminology: ComfyUI calls these selected layers
29
- "taps".)
30
 
31
- ## Files
32
 
33
- - `krea2_projector_original_weights.safetensors` — the learned projector weights above, as a reference.
34
- - `solo/projector_solo_bNN_Lxx.safetensors` — 12 single-layer probes. Each isolates one selected
35
- hidden-state layer (zeroing the other 11) so you can see what that individual layer contributes to the
36
- image.
37
 
38
- Each `solo/` file is a `diffusion_model.txtfusion.projector.diff` patch (one `[1,12]` tensor, ~300 bytes),
39
- so it loads as a standard ComfyUI LoRA — no custom node needed.
 
 
 
 
 
 
40
 
41
- ## Usage (ComfyUI)
 
 
42
 
43
- Put a `solo/` file in `models/loras/`, add a **LoraLoaderModelOnly** node between your model loader and the
44
- sampler, and run it at **strength 1**. The image is then driven by a single selected layer's contribution.
45
- Sweep across the 12 to compare. Works with Krea 2 Raw or Turbo (the projector is identical).
 
 
 
 
 
46
 
47
- ## How they were made
 
 
 
48
 
49
- A `solo` patch keeps one selected layer and zeros the rest: `diff = original·(gain − 1)` with `gain = 1`
50
- for the kept layer and `0` elsewhere, so at strength 1 the projector reduces to that single layer.
51
- Deterministic edits of the model's existing weights — no training.
52
 
53
  ## License
54
 
55
- Derivatives of Krea 2 weights, released under the **Krea 2 Community License**
56
- ([license](https://huggingface.co/krea/Krea-2-Raw/blob/main/LICENSE.pdf) · base model
57
- [krea/Krea-2-Raw](https://huggingface.co/krea/Krea-2-Raw)).
 
15
 
16
  # Krea 2 Projector Explorations
17
 
18
+ Small, Krea-derived interpretability artifacts for Krea 2's text conditioning — the learned layer-mix
19
+ ("multilayer feature aggregation") plus single-layer probes. **Full toolkit, methods, figures, and write-up:
20
+ [github.com/fblissjr/krea-explorations](https://github.com/fblissjr/krea-explorations).**
21
+
22
+ Krea 2's text encoder is a frozen Qwen3-VL-4B; the DiT takes **12 selected encoder hidden-state layers**
23
+ `[2,5,8,11,14,17,20,23,26,29,32,35]` (`select_layers`), combines them with cross-layer attention, then a
24
+ learned `Linear(12 → 1)` projector (`txtfusion.projector`). That matrix is the model's own per-layer
25
+ weighting — **identical in Raw and Turbo** (cosine 1.0):
26
 
27
  | layer | L2 | L5 | L8 | L11 | L14 | L17 | L20 | L23 | L26 | L29 | L32 | L35 |
28
  |-------|----|----|----|----|----|----|----|----|----|----|----|----|
29
  | w | -0.05 | -0.16 | +0.37 | +0.50 | +0.71 | +0.39 | +0.40 | **-1.44** | -0.51 | -0.89 | -0.61 | +0.11 |
30
 
31
+ It combines **contrastively** ("mid plus, deep minus"), not as an average.
 
32
 
33
+ ## What we measured
34
 
35
+ These are characterizations of an open model's learned behavior (not architecture — the architecture is
36
+ public); most are low-effort to reproduce. Full method + confidence levels in the GitHub repo.
 
 
37
 
38
+ - **L20 is a learned *directional* attention hub.** In the cross-layer attention, ~91–95% of content tokens
39
+ route to layer 20 — content-driven (not a padding artifact) and a *directional* effect (not a magnitude
40
+ sink). Holds across 5 prompts and on **both Raw and Turbo**. The token-side "refiner" blocks, by contrast,
41
+ are diffuse (no hub).
42
+ - **The projector-rebalance lever is a detail/intensity knob, not an attribute gate.** Benign attributes
43
+ (expression, "wet", blush) come through the aggregation and render with or without rebalancing; boosting
44
+ the deep layers mainly shifts detail / contrast / intensity — consistent with the deep layers carrying
45
+ fine detail.
46
 
47
+ Per-layer reweighting of Krea 2's conditioning was introduced by
48
+ [nova452/ComfyUI-ConditioningKrea2Rebalance](https://github.com/nova452/ComfyUI-ConditioningKrea2Rebalance)
49
+ and refined by [huwhitememes/comfyui-krea2-conditioning](https://github.com/huwhitememes/comfyui-krea2-conditioning).
50
 
51
+ ## Files
52
+
53
+ - `krea2_projector_original_weights.safetensors` — a **reference copy** of the 12 learned projector weights
54
+ above (the `[1,12]` tensor itself). Read-only reference, not a LoRA to apply.
55
+ - `solo/projector_solo_bNN_Lxx.safetensors` — 12 **diagnostic probes**. Each is a projector `.diff` that, at
56
+ strength 1, keeps one of the projector's 12 inputs and zeroes the other 11, so the DiT conditions on a
57
+ single slot — useful to *see* what that slot contributes (deep slots render coherent images, shallow are
58
+ noise, L14 carries text/structure, L35 alone is unusable).
59
 
60
+ **Important:** the projector's 12 inputs are the **attention-mixed slots** (output of the 2 layerwise blocks),
61
+ not pristine encoder layers — and because the cross-layer attention routes through L20, every slot already
62
+ carries L20 content. So a "solo Lx" isolates the slot *indexed by* layer x, not a clean layer x. These are
63
+ **interpretability probes, not generation LoRAs** (keeping one input by design gives a partial/degraded image).
64
 
65
+ Each `solo/` file is a `diffusion_model.txtfusion.projector.diff` patch (one `[1,12]` tensor, ~300 bytes),
66
+ loadable via the stock `LoraLoaderModelOnly` — no custom node. (ComfyUI calls the selected layers "taps".)
 
67
 
68
  ## License
69
 
70
+ These artifacts derive from Krea 2 and are covered by the **Krea 2 Community License** (see the base model
71
+ linked above).