Update README.md
Browse files# 🛠️ Gemma 4 31B SwiGLU Fusion Toolkit
This toolkit provides a robust pipeline for generating and injecting custom, functionally correct MLP weights into a Gemma 4 31B safetensors model. It is specifically designed to handle Gemma 4's unique architectural features—namely its interleaved sliding-window attention (SWA) and global full-context layers—while preserving model stability through advanced alignment and fusion techniques.
## ✨ Features
* **Targeted Architecture:** Explicitly accounts for Gemma 4's interleaved attention structure (5 SWA + 1 Global per period).
* **Double-Wide MLP Support:** Automatically detects and generates intermediate sizes for global full-context layers.
* **Sign-Symmetric Alignment:** Generates weights using paired frame vectors ($+gc+gc$ and $-gc-gc$) to ensure the net activation remains zero-centered, preventing distribution shifts.
* **Shape-Contoured Fusion (SCF):** Instead of naive addition, the fusion process smoothly merges synthetic weights into the model's existing learned contours, minimizing destabilization.
* **Controlled Adjustment:** Implements dynamic scaling ($\alpha$) and clamping ($\gamma$-cap) to control the magnitude of weight updates.
* **Selective Fusion:** Allows users to target specific layers for modification.
## ⚙️ Pipeline Overview
The toolkit operates in two distinct stages (Once you have generated the source Neuron and derivative Neurons):
### 1. The Generator (`weights.py`)
This script creates synthetic, functionally correct MLP projections (`gate_proj`, `up_proj`, `down_proj`) based on reference source neurons. It maps piece-wise linear behaviors into the non-linear SwiGLU block without introducing mean-shift destabilization.
* **Key Function:** Generates balanced, sign-symmetric delta weights.
* **Output:** Saves the generated delta weights to an output directory.
### 2. The Fuser (`applyweights.py`)
This script applies the safetensor delta files generated by `weights.py` to your existing Gemma 4 31B model using Shape-Contoured Fusion (SCF).
* **`down_proj` (Contoured Multiplicative Delta):** The update is scaled by the existing weight profile and variance-normalized to ensure smooth integration.
$$\text{W}_{\text{down}} = \text{W}_{\text{down}} + (\alpha_{\text{dynamic}} \cdot \Delta_{\text{down}} \cdot \text{W}_{\text{down}})$$
* **`gate_proj` (Multiplicative Gamma Scaling):** Applies a clamped fractional adjustment to prevent unbounded activation spikes.
$$\text{W}_{\text{gate}} = \text{W}_{\text{gate}} \cdot (1 + \text{clamp}(\Delta_{\text{gate}}, -\gamma, \gamma))$$
* **`up_proj` (Untouched):** The linear value path is intentionally skipped during fusion.
## 🚀 Usage
### Stage 1: Generating Delta Weights
Use `weights.py` to generate the synthetic delta weights.
```bash
python weights.py \
--base-model /path/to/original_gemma_model \
--output-dir generated_weights \
--seed 42
```
### Stage 2: Applying Weights (Fusion)
Use `applyweights.py` to merge the generated weights into your target model.
```bash
python applyweights.py \
--model /path/to/original_gemma_model \
--weights generated_weights \
--output /path/to/fused_gemma_model \
--alpha 0.02 \
--gamma-cap 0.05
```
## 💡 Helpful Flags
| Flag | Description | Default |
| :--- | :--- | :--- |
| `--dry-run` | Simulates the fusion process, identifying missing keys, calculating partial coverage matrices, and verifying sharded indexing without writing files. | N/A |
| `--layers` | A space-separated list of specific layer indices to fuse (e.g., `--layers 5 11 17`), skipping the rest. | None |
| `--alpha` | Controls the variance scale multiplier for the `down_proj` update. | `0.02` |
| `--gamma-cap` | Sets the maximum fractional adjustment allowed for the `gate_proj`. | `0.05` |