Update README.md

README.md

@@ -1,193 +1,195 @@
----
-license: mit
-language:
-- en
-- zh
-tags:
-- transformer
-- interpretability
-- mechanistic-interpretability
-- language-model
-- signal-decomposition
-- sparse-representations
-- pytorch
-datasets:
-- openwebtext
-pipeline_tag: text-generation
----
-
-# reFlow
-
-[ [中文](README_CN.md) | English ]
-
-**A Metal Soul In My Hand** — A feature-decoupled Transformer architecture with native interpretability.
-
-reFlow factorizes the embedding matrix $E \in \mathbb{R}^{V \times d}$ into a **Recipe Matrix** $W_{recipe} \in \mathbb{R}^{V \times S}$ and a **Signal Basis Matrix** $W_{basis} \in \mathbb{R}^{S \times d}$, forcing the model to maintain a set of continuous, low-redundancy signal bases in latent space. The same factored product $W_{recipe} \times W_{basis}$ serves as both the input embedding and the output projection, forming an end-to-end signal-manifold computation loop without a separate LM head.
-
-## Online Demo
-
-**Try reFlow in your browser:**
-- [HuggingFace Space](https://huggingface.co/spaces/reuAC/reFlow) (Global Access)
-- [ModelScope Studio](https://www.modelscope.cn/studios/recuAC/reFlow) (China Access)
-
-## Key Results
-
-**Convergence.** At matched depth and scale (36 layers, ~515M parameters), reFlow-1-Big achieves a validation loss within ~1% of GPT-2-New (514M). Three scale points — Small (46.47M), reFlow-1 (463.67M), Big (515.06M) — confirm strict scaling law compliance (val loss: 3.55 → 3.01 → 2.92).
-
-**Emergent Interpretable Structure** (pure language modeling objective, no auxiliary loss):
-- Recipe-space semantic algebra: king + woman − man → queen (rank #1), 3/3 tests passed
-- Natural sparsity: each token activates ~11% of signals (mean 117/1024), Gini coefficient 0.085
-- Causal traceability: single-signal ablation collapses target probability from 8.31% to 0.03%
-- Information crystallization boundary: semantic interventions are effective at L0–L12 but inert beyond L18
-- Hard sparsity (Top-64) systematically destroys recipe-space semantic structure (algebra 3/3 → 0/3, silhouette +0.11 → −0.02)
-
-> **Paper**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf) — Theoretical derivation, 12 interpretability experiments, and scaling/ablation analysis.
->
-> **Pretrained Weights**: [HuggingFace](https://huggingface.co/reuAC/reFlow)
-
-## Project Structure
-
-```
-reFlow/
-├── train.py              # Training script (single GPU / DDP)
-├── sample.py             # Text generation from trained models
-├── experiment.py          # 12-experiment interpretability suite (Chinese)
-├── experiment_en.py       # 12-experiment interpretability suite (English)
-├── check.py              # Checkpoint parameter inspector
-├── bench.py              # Performance benchmarking
-├── models/
-│   ├── gpt2.py           # Standard GPT-2 baseline
-│   ├── gpt2-new.py       # Modernized GPT-2 (RoPE + SwiGLU + RMSNorm)
-│   ├── reflow.py         # reFlow base architecture
-│   ├── reflow-topk.py    # reFlow with ReLU + Top-K hard sparsity
-│   └── reflow-lite.py    # reFlow with GQA + reduced MLP
-├── config/               # Training / sampling / eval configurations
-├── data/
-│   ├── openwebtext/      # OpenWebText dataset preparation
-│   └── sft-lima/         # LIMA SFT dataset preparation
-└── out/                  # Checkpoints and experiment reports
-```
-
-## Installation
-
-### Prerequisites
-
-- Python 3.10+
-- CUDA-compatible GPU (tested on Tesla T4 x4)
-
-### 1. PyTorch (CUDA 12.8)
-
-```bash
-pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
-```
-
-> Adjust the CUDA version in the URL to match your driver. See [PyTorch Get Started](https://pytorch.org/get-started/locally/).
-
-### 2. Core Dependencies
-
-```bash
-pip install datasets tiktoken wandb tqdm
-```
-
-### 3. Experiment Suite Dependencies
-
-The interpretability experiments (`experiment.py`) require additional packages:
-
-```bash
-pip install numpy matplotlib seaborn scikit-learn scipy adjustText
-```
-
-### Quick Install (All-in-One)
-
-```bash
-pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
-pip install datasets tiktoken wandb tqdm numpy matplotlib seaborn scikit-learn scipy adjustText
-```
-
-## Data Preparation
-
-### OpenWebText
-
-```bash
-python data/openwebtext/prepare.py
-```
-
-This downloads the OpenWebText corpus (~54 GB) and tokenizes it with the GPT-2 BPE tokenizer. Output: `data/openwebtext/train.bin` (~17 GB, ~9B tokens) and `val.bin`.
-
-## Training
-
-All configurations are in `config/`. No CLI overrides — all hyperparameters must be set in the config file.
-
-### Single GPU
-
-```bash
-python train.py config/train_reflow_1.py
-```
-
-### Multi-GPU (DDP)
-
-```bash
-torchrun --standalone --nproc_per_node=4 train.py config/train_reflow_1.py
-```
-
-### Available Training Configs
-
-| Config | Architecture | Layers | Params | Notes |
-|--------|-------------|--------|--------|-------|
-| `train_gpt2.py` | GPT-2 | 36 | 505.62M | Standard baseline |
-| `train_gpt2_new.py` | GPT-2-New | 36 | 514.01M | + RoPE, SwiGLU, RMSNorm |
-| `train_reflow_1.py` | reFlow | 32 | 463.67M | Base reFlow, constant lr |
-| `train_reflow_1_big.py` | reFlow | 36 | 515.06M | lr decay, for interpretability |
-| `train_reflow_1_topk_big.py` | reFlow-TopK | 36 | 515.06M | + ReLU + Top-64 sparsity |
-| `train_reflow_1_lite.py` | reFlow-Lite | 32 | 413.34M | + GQA, reduced MLP |
-| `train_reflow_1_small.py` | reFlow | 6 | 46.47M | Small-scale validation |
-
-### Resume Training
-
-Append `_resume` to the config name (e.g., `train_reflow_1_big_resume.py`).
-
-## Text Generation
-
-```bash
-python sample.py config/sample_reflow_1.py
-```
-
-Edit the config file to change the prompt, temperature, top-k, etc.
-
-## Interpretability Experiments
-
-The experiment suite runs 12 analyses on a trained reFlow model. Both Chinese and English versions are available:
-
-```bash
-python experiment_en.py config/train_reflow_1_big.py   # English
-python experiment.py config/train_reflow_1_big.py      # Chinese
-```
-
-An interactive menu will appear:
-
-| # | Experiment | Group |
-|---|-----------|-------|
-| 1 | Recipe Atlas — recipe-space nearest neighbors | A. Signal Identity |
-| 2 | Sparsity Profile — activation sparsity analysis | A. Signal Identity |
-| 3 | Basis Geometry — singular value & effective rank | A. Signal Identity |
-| 4 | Semantic Galaxy — PCA clustering visualization | B. Semantic Properties |
-| 5 | Semantic Algebra — vector arithmetic (king − man + woman = queen) | B. Semantic Properties |
-| 6 | Typo Resilience — robustness to spelling errors | B. Semantic Properties |
-| 7 | Layer Evolution — per-layer probability crystallization | C. Mechanistic Analysis |
-| 8 | Signal Flow — signal activation heatmaps across layers | C. Mechanistic Analysis |
-| 9 | Causal Ablation — progressive signal knockout curves | C. Mechanistic Analysis |
-| 10 | Emotion Surgery — sentiment steering via signal injection | D. Control & Steering |
-| 11 | Concept Inception — binary-search concept implantation | D. Control & Steering |
-| 12 | Genetic Hijack — global recipe matrix manipulation | D. Control & Steering |
-
-Enter `all` to run all experiments, or specific numbers (e.g., `1 3 5`). Reports are saved to `out/<model>/audit_reports/`.
-
-## Checkpoint Inspection
-
-```bash
-python check.py config/train_reflow_1.py out/reflow-1/ckpt.pt
-```
-
-## License
-
-MIT License. Based on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.
+---
+license: mit
+language:
+- en
+- zh
+tags:
+- transformer
+- interpretability
+- mechanistic-interpretability
+- language-model
+- signal-decomposition
+- sparse-representations
+- pytorch
+datasets:
+- openwebtext
+pipeline_tag: text-generation
+---
+
+# reFlow  
+
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19160838.svg)](https://doi.org/10.5281/zenodo.19160838)
+
+[ [中文](README_CN.md) | English ]
+
+**A Metal Soul In My Hand** — A feature-decoupled Transformer architecture with native interpretability.
+
+reFlow factorizes the embedding matrix $E \in \mathbb{R}^{V \times d}$ into a **Recipe Matrix** $W_{recipe} \in \mathbb{R}^{V \times S}$ and a **Signal Basis Matrix** $W_{basis} \in \mathbb{R}^{S \times d}$, forcing the model to maintain a set of continuous, low-redundancy signal bases in latent space. The same factored product $W_{recipe} \times W_{basis}$ serves as both the input embedding and the output projection, forming an end-to-end signal-manifold computation loop without a separate LM head.
+
+## Online Demo
+
+**Try reFlow in your browser:**
+- [HuggingFace Space](https://huggingface.co/spaces/reuAC/reFlow) (Global Access)
+- [ModelScope Studio](https://www.modelscope.cn/studios/recuAC/reFlow) (China Access)
+
+## Key Results
+
+**Convergence.** At matched depth and scale (36 layers, ~515M parameters), reFlow-1-Big achieves a validation loss within ~1% of GPT-2-New (514M). Three scale points — Small (46.47M), reFlow-1 (463.67M), Big (515.06M) — confirm strict scaling law compliance (val loss: 3.55 → 3.01 → 2.92).
+
+**Emergent Interpretable Structure** (pure language modeling objective, no auxiliary loss):
+- Recipe-space semantic algebra: king + woman − man → queen (rank #1), 3/3 tests passed
+- Natural sparsity: each token activates ~11% of signals (mean 117/1024), Gini coefficient 0.085
+- Causal traceability: single-signal ablation collapses target probability from 8.31% to 0.03%
+- Information crystallization boundary: semantic interventions are effective at L0–L12 but inert beyond L18
+- Hard sparsity (Top-64) systematically destroys recipe-space semantic structure (algebra 3/3 → 0/3, silhouette +0.11 → −0.02)
+
+> **Paper**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf) — Theoretical derivation, 12 interpretability experiments, and scaling/ablation analysis.
+>
+> **Pretrained Weights**: [HuggingFace](https://huggingface.co/reuAC/reFlow)
+
+## Project Structure
+
+```
+reFlow/
+├── train.py              # Training script (single GPU / DDP)
+├── sample.py             # Text generation from trained models
+├── experiment.py          # 12-experiment interpretability suite (Chinese)
+├── experiment_en.py       # 12-experiment interpretability suite (English)
+├── check.py              # Checkpoint parameter inspector
+├── bench.py              # Performance benchmarking
+├── models/
+│   ├── gpt2.py           # Standard GPT-2 baseline
+│   ├── gpt2-new.py       # Modernized GPT-2 (RoPE + SwiGLU + RMSNorm)
+│   ├── reflow.py         # reFlow base architecture
+│   ├── reflow-topk.py    # reFlow with ReLU + Top-K hard sparsity
+│   └── reflow-lite.py    # reFlow with GQA + reduced MLP
+├── config/               # Training / sampling / eval configurations
+├── data/
+│   ├── openwebtext/      # OpenWebText dataset preparation
+│   └── sft-lima/         # LIMA SFT dataset preparation
+└── out/                  # Checkpoints and experiment reports
+```
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10+
+- CUDA-compatible GPU (tested on Tesla T4 x4)
+
+### 1. PyTorch (CUDA 12.8)
+
+```bash
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+```
+
+> Adjust the CUDA version in the URL to match your driver. See [PyTorch Get Started](https://pytorch.org/get-started/locally/).
+
+### 2. Core Dependencies
+
+```bash
+pip install datasets tiktoken wandb tqdm
+```
+
+### 3. Experiment Suite Dependencies
+
+The interpretability experiments (`experiment.py`) require additional packages:
+
+```bash
+pip install numpy matplotlib seaborn scikit-learn scipy adjustText
+```
+
+### Quick Install (All-in-One)
+
+```bash
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+pip install datasets tiktoken wandb tqdm numpy matplotlib seaborn scikit-learn scipy adjustText
+```
+
+## Data Preparation
+
+### OpenWebText
+
+```bash
+python data/openwebtext/prepare.py
+```
+
+This downloads the OpenWebText corpus (~54 GB) and tokenizes it with the GPT-2 BPE tokenizer. Output: `data/openwebtext/train.bin` (~17 GB, ~9B tokens) and `val.bin`.
+
+## Training
+
+All configurations are in `config/`. No CLI overrides — all hyperparameters must be set in the config file.
+
+### Single GPU
+
+```bash
+python train.py config/train_reflow_1.py
+```
+
+### Multi-GPU (DDP)
+
+```bash
+torchrun --standalone --nproc_per_node=4 train.py config/train_reflow_1.py
+```
+
+### Available Training Configs
+
+| Config | Architecture | Layers | Params | Notes |
+|--------|-------------|--------|--------|-------|
+| `train_gpt2.py` | GPT-2 | 36 | 505.62M | Standard baseline |
+| `train_gpt2_new.py` | GPT-2-New | 36 | 514.01M | + RoPE, SwiGLU, RMSNorm |
+| `train_reflow_1.py` | reFlow | 32 | 463.67M | Base reFlow, constant lr |
+| `train_reflow_1_big.py` | reFlow | 36 | 515.06M | lr decay, for interpretability |
+| `train_reflow_1_topk_big.py` | reFlow-TopK | 36 | 515.06M | + ReLU + Top-64 sparsity |
+| `train_reflow_1_lite.py` | reFlow-Lite | 32 | 413.34M | + GQA, reduced MLP |
+| `train_reflow_1_small.py` | reFlow | 6 | 46.47M | Small-scale validation |
+
+### Resume Training
+
+Append `_resume` to the config name (e.g., `train_reflow_1_big_resume.py`).
+
+## Text Generation
+
+```bash
+python sample.py config/sample_reflow_1.py
+```
+
+Edit the config file to change the prompt, temperature, top-k, etc.
+
+## Interpretability Experiments
+
+The experiment suite runs 12 analyses on a trained reFlow model. Both Chinese and English versions are available:
+
+```bash
+python experiment_en.py config/train_reflow_1_big.py   # English
+python experiment.py config/train_reflow_1_big.py      # Chinese
+```
+
+An interactive menu will appear:
+
+| # | Experiment | Group |
+|---|-----------|-------|
+| 1 | Recipe Atlas — recipe-space nearest neighbors | A. Signal Identity |
+| 2 | Sparsity Profile — activation sparsity analysis | A. Signal Identity |
+| 3 | Basis Geometry — singular value & effective rank | A. Signal Identity |
+| 4 | Semantic Galaxy — PCA clustering visualization | B. Semantic Properties |
+| 5 | Semantic Algebra — vector arithmetic (king − man + woman = queen) | B. Semantic Properties |
+| 6 | Typo Resilience — robustness to spelling errors | B. Semantic Properties |
+| 7 | Layer Evolution — per-layer probability crystallization | C. Mechanistic Analysis |
+| 8 | Signal Flow — signal activation heatmaps across layers | C. Mechanistic Analysis |
+| 9 | Causal Ablation — progressive signal knockout curves | C. Mechanistic Analysis |
+| 10 | Emotion Surgery — sentiment steering via signal injection | D. Control & Steering |
+| 11 | Concept Inception — binary-search concept implantation | D. Control & Steering |
+| 12 | Genetic Hijack — global recipe matrix manipulation | D. Control & Steering |
+
+Enter `all` to run all experiments, or specific numbers (e.g., `1 3 5`). Reports are saved to `out/<model>/audit_reports/`.
+
+## Checkpoint Inspection
+
+```bash
+python check.py config/train_reflow_1.py out/reflow-1/ckpt.pt
+```
+
+## License
+
+MIT License. Based on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.