Upload 4 files

README.md

@@ -1,183 +1,183 @@
----
-license: mit
-language:
-- en
-- zh
-tags:
-- transformer
-- interpretability
-- mechanistic-interpretability
-- language-model
-- signal-decomposition
-- sparse-representations
-- pytorch
-datasets:
-- openwebtext
-pipeline_tag: text-generation
----
-
-# reFlow
-
-**A Metal Soul In My Hand** — A feature-decoupled Transformer architecture with native interpretability.
-
-reFlow reconstructs the traditional full-rank embedding matrix into the product of a **Recipe Matrix** $W_{recipe} \in \mathbb{R}^{V \times S}$ and a **Signal Basis Matrix** $W_{basis} \in \mathbb{R}^{S \times d}$, forcing the model to maintain a set of continuous, low-redundancy signal bases in latent space. A dynamic vocabulary matrix $W_{vocab} = W_{recipe} \times W_{basis}$ is reconstructed in real-time at each forward pass, serving simultaneously as both the embedding matrix and the output projection matrix.
-
-> **Paper**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf)
-
-## Project Structure
-
-```
-reFlow/
-├── train.py              # Training script (single GPU / DDP)
-├── sample.py             # Text generation from trained models
-├── experiment.py          # 12-experiment interpretability suite (Chinese)
-├── experiment_en.py       # 12-experiment interpretability suite (English)
-├── check.py              # Checkpoint parameter inspector
-├── bench.py              # Performance benchmarking
-├── models/
-│   ├── gpt2.py           # Standard GPT-2 baseline
-│   ├── gpt2-new.py       # Modernized GPT-2 (RoPE + SwiGLU + RMSNorm)
-│   ├── reflow.py         # reFlow base architecture
-│   ├── reflow-topk.py    # reFlow with ReLU + Top-K hard sparsity
-│   └── reflow-lite.py    # reFlow with GQA + reduced MLP
-├── config/               # Training / sampling / eval configurations
-├── data/
-│   ├── openwebtext/      # OpenWebText dataset preparation
-│   └── sft-lima/         # LIMA SFT dataset preparation
-└── out/                  # Checkpoints and experiment reports
-```
-
-## Installation
-
-### Prerequisites
-
-- Python 3.10+
-- CUDA-compatible GPU (tested on Tesla T4 x4)
-
-### 1. PyTorch (CUDA 12.8)
-
-```bash
-pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
-```
-
-> Adjust the CUDA version in the URL to match your driver. See [PyTorch Get Started](https://pytorch.org/get-started/locally/).
-
-### 2. Core Dependencies
-
-```bash
-pip install datasets tiktoken wandb tqdm
-```
-
-### 3. Experiment Suite Dependencies
-
-The interpretability experiments (`experiment.py`) require additional packages:
-
-```bash
-pip install numpy matplotlib seaborn scikit-learn scipy adjustText
-```
-
-### Quick Install (All-in-One)
-
-```bash
-pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
-pip install datasets tiktoken wandb tqdm numpy matplotlib seaborn scikit-learn scipy adjustText
-```
-
-## Data Preparation
-
-### OpenWebText
-
-```bash
-python data/openwebtext/prepare.py
-```
-
-This downloads the OpenWebText corpus (~54 GB) and tokenizes it with the GPT-2 BPE tokenizer. Output: `data/openwebtext/train.bin` (~17 GB, ~9B tokens) and `val.bin`.
-
-## Training
-
-All configurations are in `config/`. No CLI overrides — all hyperparameters must be set in the config file.
-
-### Single GPU
-
-```bash
-python train.py config/train_reflow_1.py
-```
-
-### Multi-GPU (DDP)
-
-```bash
-torchrun --standalone --nproc_per_node=4 train.py config/train_reflow_1.py
-```
-
-### Available Training Configs
-
-| Config | Architecture | Layers | Params | Notes |
-|--------|-------------|--------|--------|-------|
-| `train_gpt2.py` | GPT-2 | 36 | 505.62M | Standard baseline |
-| `train_gpt2_new.py` | GPT-2-New | 36 | 514.01M | + RoPE, SwiGLU, RMSNorm |
-| `train_reflow_1.py` | reFlow | 32 | 463.67M | Base reFlow, constant lr |
-| `train_reflow_1_big.py` | reFlow | 36 | 515.06M | lr decay, for interpretability |
-| `train_reflow_1_topk_big.py` | reFlow-TopK | 36 | 515.06M | + ReLU + Top-64 sparsity |
-| `train_reflow_1_lite.py` | reFlow-Lite | 32 | 413.34M | + GQA, reduced MLP |
-| `train_reflow_1_small.py` | reFlow | 6 | 46.47M | Small-scale validation |
-
-### Resume Training
-
-Append `_resume` to the config name (e.g., `train_reflow_1_big_resume.py`).
-
-## Text Generation
-
-```bash
-python sample.py config/sample_reflow_1.py
-```
-
-Edit the config file to change the prompt, temperature, top-k, etc.
-
-## Interpretability Experiments
-
-The experiment suite runs 12 analyses on a trained reFlow model. Both Chinese and English versions are available:
-
-```bash
-python experiment_en.py config/train_reflow_1_big.py   # English
-python experiment.py config/train_reflow_1_big.py      # Chinese
-```
-
-An interactive menu will appear:
-
-| # | Experiment | Group |
-|---|-----------|-------|
-| 1 | Recipe Atlas — recipe-space nearest neighbors | A. Signal Identity |
-| 2 | Sparsity Profile — activation sparsity analysis | A. Signal Identity |
-| 3 | Basis Geometry — singular value & effective rank | A. Signal Identity |
-| 4 | Semantic Galaxy — PCA clustering visualization | B. Semantic Properties |
-| 5 | Semantic Algebra — vector arithmetic (king − man + woman = queen) | B. Semantic Properties |
-| 6 | Typo Resilience — robustness to spelling errors | B. Semantic Properties |
-| 7 | Layer Evolution — per-layer probability crystallization | C. Mechanistic Analysis |
-| 8 | Signal Flow — signal activation heatmaps across layers | C. Mechanistic Analysis |
-| 9 | Causal Ablation — progressive signal knockout curves | C. Mechanistic Analysis |
-| 10 | Emotion Surgery — sentiment steering via signal injection | D. Control & Steering |
-| 11 | Concept Inception — binary-search concept implantation | D. Control & Steering |
-| 12 | Genetic Hijack — global recipe matrix manipulation | D. Control & Steering |
-
-Enter `all` to run all experiments, or specific numbers (e.g., `1 3 5`). Reports are saved to `out/<model>/audit_reports/`.
-
-## Checkpoint Inspection
-
-```bash
-python check.py config/train_reflow_1.py out/reflow-1/ckpt.pt
-```
-
-## License
-
-MIT License. Based on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.  
-
-```bibtex
-@misc{reuac_2026,
-    author       = { reuAC },                                                                                                                          
-    title        = { reFlow (Revision 672259a) },
-    year         = 2026,
-    url          = { https://huggingface.co/reuAC/reFlow },
-    doi          = { 10.57967/hf/8047 },
-    publisher    = { Hugging Face }
-}
-```
+---
+license: mit
+language:
+- en
+- zh
+tags:
+- transformer
+- interpretability
+- mechanistic-interpretability
+- language-model
+- signal-decomposition
+- sparse-representations
+- pytorch
+datasets:
+- openwebtext
+pipeline_tag: text-generation
+---
+
+# reFlow
+
+**A Metal Soul In My Hand** — A feature-decoupled Transformer architecture with native interpretability.
+
+reFlow factorizes the embedding matrix $E \in \mathbb{R}^{V \times d}$ into a **Recipe Matrix** $W_{recipe} \in \mathbb{R}^{V \times S}$ and a **Signal Basis Matrix** $W_{basis} \in \mathbb{R}^{S \times d}$, forcing the model to maintain a set of continuous, low-redundancy signal bases in latent space. The same factored product $W_{recipe} \times W_{basis}$ serves as both the input embedding and the output projection, forming an end-to-end signal-manifold computation loop without a separate LM head.
+
+## Key Results
+
+**Convergence.** At matched depth and scale (36 layers, ~515M parameters), reFlow-1-Big achieves a validation loss within ~1% of GPT-2-New (514M). Three scale points — Small (46.47M), reFlow-1 (463.67M), Big (515.06M) — confirm strict scaling law compliance (val loss: 3.55 → 3.01 → 2.92).
+
+**Emergent Interpretable Structure** (pure language modeling objective, no auxiliary loss):
+- Recipe-space semantic algebra: king + woman − man → queen (rank #1), 3/3 tests passed
+- Natural sparsity: each token activates ~11% of signals (mean 117/1024), Gini coefficient 0.085
+- Causal traceability: single-signal ablation collapses target probability from 8.31% to 0.03%
+- Information crystallization boundary: semantic interventions are effective at L0–L12 but inert beyond L18
+- Hard sparsity (Top-64) systematically destroys recipe-space semantic structure (algebra 3/3 → 0/3, silhouette +0.11 → −0.02)
+
+> **Paper**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf) — Theoretical derivation, 12 interpretability experiments, and scaling/ablation analysis.
+
+## Project Structure
+
+```
+reFlow/
+├── train.py              # Training script (single GPU / DDP)
+├── sample.py             # Text generation from trained models
+├── experiment.py          # 12-experiment interpretability suite (Chinese)
+├── experiment_en.py       # 12-experiment interpretability suite (English)
+├── check.py              # Checkpoint parameter inspector
+├── bench.py              # Performance benchmarking
+├── models/
+│   ├── gpt2.py           # Standard GPT-2 baseline
+│   ├── gpt2-new.py       # Modernized GPT-2 (RoPE + SwiGLU + RMSNorm)
+│   ├── reflow.py         # reFlow base architecture
+│   ├── reflow-topk.py    # reFlow with ReLU + Top-K hard sparsity
+│   └── reflow-lite.py    # reFlow with GQA + reduced MLP
+├── config/               # Training / sampling / eval configurations
+├── data/
+│   ├── openwebtext/      # OpenWebText dataset preparation
+│   └── sft-lima/         # LIMA SFT dataset preparation
+└── out/                  # Checkpoints and experiment reports
+```
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10+
+- CUDA-compatible GPU (tested on Tesla T4 x4)
+
+### 1. PyTorch (CUDA 12.8)
+
+```bash
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+```
+
+> Adjust the CUDA version in the URL to match your driver. See [PyTorch Get Started](https://pytorch.org/get-started/locally/).
+
+### 2. Core Dependencies
+
+```bash
+pip install datasets tiktoken wandb tqdm
+```
+
+### 3. Experiment Suite Dependencies
+
+The interpretability experiments (`experiment.py`) require additional packages:
+
+```bash
+pip install numpy matplotlib seaborn scikit-learn scipy adjustText
+```
+
+### Quick Install (All-in-One)
+
+```bash
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+pip install datasets tiktoken wandb tqdm numpy matplotlib seaborn scikit-learn scipy adjustText
+```
+
+## Data Preparation
+
+### OpenWebText
+
+```bash
+python data/openwebtext/prepare.py
+```
+
+This downloads the OpenWebText corpus (~54 GB) and tokenizes it with the GPT-2 BPE tokenizer. Output: `data/openwebtext/train.bin` (~17 GB, ~9B tokens) and `val.bin`.
+
+## Training
+
+All configurations are in `config/`. No CLI overrides — all hyperparameters must be set in the config file.
+
+### Single GPU
+
+```bash
+python train.py config/train_reflow_1.py
+```
+
+### Multi-GPU (DDP)
+
+```bash
+torchrun --standalone --nproc_per_node=4 train.py config/train_reflow_1.py
+```
+
+### Available Training Configs
+
+| Config | Architecture | Layers | Params | Notes |
+|--------|-------------|--------|--------|-------|
+| `train_gpt2.py` | GPT-2 | 36 | 505.62M | Standard baseline |
+| `train_gpt2_new.py` | GPT-2-New | 36 | 514.01M | + RoPE, SwiGLU, RMSNorm |
+| `train_reflow_1.py` | reFlow | 32 | 463.67M | Base reFlow, constant lr |
+| `train_reflow_1_big.py` | reFlow | 36 | 515.06M | lr decay, for interpretability |
+| `train_reflow_1_topk_big.py` | reFlow-TopK | 36 | 515.06M | + ReLU + Top-64 sparsity |
+| `train_reflow_1_lite.py` | reFlow-Lite | 32 | 413.34M | + GQA, reduced MLP |
+| `train_reflow_1_small.py` | reFlow | 6 | 46.47M | Small-scale validation |
+
+### Resume Training
+
+Append `_resume` to the config name (e.g., `train_reflow_1_big_resume.py`).
+
+## Text Generation
+
+```bash
+python sample.py config/sample_reflow_1.py
+```
+
+Edit the config file to change the prompt, temperature, top-k, etc.
+
+## Interpretability Experiments
+
+The experiment suite runs 12 analyses on a trained reFlow model. Both Chinese and English versions are available:
+
+```bash
+python experiment_en.py config/train_reflow_1_big.py   # English
+python experiment.py config/train_reflow_1_big.py      # Chinese
+```
+
+An interactive menu will appear:
+
+| # | Experiment | Group |
+|---|-----------|-------|
+| 1 | Recipe Atlas — recipe-space nearest neighbors | A. Signal Identity |
+| 2 | Sparsity Profile — activation sparsity analysis | A. Signal Identity |
+| 3 | Basis Geometry — singular value & effective rank | A. Signal Identity |
+| 4 | Semantic Galaxy — PCA clustering visualization | B. Semantic Properties |
+| 5 | Semantic Algebra — vector arithmetic (king − man + woman = queen) | B. Semantic Properties |
+| 6 | Typo Resilience — robustness to spelling errors | B. Semantic Properties |
+| 7 | Layer Evolution — per-layer probability crystallization | C. Mechanistic Analysis |
+| 8 | Signal Flow — signal activation heatmaps across layers | C. Mechanistic Analysis |
+| 9 | Causal Ablation — progressive signal knockout curves | C. Mechanistic Analysis |
+| 10 | Emotion Surgery — sentiment steering via signal injection | D. Control & Steering |
+| 11 | Concept Inception — binary-search concept implantation | D. Control & Steering |
+| 12 | Genetic Hijack — global recipe matrix manipulation | D. Control & Steering |
+
+Enter `all` to run all experiments, or specific numbers (e.g., `1 3 5`). Reports are saved to `out/<model>/audit_reports/`.
+
+## Checkpoint Inspection
+
+```bash
+python check.py config/train_reflow_1.py out/reflow-1/ckpt.pt
+```
+
+## License
+
+MIT License. Based on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.

README_CN.md

@@ -2,9 +2,20 @@
 
 **A Metal Soul In My Hand** — 具备原生可解释性的特征解耦 Transformer 架构。
 
-reFlow 将传统全秩词嵌入矩阵重构为**配方矩阵** $W_{recipe} \in \mathbb{R}^{V \times S}$ 与**信号基底矩阵** $W_{basis} \in \mathbb{R}^{S \times d}$ 的乘积形式，迫使模型在潜空间中维护一组连续、低冗余的信号基底。动态词表矩阵 $W_{vocab} = W_{recipe} \times W_{basis}$ 在每次前向传播中实时重构，同时作为嵌入矩阵与输出投影矩阵使用。
+reFlow 将嵌入矩阵 $E \in \mathbb{R}^{V \times d}$ 分解为**配方矩阵** $W_{recipe} \in \mathbb{R}^{V \times S}$ 与**信号基底矩阵** $W_{basis} \in \mathbb{R}^{S \times d}$ 的乘积形式，迫使模型在潜空间中维护一组连续、低冗余的信号基底。同一乘积 $W_{recipe} \times W_{basis}$ 同时用于输入嵌入与输出投影，构成端到端的信号流形计算闭环，无需独立 LM Head。
 
-> **论文**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf)
+## 核心结果
+
+**收敛性。** 在对齐深度与参数量（36 层，~515M）的条件下，reFlow-1-Big 的验证损失与 GPT-2-New（514M）差距仅约 1%。三个参数规模点 — Small（46.47M）、reFlow-1（463.67M）、Big（515.06M）— 验证损失分别为 3.55、3.01、2.92，严格遵循缩放定律。
+
+**自发涌现的可解释结构**（纯语言建模目标，无辅助损失）：
+- 配方空间语义代数：king + woman − man → queen（排名 #1），3/3 测试通过
+- 自然稀疏性：每个 token 平均激活约 11% 的信号（均值 117/1024），Gini 系数 0.085
+- 因果可追踪性：消融单个信号即可将目标概率从 8.31% 摧毁至 0.03%
+- 信息结晶边界：语义干预在 L0–L12 有效，L18 之后失效
+- 硬稀疏约束（Top-64）系统性摧毁配方空间语义结构（代数 3/3 → 0/3，轮廓系数 +0.11 → −0.02）
+
+> **论文**: [English (PDF)](./paper/paper.pdf) | [中文 (PDF)](./paper/paper-cn.pdf) — 理论推导、12 项可解释性实验及缩放/消融分析。
 
 ## 项目结构
 

paper/paper-cn.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8342d5e5b6c3d6e4d1b6dbc3ccde4933ab8b6cf9cd346a999b22d0dafece0cc3
-size 315963
+oid sha256:24d5e8402b4f41db3e3a26317876c4236ff5b6c534b07238567a2a912b4f4a22
+size 316420

paper/paper.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9afa7ec6025e700d5876a6be83f8e4ac5f1bd37c55104a06129808db951d97e0
-size 296359
+oid sha256:4fbbc48ebae91e0f4009d7de9f0313b560f908998680780ae47d816ca49fa3e4
+size 114729