File size: 2,935 Bytes
148b631 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
# 🧪 RippleGPT Validation Suite
This module validates the hypothesis that the **RippleGPT** architecture (Decay-Biased Attention + Multiplicative Gating) can understand **hierarchical code structures** better than standard Transformer architectures.
## 🎯 Objective
Tests if the "Ripple Field" mechanism can:
1. **Close parentheses/braces correctly** - Requires attention to open scopes
2. **Indent Python correctly** - Requires understanding block hierarchy
3. **Complete code consistently** - Requires long-range context
## 📦 Dataset
We use [bigcode/the-stack-smol](https://huggingface.co/datasets/bigcode/the-stack-smol), a clean subset of Python code from The Stack.
## 🚀 Quick Start
### 1. Install Dependencies
```bash
cd /path/to/RippleGPT
pip install -r requirements.txt
```
### 2. Prepare Data
```bash
python validation/code/prepare_code_data.py
```
This script:
- Downloads Python code from the-stack-smol (streaming, ~5MB)
- Tokenizes at character level
- Saves to `validation/code/data/`
### 3. Train Model
```bash
python validation/code/train_code.py
```
Trains RippleGPT for 3000 iterations (~15min on M1/M2).
### 4. Run Validation
```bash
python validation/code/validate_code.py
```
Executes all validation tests and generates a report.
## 📊 Validation Metrics
### Test 1: Parentheses/Brace Closing
```python
# Input: "def foo(a, b"
# Expect: "def foo(a, b):"
```
### Test 2: Python Indentation
```python
# Input: "if x > 0:\n"
# Expect: "if x > 0:\n return" (4 spaces)
```
### Test 3: Function Structure
```python
# Input: "def calculate_sum(numbers):\n total = 0\n for n in numbers:\n total +="
# Expect: Complete with " n" and close the loop correctly
```
### Test 4: Long Context (Extrapolation)
Tests if the model maintains coherence in functions with 50+ lines.
## 📁 Structure
```
validation/code/
├── README.md # This file
├── prepare_code_data.py # Prepares dataset
├── train_code.py # Trains model on code
├── validate_code.py # Runs validations
├── test_cases.py # Defined test cases
├── metrics.py # Evaluation functions
└── data/ # Processed data (generated)
├── train.bin
├── val.bin
└── meta.pkl
```
## 🔬 Scientific Hypothesis
The "Folded Cloth" (Ripple Field) architecture should outperform linear models in tasks requiring:
- **Scope Attention** - Natural decay helps "remember" open brackets
- **Hierarchical Structure** - Multiplicative gating modulates importance of structural tokens
## 📈 Expected Results
| Metric | Standard GPT | RippleGPT |
|--------|--------------|-----------|
| Bracket Accuracy | ~70% | **~85%+** |
| Indent Accuracy | ~60% | **~80%+** |
| Function Coherence | Lower | **Higher** |
---
**Author:** Victor Carvalho Tavernari
**Project:** RippleGPT Validation Suite
|