Upload PLAN.md with huggingface_hub

PLAN.md

@@ -0,0 +1,330 @@
+# 🧬 Self-Evolving Neural Network — Project Plan & Status
+
+## Current Status: ✅ Core Complete, GUI Working, Dataset Integrated, 🔬 English Training RUNNING
+
+### Active Training Runs (Aug 1)
+| Run | Command | Status | Checkpoints |
+|-----|---------|--------|-------------|
+| FABLE numeric regression | `--dataset fable --n-samples 10000 --generations 30 --pop-size 6 --train-epochs 10` | ✅ RUNNING (PID 22047, 120-min budget) | `evo_checkpoints/` |
+| **English understanding** | `--text-dataset cornell-movie-review-data/rotten_tomatoes --n-samples 3000 --generations 10 --pop-size 4 --train-epochs 5` | 🔬 RUNNING (PID 33878, 90-min budget) | `evo_checkpoints_text/` |
+
+> ⚠️ Note: the newer `huggingface_hub` (≥1.26) requires **namespaced** dataset ids — use
+> `cornell-movie-review-data/rotten_tomatoes`, NOT bare `rotten_tomatoes` (that errors with `HfUriError`).
+> The FABLE id `Crownelius/Complete-FABLE.5-traces-2M` already has a namespace, so it's unaffected.
+
+### 🐛 Critical data bug found & fixed (Aug 1)
+The first English run scored **50% (chance) on held-out reviews** despite high training fitness.
+Root cause: `rotten_tomatoes` train split is **sorted by label** (rows 0–4000 all-positive,
+6000+ all-negative). `load_text_dataset` used `split="train[:3000]"` → the model trained on a
+**single class** and learned "always predict positive" (100% on that slice, 50% on balanced test).
+
+**Fix:** `load_text_dataset` now loads the full split and draws a **seeded random sample**
+(`np.random.RandomState(42).choice(..., replace=False)`) instead of a head-slice.
+Verified: 3000-sample draw is now 50.0% positive. ⚠️ Same hazard applies to any HF dataset
+with sorted labels — never head-slice without checking label distribution.
+
+### 🔧 Evolution-engine upgrades (Aug 1)
+- **Classification fitness is now validation-accuracy based** (`best_val_acc² × 100 / penalty`)
+  instead of `1/val_loss` — loss-only fitness rewarded memorization (the cause of the
+  first run's 100% train / 50% test split). Regression fitness unchanged.
+- **LR scheduling**: `ReduceLROnPlateau` (factor 0.5, patience 2) during each eval.
+- **Diversity pressure** (`apply_diversity_pressure`): exact duplicate configs → fitness
+  ×0.5; near-identical-to-best topologies → ×0.9. The current champion is exempt so
+  stagnation tracking and best-genome re-selection stay valid.
+- **`--continue` is text-aware**: `SelfTrainer.continue_evolution` now passes
+  `task_type`/`vectorizer`/`embed_dim`/`checkpoint_dir` through, and `run()` only
+  initializes a fresh population when none was pre-seeded (fixes the old bug where
+  `--continue` clobbered the seeded population and rebuilt a regression engine).
+- **`load_fable_dataset`** also now random-samples (seeded) instead of head-slicing.
+
+### 🌍 English text path — end-to-end raw-English inference (Aug 1)
+- `save_best_model` now **bakes the fitted TextVectorizer into `best_model.keras`** for
+  text runs → the saved model accepts **raw English strings** (no separate tokenizer needed).
+  Also saves `best_model_ids.keras` (token-id core) and `vectorizer_config.json`
+  (incl. the exact fitted vocab list for token-id parity in eval).
+- `eval_text_model.py` scores the best model on held-out test reviews in 3 auto modes:
+  1) serving (raw strings), 2) token-id (reuses saved vocab), 3) rebuild preview.
+- `auto_pipeline.sh` (running): waits for FABLE + TEXT to finish → evals the text run →
+  launches scaled-up run `evo_checkpoints_text_v2` (8K samples, vocab 20k, embed 128) → evals it.
+- `check_progress.py`: prints generation-by-generation fitness for both checkpoint dirs.
+
+
+### 🧠 v3-Gemma strategy (Aug 1) — "extract the GPT of Gemma + self-evolve on FABLE"
+
+**Discovery:** FABLE.5-traces-2M's `row_json` contains REAL ENGLISH — it's 229K AI
+coding-session traces with actual user prompts (`message.content`) + completions.
+So a pretrained language model CAN read FABLE — the earlier "FABLE is numeric-only"
+assumption was wrong (only our hand-engineered 10 features were numeric).
+
+**New plan (v3, Colab GPU):**
+1. Load `Qwen/Qwen2.5-0.5B-Instruct` (open GPT-class transformer, Apache 2.0, un-gated
+   repo, ~1.1 GB FP16 — the "GPT" we borrow English from; frozen, forward-only).
+   (Swapped from gemma-3-270m for reliability: no gated terms acceptance needed,
+   standard Qwen2 architecture, no `trust_remote_code`, 896-dim hidden state.)
+2. Extract FABLE text (`row_json` → content + completion) → mean-pooled Gemma last
+   hidden state = per-row English-understanding embeddings (input_dim ≈ 1152).
+3. Feed embeddings as X into `EvolutionEngine` (task_type=regression, target =
+   `log1p(seen_count)`) — the SAME engine as local runs, now on a T4 GPU (10-20× faster).
+4. Eval held-out R²/MAE → push `evo_checkpoints_gemma/` back to HF.
+
+**Deliverable:** `evo_gemma_fable_colab.ipynb` (18 cells, generated by
+`make_gemma_colab_notebook.py` — run it to regenerate). Prereqs: HF token (only for
+pushing results back; model download is un-gated) + project code pushed to REPO_ID
+via `hf_upload.py`.
+This is the bridge toward a true mini-GPT: attention + generative head come later (v4/v5).
+
+### Files Created
+| File | Lines | Description |
+|------|-------|-------------|
+| `self_evolving_model.py` | 996+ | Core evolution engine with genome system, training, checkpointing |
+| `evo_gui.py` | 578+ | Flask web GUI with real-time monitoring, Chart.js fitness plots |
+| `hf_upload.py` | — | Upload model/genome/history + auto-generated model card to Hugging Face |
+| `make_gemma_colab_notebook.py` | — | Generates `evo_gemma_fable_colab.ipynb` (v3-Qwen strategy) |
+| `evo_gemma_fable_colab.ipynb` | — | Colab GPU: Qwen2.5-0.5B embeddings → self-evolved head on FABLE text |
+| `evo_v2_colab.ipynb` | — | Ready-to-run Google Colab notebook (v2/v3 GPU training, scale-up, push-back) |
+| `make_growth_chart.py` | — | Generates `growth_limit.html` — theoretical parameter-growth line chart |
+| `growth_limit.html` | — | Dependency-free SVG chart: max growth 4,216 → 345,352 params (extrap. to 1M @ ~16 layers) |
+| `requirements.txt` | — | Pinned dependencies for laptop/Colab reproducibility |
+| `PLAN.md` | — | This file — project plan and continuation guide |
+
+### Checkpoints (from previous test runs)
+| File | Description |
+|------|-------------|
+| `evo_checkpoints/checkpoint.pkl` | Serialized evolution state |
+| `evo_checkpoints/best_model.keras` | Best evolved model |
+| `evo_checkpoints/best_genome.json` | Best genome architecture config |
+| `evo_checkpoints/evolution_history.json` | Fitness history across generations |
+
+---
+
+## What's Been Built
+
+### 1. Genome System (`Genome` class)
+- Neural network architecture encoded as mutable genome
+- Layers, units, activations, dropout, batch norm, learning rate, optimizer
+- **Mutation operators**: add/remove layers, change units, swap activations, perturb LR
+- **Crossover**: layer-by-layer parent mixing with hyperparameter inheritance
+
+### 2. Evolution Engine (`EvolutionEngine` class)
+- Population-based evolutionary optimization
+- Elitism selection (top 30% survive)
+- Reproduction via mutation + crossover
+- Automatic checkpointing every generation
+- Stop event for GUI integration (`threading.Event`)
+- Generation callback for real-time GUI updates
+
+### 3. Dataset Integration (`DataHandler.load_fable_dataset`)
+- Loads `Crownelius/Complete-FABLE.5-traces-2M` from HuggingFace (228K rows)
+- Extracts 10-dimensional feature vectors from heterogeneous JSON coding traces:
+  - Message length, word count, code keyword frequency
+  - Completion length, chain-of-thought length
+  - Tool use indicators, output complexity
+  - Session entropy (deterministic md5 hash)
+  - Special character ratio
+- StandardScaler normalization on features
+- Log-normalization on target (seen_count)
+
+### 4. Web GUI (`evo_gui.py`)
+- **Flask** server with single-page HTML/JS frontend
+- **Dark theme** with monospace font (GitHub-style)
+- **Chart.js** real-time fitness line chart (best + avg)
+- **Controls**: Start/Stop buttons, parameter inputs
+- **Statistics panel**: generation, fitness, improvement %, population
+- **Genome visualization**: layer chips, architecture display
+- **Population leaderboard**: sorted by fitness with medals
+- **Evolution log**: scrolling text log
+- **API endpoints**:
+  - `GET /` — HTML dashboard
+  - `GET /api/status` — JSON state
+  - `POST /api/start` — Start evolution with params
+  - `POST /api/stop` — Stop evolution
+
+### 5. Model Building (`ModelEvaluator`)
+- Builds Keras models from genome configs
+- Supports 4 optimizers: Adam, SGD, RMSprop, AdamW
+- 7 activation functions: relu, tanh, sigmoid, elu, selu, swish, linear
+- Batch normalization and dropout support
+- Fitness = 1 / (val_loss × complexity_penalty)
+
+### 6. Gated Feature Architecture (`top3_features`)
+- Each genome encodes which **3 features** (by index) feed layer 1
+- The remaining features are concatenated into **every layer after the first** (incl. output)
+- Feature selection **evolves** via mutation/crossover (e.g. `top3=[1,3,7]`)
+- Old checkpoints without the field default to `[0,1,2]` (backward compatible)
+
+### 7. ⛔ Safety Gates (`SafetyGates` class) — the AI CANNOT change these
+- **Hard walls**: max 16 layers, 2048 units/layer, 5M total params — enforced on EVERY
+  evaluation via `enforce_genome()` (even corrupt/old checkpoints get clamped)
+- **Failure gates**: NaN/Inf or exploded val_loss → fitness 0; over-param wall → fitness 0
+- **Fitness clamp**: fitness bounded to `[0, 1e9]`
+- **Runtime gates**: `--max-minutes` wall-clock budget; `--stagnation-limit` (default 12 gens
+  without improvement → halt) — both trip safely and log to `safety_trips`
+- Safety constants live OUTSIDE the genome (module-level), never checkpointed,
+  never mutated by evolution — editing `SafetyGates` requires source-code changes
+
+### 8. English understanding — honest status
+- The model does **NOT** understand English. It is a numeric regressor: FABLE traces are
+  converted to 10 numbers (message length, word count, code-keyword freq, ...) and it
+  predicts a count. There is no language model in this stack.
+- Making it "understand English" requires a different architecture (embedding/text model),
+  which is beyond this neuroevolution setup.
+
+## Scaling Pipeline (v1 → HF → v2/v3)
+
+```
+LOCAL CPU (v1) ──▶ hf_upload.py ──▶ Hugging Face ──▶ evo_v2_colab.ipynb (GPU, bigger caps) ──▶ push back
+   small models        model card        durable backup      scale to 512-1024 units / 8-10 layers
+```
+
+- **Local (this box):** fast CPU iterations; checkpoints auto-save every generation
+- **HF upload:** `python3 hf_upload.py --repo user/repo --token $HF_TOKEN` (dry-run with `--dry-run`)
+- **Colab v2/v3:** open `evo_v2_colab.ipynb` → Runtime ▸ Run all → paste HF_TOKEN. It pulls the repo,
+  continues evolution with `--max-units/--max-layers`, and pushes results back
+- **Laptop:** `pip install -r requirements.txt` then run as normal
+- ⚠️ This cloud VM resets ~20 min after session end — push checkpoints to HF early and often
+
+### Genome Search Space (scalable)
+- Default: max 6 layers × 256 units (~345K params ceiling)
+- `--max-units 512 --max-layers 8` → ~1.1M params ceiling (v2)
+- `--max-units 1024 --max-layers 10` → larger (v3, GPU)
+- See `growth_limit.html` for the theoretical growth line
+- ⛔ All of the above are still capped by `SafetyGates` hard walls (16 layers / 2048 units / 5M params)
+
+---
+
+## Scaling Pipeline (v1 → HF → v2/v3)
+
+```
+LOCAL CPU (v1) ──▶ hf_upload.py ──▶ Hugging Face ──▶ evo_v2_colab.ipynb (GPU, bigger caps) ──▶ push back
+   small models        model card        durable backup      scale to 512-1024 units / 8-10 layers
+```
+
+- **Local (this box):** fast CPU iterations; checkpoints auto-save every generation
+- **HF upload:** `python3 hf_upload.py --repo user/repo --token $HF_TOKEN` (dry-run with `--dry-run`)
+- **Colab v2/v3:** open `evo_v2_colab.ipynb` → Runtime ▸ Run all → paste HF_TOKEN. It pulls the repo,
+  continues evolution with `--max-units/--max-layers`, and pushes results back
+- **Laptop:** `pip install -r requirements.txt` then run as normal
+- ⚠️ This cloud VM resets ~20 min after session end — push checkpoints to HF early and often
+
+### Genome Search Space (scalable)
+- Default: max 6 layers × 256 units (~345K params ceiling)
+- `--max-units 512 --max-layers 8` → ~1.1M params ceiling (v2)
+- `--max-units 1024 --max-layers 10` → larger (v3, GPU)
+- See `growth_limit.html` for the theoretical growth line
+
+## How to Run
+
+### CLI Mode (terminal)
+```bash
+# Quick run with synthetic data
+python3 self_evolving_model.py --generations 20 --pop-size 8
+
+# With FABLE dataset
+python3 self_evolving_model.py --dataset fable --n-samples 10000 --generations 30
+
+# Continue from previous best
+python3 self_evolving_model.py --continue --generations 50
+```
+
+### GUI Mode (web browser)
+```bash
+# Standalone (loads dataset on demand)
+python3 evo_gui.py --port 5000 --n-samples 5000
+
+# Via CLI flag (pre-loads dataset)
+python3 self_evolving_model.py --gui --gui-port 5000 --n-samples 5000
+```
+Then open `http://localhost:5000` in browser.
+
+---
+
+## Verified Working ✅
+- [x] Genome mutation/crossover
+- [x] Model building from genomes
+- [x] Evolution loop with elitism
+- [x] Checkpoint save/load/resume
+- [x] CLI argument parsing
+- [x] Dataset loading from HuggingFace
+- [x] Feature extraction from JSON traces
+- [x] Flask GUI serving HTML
+- [x] GUI API endpoints (start/stop/status)
+- [x] Background thread evolution
+- [x] Real-time GUI updates via polling
+- [x] Chart.js fitness visualization
+- [x] Population leaderboard
+- [x] Stop signal from GUI → engine
+
+---
+
+## Next Steps (TODO)
+
+### Priority 1: Run Full Training
+```bash
+# Start GUI with full dataset
+python3 evo_gui.py --port 5000 --n-samples 10000
+```
+Then in browser: set Generations=30, Pop=6, Train Epochs=10, click Start.
+
+### Priority 2: Potential Improvements
+1. **Streaming dataset loading** — current approach downloads full 228K rows then slices. Use HF streaming for faster loads.
+2. **GPU support** — if CUDA available, use `tf.distribute.MirroredStrategy` for faster training.
+3. **Dataset reload button** in GUI — allow changing sample count without restart.
+4. **Export best model** — add button to download the best evolved model.
+5. **Diversity pressure** — penalize genomes too similar to prevent convergence to local optima.
+6. **Learning rate scheduling** — add cosine annealing or step decay to genome.
+
+### Priority 3: Architecture Enhancements
+1. **Multi-objective evolution** — optimize both fitness AND model size.
+2. **Transfer learning** — start from pre-trained model weights.
+3. **Neural Architecture Search (NAS)** — add skip connections, attention layers.
+4. **Distributed evolution** — run populations across multiple machines.
+
+---
+
+## Architecture Diagram
+
+```
+┌──────────────���──────────────────────────────────────┐
+│                    evo_gui.py                        │
+│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
+│  │  Flask    │  │  Chart.js│  │  HTML Dashboard  │  │
+│  │  Routes   │──│  Live UI │──│  Controls/Stats  │  │
+│  └────┬─────┘  └──────────┘  └──────────────────┘  │
+│       │ /api/start                                   │
+│       ▼                                              │
+│  ┌──────────────────────────────────────────┐        │
+│  │  Background Thread                        │        │
+│  │  ┌─────────────┐  ┌──────────────────┐  │        │
+│  │  │  DataHandler │  │ EvolutionEngine  │  │        │
+│  │  │  (HF Dataset)│──│  (Population)    │  │        │
+│  │  └─────────────┘  └────────┬─────────┘  │        │
+│  │                             │ callback    │        │
+│  │                             ▼             │        │
+│  │  ┌─────────────────────────────────────┐ │        │
+│  │  │  ModelEvaluator                     │ │        │
+│  │  │  Genome → Keras Model → Fitness     │ │        │
+│  │  └─────────────────────────────────────┘ │        │
+│  └──────────────────────────────────────────┘        │
+└─────────────────────────────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────────────────────┐
+│              self_evolving_model.py                    │
+│  ┌──────────┐  ┌───────────┐  ┌─────────────────┐  │
+│  │  Genome   │  │  Data     │  │  Evolution      │  │
+│  │  (config) │  │  Handler  │  │  Engine         │  │
+│  └──────────┘  └───────────┘  └─────────────────┘  │
+└─────────────────────────────────────────────────────┘
+```
+
+---
+
+## Dependencies
+- Python 3.12+
+- TensorFlow 2.21.0
+- NumPy 2.4.6
+- Flask 3.1.3
+- datasets 5.0.1 (HuggingFace)
+- huggingface_hub 1.26.0
+
+All installed and verified working.