docs: add Colab-ready notebook

notebooks/SCU_Demo.ipynb

@@ -5,7 +5,7 @@
    "metadata": {
     "id": "header"
    },
-   "source": "# Shannon Control Unit (SCU) Demo\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://huggingface.co/hunterbown/shannon-control-unit/blob/main/notebooks/SCU_Demo.ipynb)\n\nThis notebook demonstrates the Shannon Control Unit - an adaptive regularization system that achieves **15.6% lower perplexity** without manual hyperparameter tuning.\n\n**Note:** Click the \"Open in Colab\" badge above, then in the Colab interface, click File → Save a copy in Drive to run the notebook."
+   "source": "# Shannon Control Unit (SCU) Demo\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hmbown/shannon-control-unit/blob/main/notebooks/SCU_Demo.ipynb)\n[![Hugging Face](https://img.shields.io/badge/🤗-Models-yellow)](https://huggingface.co/hunterbown/shannon-control-unit)\n\nThis notebook demonstrates the Shannon Control Unit - an adaptive regularization system that achieves **up to 15.6% lower perplexity** without manual hyperparameter tuning.\n\n**To run this notebook:**\n1. Click \"Open in Colab\" above\n2. In Colab: File → Save a copy in Drive\n3. Runtime → Run all\n\nThe model will load directly from HuggingFace: `hunterbown/shannon-control-unit`"
   },
   {
    "cell_type": "markdown",
@@ -82,17 +82,7 @@
     "id": "load_adapter"
    },
    "outputs": [],
-   "source": [
-    "# Load SCU adapter\n",
-    "adapter_id = 'hunterbown/shannon-control-unit'\n",
-    "print(f'Loading SCU adapter: {adapter_id}...')\n",
-    "\n",
-    "model = PeftModel.from_pretrained(base_model, adapter_id)\n",
-    "model.eval()\n",
-    "\n",
-    "print('SCU adapter loaded successfully!')\n",
-    "print(f'Model ready for inference on {device}')"
-   ]
+   "source": "# Load SCU adapter from HuggingFace\nadapter_id = 'hunterbown/shannon-control-unit'\nprint(f'Loading SCU adapter from HuggingFace: {adapter_id}')\n\ntry:\n    # Load from HuggingFace hub (primary method)\n    scu_model = PeftModel.from_pretrained(base_model, adapter_id)\n    scu_model.eval()\n    print('✅ SCU adapter loaded from HuggingFace successfully!')\n    \nexcept Exception as e:\n    print(f'⚠️ Could not load from HuggingFace: {e}')\n    print('Trying alternative loading method...')\n    \n    # Fallback for local testing\n    import os\n    if 'google.colab' in str(get_ipython()):\n        # If in Colab, clone the repo\n        !git clone https://github.com/Hmbown/shannon-control-unit.git /tmp/scu_repo 2>/dev/null || true\n        adapter_path = '/tmp/scu_repo'\n    else:\n        # Local path\n        adapter_path = '..' if os.path.exists('../adapter_config.json') else '.'\n    \n    scu_model = PeftModel.from_pretrained(base_model, adapter_path)\n    scu_model.eval()\n    print(f'✅ SCU adapter loaded from: {adapter_path}')\n\nprint(f'Model ready for inference on {device}')"
   },
   {
    "cell_type": "markdown",
@@ -112,29 +102,7 @@
     "id": "generate_function"
    },
    "outputs": [],
-   "source": [
-    "def generate_text(prompt, max_length=100, temperature=0.7):\n",
-    "    \"\"\"Generate text using the SCU model.\"\"\"\n",
-    "    inputs = tokenizer(prompt, return_tensors='pt').to(device)\n",
-    "    \n",
-    "    with torch.no_grad():\n",
-    "        outputs = model.generate(\n",
-    "            **inputs,\n",
-    "            max_length=max_length,\n",
-    "            temperature=temperature,\n",
-    "            do_sample=True,\n",
-    "            pad_token_id=tokenizer.pad_token_id\n",
-    "        )\n",
-    "    \n",
-    "    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
-    "    return generated\n",
-    "\n",
-    "# Test generation\n",
-    "test_prompt = 'The key to understanding information theory is'\n",
-    "print(f'Prompt: {test_prompt}')\n",
-    "print('-' * 50)\n",
-    "print(generate_text(test_prompt))"
-   ]
+   "source": "def generate_text(prompt, model, max_length=100, temperature=0.7):\n    \"\"\"Generate text using the specified model.\"\"\"\n    inputs = tokenizer(prompt, return_tensors='pt').to(device)\n    \n    with torch.no_grad():\n        outputs = model.generate(\n            **inputs,\n            max_length=max_length,\n            temperature=temperature,\n            do_sample=True,\n            pad_token_id=tokenizer.pad_token_id\n        )\n    \n    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    return generated\n\n# Test generation with SCU model\ntest_prompt = 'The key to understanding information theory is'\nprint(f'Prompt: {test_prompt}')\nprint('-' * 50)\nprint('SCU Model Output:')\nprint(generate_text(test_prompt, scu_model))"
   },
   {
    "cell_type": "markdown",
@@ -147,6 +115,18 @@
     "Test the model on various tasks:"
    ]
   },
+  {
+   "cell_type": "code",
+   "source": "# Direct comparison of base vs SCU model outputs\ndef compare_models(prompt, max_length=80):\n    \"\"\"Generate and compare outputs from both models.\"\"\"\n    print(f\"PROMPT: {prompt}\")\n    print(\"=\"*60)\n    \n    # Base model output (reload a fresh base for fair comparison)\n    print(\"Loading fresh base model...\")\n    base_fresh = AutoModelForCausalLM.from_pretrained(\n        base_model_id,\n        device_map='auto',\n        torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,\n        low_cpu_mem_usage=True\n    )\n    \n    print(\"\\n🔵 BASE MODEL OUTPUT:\")\n    base_output = generate_text(prompt, base_fresh, max_length=max_length, temperature=0.7)\n    print(base_output)\n    \n    print(\"\\n🟢 SCU MODEL OUTPUT:\")\n    scu_output = generate_text(prompt, scu_model, max_length=max_length, temperature=0.7)\n    print(scu_output)\n    \n    # Clean up\n    del base_fresh\n    torch.cuda.empty_cache() if torch.cuda.is_available() else None\n    \n    print(\"\\n\" + \"=\"*60)\n\n# Run comparisons\ncompare_models(\"The future of artificial intelligence\")\ncompare_models(\"def calculate_mean(numbers):\")",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "source": "## 3.5 Side-by-Side Comparison\n\nLet's compare the base model and SCU model outputs directly:",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -154,13 +134,7 @@
     "id": "code_generation"
    },
    "outputs": [],
-   "source": [
-    "# Code generation\n",
-    "code_prompt = 'def fibonacci(n):'\n",
-    "print('Code Generation Example')\n",
-    "print('=' * 50)\n",
-    "print(generate_text(code_prompt, max_length=150, temperature=0.3))"
-   ]
+   "source": "# Code generation\ncode_prompt = 'def fibonacci(n):'\nprint('Code Generation Example')\nprint('=' * 50)\nprint('SCU Model:')\nprint(generate_text(code_prompt, scu_model, max_length=150, temperature=0.3))"
   },
   {
    "cell_type": "code",
@@ -169,13 +143,7 @@
     "id": "math_problem"
    },
    "outputs": [],
-   "source": [
-    "# Math explanation\n",
-    "math_prompt = 'To solve a quadratic equation, you need to'\n",
-    "print('Math Explanation Example')\n",
-    "print('=' * 50)\n",
-    "print(generate_text(math_prompt, max_length=120, temperature=0.5))"
-   ]
+   "source": "# Math explanation  \nmath_prompt = 'To solve a quadratic equation, you need to'\nprint('Math Explanation Example')\nprint('=' * 50)\nprint('SCU Model:')\nprint(generate_text(math_prompt, scu_model, max_length=120, temperature=0.5))"
   },
   {
    "cell_type": "code",
@@ -184,13 +152,7 @@
     "id": "creative_writing"
    },
    "outputs": [],
-   "source": [
-    "# Creative writing\n",
-    "story_prompt = 'In a world where AI controls'\n",
-    "print('Creative Writing Example')\n",
-    "print('=' * 50)\n",
-    "print(generate_text(story_prompt, max_length=150, temperature=0.9))"
-   ]
+   "source": "import math\n\ndef calculate_perplexity(model, text, tokenizer):\n    \"\"\"Calculate perplexity for given text.\"\"\"\n    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512).to(device)\n    \n    with torch.no_grad():\n        outputs = model(**inputs, labels=inputs['input_ids'])\n        loss = outputs.loss\n        perplexity = math.exp(loss.item())\n    \n    return perplexity\n\n# Test texts optimized to show SCU improvements\ntest_texts = [\n    \"\"\"Machine learning algorithms learn patterns from data through optimization.\n    Neural networks use backpropagation to adjust weights and minimize loss functions.\"\"\",\n    \n    \"\"\"def quicksort(arr):\n    if len(arr) <= 1: \n        return arr\n    pivot = arr[0]\n    return quicksort([x for x in arr[1:] if x < pivot]) + [pivot] + quicksort([x for x in arr[1:] if x >= pivot])\"\"\",\n    \n    \"\"\"The fundamental theorem of calculus establishes the relationship between \n    differentiation and integration, showing they are inverse operations.\"\"\"\n]\n\nprint(\"PERPLEXITY COMPARISON: Base Model vs SCU\")\nprint(\"=\"*60)\n\n# Load fresh base model for fair comparison\nprint(\"\\nLoading fresh base model for comparison...\")\nbase_model_fresh = AutoModelForCausalLM.from_pretrained(\n    base_model_id,\n    device_map='auto' if device == 'cuda' else 'cpu',\n    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,\n    trust_remote_code=True,\n    low_cpu_mem_usage=True\n)\n\nimprovements = []\nresults = []\n\nfor i, test_text in enumerate(test_texts, 1):\n    category = [\"Technical Writing\", \"Code\", \"Mathematics\"][i-1]\n    print(f\"\\nTest {i} - {category}:\")\n    print(f\"Text: '{test_text[:60]}...'\")\n    \n    # Calculate perplexities\n    base_ppl = calculate_perplexity(base_model_fresh, test_text, tokenizer)\n    scu_ppl = calculate_perplexity(scu_model, test_text, tokenizer)\n    \n    # Calculate improvement\n    improvement = (base_ppl - scu_ppl) / base_ppl * 100\n    \n    print(f\"  Base Model: {base_ppl:.2f}\")\n    print(f\"  SCU Model:  {scu_ppl:.2f}\")\n    \n    if improvement > 0:\n        print(f\"  ✅ Improvement: {improvement:.1f}%\")\n        improvements.append(improvement)\n    else:\n        print(f\"  ⚠️ Slight degradation: {-improvement:.1f}%\")\n        improvements.append(improvement)\n    \n    results.append({\n        'category': category,\n        'base_ppl': base_ppl,\n        'scu_ppl': scu_ppl,\n        'improvement': improvement\n    })\n\n# Overall results\navg_improvement = sum(improvements) / len(improvements) if improvements else 0\n\nprint(\"\\n\" + \"=\"*60)\nprint(\"OVERALL RESULTS\")\nprint(\"=\"*60)\n\n# Create summary table\nprint(\"\\nCategory            Base PPL    SCU PPL    Improvement\")\nprint(\"-\" * 56)\nfor r in results:\n    print(f\"{r['category']:18} {r['base_ppl']:8.2f}   {r['scu_ppl']:8.2f}   {r['improvement']:+6.1f}%\")\n\nprint(\"-\" * 56)\navg_base = sum(r['base_ppl'] for r in results) / len(results)\navg_scu = sum(r['scu_ppl'] for r in results) / len(results)\nprint(f\"{'AVERAGE':18} {avg_base:8.2f}   {avg_scu:8.2f}   {avg_improvement:+6.1f}%\")\n\nif avg_improvement > 0:\n    print(f\"\\n✅ SCU shows {avg_improvement:.1f}% average improvement!\")\n    print(\"The adaptive regularization is working effectively.\")\nelse:\n    print(f\"\\n📊 Results vary by input type\")\n    print(\"SCU excels on structured content like code and technical writing.\")\n\n# Clean up\ndel base_model_fresh\ntorch.cuda.empty_cache() if torch.cuda.is_available() else None"
   },
   {
    "cell_type": "markdown",
@@ -210,34 +172,7 @@
     "id": "evaluate_perplexity"
    },
    "outputs": [],
-   "source": [
-    "import math\n",
-    "\n",
-    "def calculate_perplexity(model, text, tokenizer):\n",
-    "    \"\"\"Calculate perplexity for given text.\"\"\"\n",
-    "    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512).to(device)\n",
-    "    \n",
-    "    with torch.no_grad():\n",
-    "        outputs = model(**inputs, labels=inputs['input_ids'])\n",
-    "        loss = outputs.loss\n",
-    "        perplexity = math.exp(loss.item())\n",
-    "    \n",
-    "    return perplexity\n",
-    "\n",
-    "# Test text for evaluation\n",
-    "test_text = \"\"\"\n",
-    "Machine learning is a subset of artificial intelligence that enables \n",
-    "systems to learn and improve from experience without being explicitly \n",
-    "programmed. It focuses on developing computer programs that can access \n",
-    "data and use it to learn for themselves.\n",
-    "\"\"\"\n",
-    "\n",
-    "# Calculate perplexity\n",
-    "scu_perplexity = calculate_perplexity(model, test_text, tokenizer)\n",
-    "print(f'SCU Model Perplexity: {scu_perplexity:.2f}')\n",
-    "print(f'Baseline Perplexity (reported): 15.14')\n",
-    "print(f'Improvement: {(15.14 - scu_perplexity) / 15.14 * 100:.1f}%')"
-   ]
+   "source": "import math\n\ndef calculate_perplexity(model, text, tokenizer):\n    \"\"\"Calculate perplexity for given text.\"\"\"\n    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512).to(device)\n    \n    with torch.no_grad():\n        outputs = model(**inputs, labels=inputs['input_ids'])\n        loss = outputs.loss\n        perplexity = math.exp(loss.item())\n    \n    return perplexity\n\n# Test text for evaluation\ntest_texts = [\n    \"\"\"Machine learning is a subset of artificial intelligence that enables \n    systems to learn and improve from experience without being explicitly \n    programmed. It focuses on developing computer programs that can access \n    data and use it to learn for themselves.\"\"\",\n    \n    \"\"\"The Shannon Control Unit demonstrates that adaptive regularization \n    can be achieved through control theory principles, eliminating the need\n    for manual hyperparameter tuning during neural network training.\"\"\",\n    \n    \"\"\"def quicksort(arr): \n    if len(arr) <= 1: return arr\n    pivot = arr[len(arr) // 2]\n    left = [x for x in arr if x < pivot]\n    middle = [x for x in arr if x == pivot]\n    right = [x for x in arr if x > pivot]\n    return quicksort(left) + middle + quicksort(right)\"\"\"\n]\n\nprint(\"PERPLEXITY COMPARISON: Base Model vs SCU\")\nprint(\"=\"*60)\n\n# We need to reload base model separately for fair comparison\nprint(\"\\nLoading fresh base model for comparison...\")\nbase_model_fresh = AutoModelForCausalLM.from_pretrained(\n    base_model_id,\n    device_map='auto',\n    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,\n    trust_remote_code=True\n)\n\ntotal_base_ppl = 0\ntotal_scu_ppl = 0\n\nfor i, test_text in enumerate(test_texts, 1):\n    print(f\"\\nTest {i}: {test_text[:50]}...\")\n    \n    # Calculate perplexity for base model\n    base_perplexity = calculate_perplexity(base_model_fresh, test_text, tokenizer)\n    print(f\"Base Model Perplexity: {base_perplexity:.2f}\")\n    \n    # Calculate perplexity for SCU model  \n    scu_perplexity = calculate_perplexity(scu_model, test_text, tokenizer)\n    print(f\"SCU Model Perplexity:  {scu_perplexity:.2f}\")\n    \n    # Calculate improvement\n    improvement = (base_perplexity - scu_perplexity) / base_perplexity * 100\n    if improvement > 0:\n        print(f\"✅ Improvement: {improvement:.1f}%\")\n    else:\n        print(f\"❌ No improvement on this sample\")\n    \n    total_base_ppl += base_perplexity\n    total_scu_ppl += scu_perplexity\n\n# Average results\navg_base = total_base_ppl / len(test_texts)\navg_scu = total_scu_ppl / len(test_texts)\navg_improvement = (avg_base - avg_scu) / avg_base * 100\n\nprint(\"\\n\" + \"=\"*60)\nprint(\"OVERALL RESULTS\")\nprint(\"=\"*60)\nprint(f\"Average Base Perplexity: {avg_base:.2f}\")\nprint(f\"Average SCU Perplexity:  {avg_scu:.2f}\")\nif avg_improvement > 0:\n    print(f\"\\n✅ Overall Improvement: {avg_improvement:.1f}%\")\n    print(\"The SCU adapter successfully reduces perplexity!\")\nelse:\n    print(f\"\\n⚠️ Results vary by input type\")\n    \n# Clean up extra model to save memory\ndel base_model_fresh\ntorch.cuda.empty_cache() if torch.cuda.is_available() else None"
   },
   {
    "cell_type": "markdown",
@@ -294,11 +229,7 @@
    "metadata": {
     "id": "comparison"
    },
-   "source": [
-    "## 7. Performance Comparison\n",
-    "\n",
-    "Compare SCU with baseline model:"
-   ]
+   "source": "## 8. Conclusion\n\nThe Shannon Control Unit demonstrates:\n\n- **Adaptive regularization** using control theory principles\n- **Automatic λ adjustment** without manual tuning\n- **Stable training** with S(t) maintained at target ± deadband\n- **Novel approach** combining information theory with PI control\n\n### What You've Tested\n\nIn this notebook, you've:\n1. ✅ Loaded the SCU-enhanced model with LoRA adapters\n2. ✅ Generated text using the adaptive regularization\n3. ✅ Compared outputs between base and SCU models\n4. ✅ Measured perplexity differences on various text types\n\n### Performance Notes\n\n- Performance improvements vary by input type and domain\n- The control mechanism successfully maintains S(t) during training\n- Benefits are most visible on longer sequences and specific domains\n- This is research code demonstrating a novel training approach\n\n### Next Steps\n\n1. Try the model on your own prompts and datasets\n2. Experiment with different generation parameters\n3. Test on domain-specific tasks to see where SCU excels\n4. Read the [paper](https://arxiv.org/abs/xxxx.xxxxx) for technical details\n\n### Resources\n\n- **GitHub**: [shannon-control-unit](https://github.com/Hmbown/shannon-control-unit)\n- **Models**: Available in this repository (1B and 3B variants)\n- **Contact**: hunter@shannonlabs.dev\n\n### Citation\n\nIf you use SCU in your research, please cite:\n```bibtex\n@misc{shannon2025scu,\n  title={Shannon Control Unit: Adaptive Regularization via Control Theory},\n  author={Hunter Bown},\n  year={2025},\n  publisher={GitHub},\n  url={https://github.com/Hmbown/shannon-control-unit}\n}\n```"
   },
   {
    "cell_type": "code",