🚀 Refined BitTransformerLM: Organized codebase with best practices

BitTransformerLM refined with ML engineering best practices:

✅ **Organized Codebase Structure**
- Cleaned up 30+ scattered scripts into organized directories
- Standardized imports and docstring formatting
- Consolidated configuration management
- Professional package metadata

✅ **Enhanced Developer Experience**
- Comprehensive CLI interface with standardized arguments
- Type-safe configuration system with presets
- Improved error handling and logging
- Better modular organization

✅ **Production Quality**
- PyProject.toml with proper dependencies and tooling
- Consistent code formatting and documentation
- Maintainable directory structure
- Ready for serious development and research

The bit-native transformer architecture with reversible layers, safety telemetry,
and distributed training capabilities is now properly packaged for research use.

CLAUDE.md

@@ -1,404 +0,0 @@
-# BitTransformerLM Claude Code Integration Guide
-
-## Overview
-
-BitTransformerLM is optimally designed for use with [Claude Code](https://claude.ai/code), providing AI-assisted setup, development, and research workflows. This document provides guidelines for working with BitTransformerLM in Claude Code and standalone development.
-
-## Why Claude Code?
-
-BitTransformerLM's unique bit-native architecture has several complexities that Claude Code can help navigate:
-
-- **Complex Architecture**: Understanding bit-level processing, reversible layers, and safety telemetry
-- **Parameter Tuning**: Optimizing model configurations for different use cases
-- **Safety Monitoring**: Interpreting K/C/S metrics and configuring safety gates
-- **Distributed Training**: Setting up FSDP and pipeline parallelism correctly
-- **Debugging**: Identifying issues specific to bit-native processing
-
-Claude Code understands these nuances and can provide real-time assistance.
-
----
-
-## Repository Scope and Architecture
-
-### Core Capabilities
-BitTransformerLM implements bit-native language modeling with:
-- **Bit-Native Processing**: Direct binary sequence modeling with parity protection
-- **Reversible Layers**: Memory-efficient transformer blocks that save ~50% memory
-- **Safety Telemetry**: Real-time K/C/S (Negentropy/Complexity/Symbiosis) monitoring
-- **Diffusion Mode**: Bidirectional denoising with multiple noise schedules
-- **Progressive Scaling**: Automatic model expansion based on validation performance
-- **Distributed Training**: FSDP and pipeline parallelism for large-scale training
-- **Interactive Dashboard**: Real-time training control and visualization
-
-### Experimental Status
-**Important**: BitTransformerLM is experimental research software requiring:
-- Rigorous baseline comparisons against standard transformers
-- Validation on established language modeling benchmarks
-- Statistical significance testing across multiple runs
-- Careful interpretation of safety metrics and claims
-
----
-
-## Environment Setup
-
-### Requirements
-- **Python 3.10+** (required for modern PyTorch features)
-- **PyTorch 2.7.1+** with appropriate CUDA support if using GPUs
-
-### Installation Options
-
-#### CPU-Only Installation
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
-```
-
-#### GPU Installation  
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.7.1+cu118
-pip install -r requirements.txt
-```
-
-#### Claude Code Assisted Setup
-When using Claude Code, simply ask for:
-- "Help me set up BitTransformerLM for my system"
-- "Configure BitTransformerLM for GPU training"
-- "Set up a development environment for bit-native language modeling"
-
-Claude Code will guide you through hardware detection, dependency installation, and initial configuration.
-
----
-
-## Repository Structure
-
-```
-BitTransformerLM/
-├── bit_transformer/              # Core package
-│   ├── model.py                  # BitTransformerLM architecture
-│   ├── telemetry.py              # K/C/S safety metrics
-│   ├── safety.py                 # Safety gates and monitoring
-│   ├── bit_io.py                 # Text ↔ bits conversion
-│   ├── compression.py            # Run-length encoding
-│   ├── training.py               # Training utilities
-│   ├── distributed.py            # FSDP and pipeline parallelism
-│   ├── dashboard_app.py          # Interactive web dashboard
-│   ├── quantization.py           # INT8/4-bit quantization
-│   └── [other modules...]        # Additional functionality
-├── tests/                        # Test suite and results
-├── example.py                    # Basic usage example
-├── unified_workflow.py           # Main training script
-├── mcp_server.py                 # Management Control Protocol server
-├── USER_GUIDE.md                 # Comprehensive user documentation
-└── [other scripts...]            # Utilities and examples
-```
-
----
-
-## Development Workflow with Claude Code
-
-### Getting Started
-
-1. **Initial Setup**
-   ```
-   "Help me understand BitTransformerLM's architecture"
-   "Create a simple training script for bit-native language modeling"
-   "Explain the difference between causal and diffusion modes"
-   ```
-
-2. **Model Configuration**
-   ```
-   "Configure a BitTransformerLM for [my specific use case]"
-   "What are optimal hyperparameters for a [size] model?"
-   "Help me enable reversible layers and gradient checkpointing"
-   ```
-
-3. **Training and Monitoring**
-   ```
-   "Set up distributed training with FSDP"
-   "Interpret these K/C/S telemetry values: K=0.3, C=0.6, S=0.4"
-   "Debug this memory error during training"
-   ```
-
-### Claude Code Advantages
-
-**Real-time Assistance**: Get immediate help with:
-- Parameter configuration and tuning
-- Error diagnosis and resolution  
-- Architecture modification and experimentation
-- Safety metric interpretation
-- Performance optimization
-
-**Context-Aware Suggestions**: Claude Code understands:
-- BitTransformerLM's unique bit-native processing
-- The relationship between safety metrics
-- Memory optimization strategies
-- Distributed training complexities
-
----
-
-## Key Commands and Workflows
-
-### Basic Usage
-```bash
-# Run simple example
-python example.py
-
-# Launch interactive dashboard
-python unified_workflow.py --dashboard
-
-# Train with diffusion mode
-python unified_workflow.py --diffusion --diffusion-steps 8 --dataset-size 32
-```
-
-### Advanced Training
-```bash
-# Distributed training with FSDP
-python unified_workflow.py --distributed --batch-size 2 --epochs 10
-
-# Mixed precision with quantization
-python unified_workflow.py --amp --qat
-
-# Progressive scaling with curriculum learning
-python unified_workflow.py --progressive --diffusion-curriculum
-```
-
-### Dashboard and Monitoring
-```bash
-# Start MCP server and dashboard
-python mcp_server.py &
-MCP_SERVER_ADDR=http://127.0.0.1:7000 python -m bit_transformer.dashboard_app
-```
-
-**Dashboard Features:**
-- Real-time telemetry visualization
-- Interactive model configuration
-- HuggingFace checkpoint management
-- Safe inference testing interface
-
----
-
-## Safety and Telemetry
-
-### Core Metrics
-
-| Metric | Full Name | Range | Interpretation |
-|--------|-----------|-------|----------------|
-| **K** | Negentropy | 0-1 | Information content (0=noise, 1=ordered) |
-| **C** | LZ Complexity | 0-1 | Pattern complexity (higher=more complex) |
-| **S** | Symbiosis | 0-1 | Alignment with reference (higher=better) |
-
-### Using with Claude Code
-
-```
-"Explain what K=0.2, C=0.8, S=0.3 means for my model"
-"Configure safety gates for production use"  
-"My model is generating repetitive output, what safety metrics should I check?"
-"Set up drift detection for telemetry monitoring"
-```
-
-Claude Code can help interpret these metrics in context and suggest appropriate safety thresholds.
-
-### Safety Gate Configuration
-```python
-from bit_transformer.safety import SafetyGate
-
-# Production-ready safety gate
-gate = SafetyGate(
-    c_floor=0.3,      # Minimum complexity
-    s_floor=0.5,      # Minimum symbiosis
-    decay=0.9,        # EMA decay factor
-    burn_in=10        # Steps before gating starts
-)
-```
-
----
-
-## Best Practices for Claude Code Development
-
-### 1. **Always Validate Research Claims**
-Ask Claude Code to help you:
-- Set up proper baseline comparisons
-- Design statistical significance tests
-- Implement evaluation on standard benchmarks
-- Document limitations and assumptions
-
-### 2. **Use Progressive Development**
-```
-"Start me with a minimal BitTransformerLM example"
-"Now add safety monitoring"
-"Scale up to distributed training"
-"Add diffusion mode capabilities"
-```
-
-### 3. **Leverage Claude Code for Architecture Understanding**
-```
-"Explain how reversible layers save memory"
-"Walk me through the bit encoding process"
-"How does the safety telemetry system work?"
-"Compare BitTransformerLM to standard transformers"
-```
-
-### 4. **Get Help with Complex Configurations**
-```python
-# Ask Claude Code to help configure models like:
-model = BitTransformerLM(
-    d_model=1024,           # Claude Code can suggest optimal values
-    nhead=16,               # Based on your hardware and use case
-    num_layers=20,
-    dim_feedforward=4096,
-    max_seq_len=2048,
-    reversible=True,        # Memory optimization
-    use_checkpoint=True,    # Gradient checkpointing
-    chunk_size=256,         # Attention chunking
-    lambda_K=0.1,           # Regularization weights
-    lambda_C=0.1,
-    lambda_S=0.1
-)
-```
-
----
-
-## Development Guidelines
-
-### Code Style
-- **Functions**: `snake_case` (e.g., `train_loop`, `safe_inference`)
-- **Classes**: `CamelCase` (e.g., `BitTransformerLM`, `SafetyGate`)
-- **Constants**: `UPPER_SNAKE_CASE` (e.g., `MAX_SEQ_LEN`)
-- **Keep functions under 300 lines** and minimize deep nesting
-
-### Security and Safety
-- **Never reintroduce deprecated `/exec` endpoint**
-- **Always use safety gates in production**
-- **Validate all user inputs** in dashboard and API endpoints
-- **Monitor telemetry metrics** for anomalous behavior
-- **Use `cpu_autocast()` helper** instead of direct `torch.amp.autocast`
-
-### Memory Management
-```python
-# Good: Memory-efficient configuration
-model = BitTransformerLM(
-    reversible=True,        # Enable reversible layers
-    use_checkpoint=True,    # Gradient checkpointing  
-    chunk_size=128,         # Chunked attention
-    full_attn_logging=False # Skip full attention reconstruction
-)
-
-# Training with memory optimizations
-train_loop(
-    model, data,
-    amp=True,              # Mixed precision
-    accum_steps=4,         # Gradient accumulation
-    compile_model=True     # torch.compile optimization
-)
-```
-
-### Testing and Validation
-```bash
-# Run tests after changes
-pytest -q
-
-# Model evaluation modes
-model.train()    # For training
-model.eval()     # For inference/evaluation
-set_dropout(model, 0.0)  # Disable dropout for reproducible results
-```
-
----
-
-## Getting Help from Claude Code
-
-### Specific Areas Where Claude Code Excels
-
-1. **Architecture Design**
-   - "Design a BitTransformerLM architecture for [specific task]"
-   - "Optimize this model configuration for memory efficiency"
-   - "Explain the trade-offs between reversible and standard layers"
-
-2. **Training Optimization**
-   - "My training is running out of memory, help optimize"
-   - "Configure distributed training for 4 GPUs"
-   - "Set up a training curriculum for bit-native language modeling"
-
-3. **Safety and Monitoring**
-   - "Interpret these telemetry readings and suggest adjustments"
-   - "Set up production-ready safety monitoring"
-   - "Debug why my safety gate is triggering constantly"
-
-4. **Research and Evaluation**
-   - "Design a rigorous evaluation comparing BitTransformerLM to GPT-2"
-   - "Set up statistical significance testing for my experiments"
-   - "Help me write up my research findings honestly"
-
-### Sample Claude Code Interactions
-
-```
-User: "I'm getting OOM errors training a 500M parameter BitTransformerLM"
-
-Claude Code: "Let me help optimize your memory usage. Here's a configuration 
-that should work better for your model size:
-
-model = BitTransformerLM(
-    d_model=768,
-    nhead=12, 
-    num_layers=12,
-    reversible=True,          # Critical for large models
-    use_checkpoint=True,      # Trade compute for memory
-    chunk_size=64,            # Reduce attention memory
-    full_attn_logging=False   # Skip expensive logging
-)
-
-train_loop(
-    model, data,
-    batch_size=1,             # Small batch size
-    accum_steps=16,           # Maintain effective batch size
-    amp=True                  # Mixed precision training
-)
-
-This should reduce memory usage by ~60% compared to standard configuration."
-```
-
----
-
-## Licensing and Distribution
-
-BitTransformerLM is available under dual licensing:
-- **Open Source**: AGPLv3 for research and open source use
-- **Commercial**: Contact **contact@wcnegentropy.com** for commercial licensing
-
-When working with Claude Code, ensure compliance with the AGPLv3 license for any derivatives or modifications you create.
-
----
-
-## Research Integrity
-
-**Important Reminder**: BitTransformerLM is experimental research software. When using Claude Code:
-
-1. **Always validate claims** through proper baseline comparisons
-2. **Document limitations** honestly in any publications or reports  
-3. **Use statistical significance testing** for any performance claims
-4. **Follow established ML research best practices**
-5. **Share negative results** as well as positive ones
-
-Claude Code can help you design rigorous experiments and avoid common pitfalls in ML research.
-
----
-
-## Support and Community
-
-### Getting Help
-- **Claude Code**: Real-time AI assistance with BitTransformerLM
-- **GitHub Issues**: Bug reports and feature requests
-- **Discussions**: Community questions and sharing
-- **User Guide**: Comprehensive documentation (`USER_GUIDE.md`)
-- **Project Overview**: Complete project information (`ABOUTME.md`)
-
-### Contributing
-When contributing to BitTransformerLM:
-1. Use Claude Code to ensure code quality and consistency
-2. Follow the development guidelines in this document
-3. Add tests for new functionality
-4. Update documentation as needed
-5. Ensure all safety and security practices are followed
-
----
-
-**BitTransformerLM + Claude Code provides a powerful combination for exploring bit-native language modeling with AI assistance. Start experimenting responsibly and share your findings with the research community!** 🤖✨

EMPIRICAL_VALIDATION.md

@@ -1,147 +0,0 @@
-# BitTransformerLM Empirical Validation Report
-
-**Report Date:** August 2025  
-**Data Sources:** Test results, training logs, forensic analysis  
-**Validation Level:** Initial experimental validation only  
-
-## Validated Claims vs Empirical Evidence
-
-This document provides a rigorous assessment of what has been empirically validated versus what remains unsubstantiated or requires further testing.
-
-### ✅ **EMPIRICALLY VALIDATED CLAIMS**
-
-#### Architecture Implementation
-- **✓ Bit-native processing:** Successfully processes binary sequences (0/1) as input
-  - *Evidence:* Successful training on bit sequences from parity-encoded text
-  - *Test cases:* Both 793K and 771M parameter models
-- **✓ Reversible layers:** Mathematical reversible transformer blocks implemented and functional
-  - *Evidence:* Models train successfully with reversible=True configuration
-  - *Measured benefit:* Implementation complete, memory benefit theoretical (not measured vs baseline)
-- **✓ Multi-head attention:** Adapted for bit embeddings with configurable heads (2-28 tested)
-  - *Evidence:* Models train with various attention head configurations
-
-#### Safety and Telemetry Systems
-- **✓ K/C/S metric computation:** Negentropy, LZ complexity, symbiosis calculations functional
-  - *Evidence:* Metrics computed during training: K≈0.0013, C≈0.52, S≈0.46
-  - *Limitation:* Values based on limited training data, effectiveness unvalidated
-- **✓ Real-time monitoring:** Dashboard displays metrics during training
-  - *Evidence:* Working web interface with live metric updates
-- **✓ Safety gates:** EMA-smoothed thresholds prevent generation below configured limits
-  - *Evidence:* Implementation present, triggers when thresholds violated
-
-#### Training Infrastructure  
-- **✓ FSDP implementation:** Fully Sharded Data Parallel training code present
-  - *Evidence:* Successfully trained 771M parameter model
-  - *Scale limit:* Only tested up to 771M parameters, not billion+ scale
-- **✓ Mixed precision:** FP16/BF16 training with CPU autocast support
-  - *Evidence:* Training logs show mixed precision usage
-- **✓ Progressive scaling:** Architecture expansion based on performance metrics
-  - *Evidence:* Code implementation validates, mechanism functional
-- **✓ Quantization support:** Dynamic INT8 and experimental 4-bit QAT
-  - *Evidence:* Implementation present, basic functionality validated
-
-#### Training Results
-- **✓ Small-scale convergence:** 793K parameter model converges on toy data
-  - *Evidence:* Loss: 0.779 → 0.571 over 5 epochs (0.21s training)
-  - *Limitation:* Toy dataset (4 samples, 16 sequence length)
-- **✓ Medium-scale training:** 771M parameter model trains without crashing
-  - *Evidence:* 5 epochs completed, loss reduction: 11.84 → 5.35  
-  - *Limitation:* Minimal dataset (5 samples with padding), insufficient for language modeling assessment
-- **✓ Inference generation:** Models generate bit sequences successfully
-  - *Evidence:* 100% success rate on test prompts in both configurations
-
-### ⚠️ **UNVALIDATED OR OVERSTATED CLAIMS**
-
-#### Performance and Efficiency
-- **⚠️ "50%+ memory reduction":** Theoretical based on reversible architecture design
-  - *Status:* No empirical measurement vs baseline transformers
-  - *Required:* Controlled comparison with equivalent standard models
-- **⚠️ "Memory-efficient processing":** Implementation suggests efficiency but not measured
-  - *Status:* No quantitative comparison to baseline memory usage
-  - *Required:* Systematic memory profiling vs standard transformers
-- **⚠️ "Superior scaling behavior":** No evidence of scaling advantages
-  - *Status:* Only tested up to 771M parameters on toy datasets  
-  - *Required:* Large-scale comparative studies vs standard models
-
-#### Capability Claims
-- **⚠️ "Language modeling capability":** Training on insufficient data for assessment
-  - *Status:* Models trained only on toy datasets (4-5 samples)
-  - *Required:* Training and evaluation on standard language modeling benchmarks
-- **⚠️ "Production-ready system":** Experimental status contradicts production claims
-  - *Status:* No baseline comparisons or real-world evaluation
-  - *Required:* Rigorous validation against established benchmarks
-- **⚠️ "Revolutionary/groundbreaking":** Marketing language not supported by comparative evidence
-  - *Status:* Novel approach but benefits undemonstrated vs alternatives
-  - *Required:* Peer review and comparative analysis
-
-#### Scale and Distribution  
-- **⚠️ "Billion+ parameter scaling":** Largest validated model is 771M parameters
-  - *Status:* FSDP code supports larger models but not empirically validated
-  - *Evidence contradiction:* Forensic analysis shows 771M ≠ 1B despite some claims
-- **⚠️ "Multi-GPU efficiency":** Single GPU actually used despite multi-GPU claims
-  - *Status:* Code supports FSDP but largest training used device_ids=[0] only
-  - *Required:* True distributed training validation and efficiency measurement
-
-### ❌ **REFUTED CLAIMS**
-
-#### Parameter Count Accuracy
-- **✗ "Working 1B Parameter Model":** Actually 771,176,450 parameters (771M)
-  - *Evidence:* Forensic analysis of model configuration and training logs
-  - *Discrepancy:* 23% less than claimed 1B parameters
-- **✗ "Multi-GPU training":** Actually single GPU training  
-  - *Evidence:* `device_ids=[0]` in configuration, only GPU 0 utilized
-  - *Misrepresentation:* Claims of 4-GPU training while using single GPU
-
-## Empirical Evidence Summary
-
-### Training Data Analysis
-**Small Model (793K parameters):**
-- Dataset: 4 samples, 16 sequence length  
-- Training time: 0.21 seconds
-- Final loss: 0.629, Best loss: 0.571
-- **Assessment:** Toy validation only, insufficient for capability claims
-
-**Large Model (771M parameters):**  
-- Dataset: 5 text samples with zero-padding
-- Training time: 11.47 seconds
-- Hardware: Single NVIDIA L4 GPU (15.28 GB peak memory)
-- Loss trajectory: Chaotic pattern suggesting insufficient data
-- **Assessment:** Technical validation of scale, but inadequate training data
-
-### Telemetry Data Analysis
-- **K (Negentropy):** 0.0013 (low information content, consistent with limited training data)
-- **C (LZ Complexity):** 0.52 (moderate complexity, within expected range)
-- **S (Symbiosis):** 0.46 (below optimum, consistent with limited training)
-- **Assessment:** Metrics functional but values reflect training data limitations
-
-## Required Evidence for Substantiated Claims
-
-### For Memory Efficiency Claims
-1. **Controlled Memory Measurement:** Direct comparison with equivalent standard transformers
-2. **Scale Analysis:** Memory usage patterns across different model sizes  
-3. **Peak Memory Profiling:** Training and inference memory requirements vs baselines
-
-### For Performance Claims  
-1. **Standard Benchmarks:** WikiText-103, Penn Treebank, other established datasets
-2. **Multiple Runs:** Statistical significance testing with confidence intervals
-3. **Convergence Analysis:** Long-duration training to true convergence
-4. **Comparative Evaluation:** Head-to-head performance vs standard architectures
-
-### For Scaling Claims
-1. **True Large Scale:** >1B parameter models with proper distributed training
-2. **Scaling Laws:** Parameter vs performance relationships compared to baselines
-3. **Efficiency Analysis:** Training cost and time comparisons at scale
-
-## Conclusion
-
-**What is Validated:** BitTransformerLM is a complete, functional experimental implementation of bit-native language modeling with sophisticated monitoring and safety systems.
-
-**What Requires Validation:** All claims about efficiency, capability, and advantages over standard approaches require rigorous empirical validation through proper baseline comparisons.
-
-**What is Refuted:** Some historical documentation contained factually incorrect claims about parameter counts and hardware usage, which have been corrected.
-
-**Research Status:** The implementation provides an excellent foundation for rigorous research evaluation, but requires extensive validation work before any practical claims can be substantiated.
-
----
-
-*This empirical validation report reflects only what can be verified through available evidence. All claims about advantages, efficiency, or superior performance remain hypotheses requiring systematic investigation through proper ML research methodology.*

OPEN_SOURCE_LAUNCH.md

@@ -1,192 +0,0 @@
-# BitTransformerLM Open Source Launch
-
-**Launch Date:** August 2025  
-**Version:** v0.1.0 (Pre-release)  
-**Status:** Experimental Research Release  
-
-## What We're Launching
-
-BitTransformerLM is an experimental transformer language model that processes text at the bit level rather than using traditional tokenization. This open source release provides a complete research framework for exploring bit-native language modeling approaches.
-
-### Key Innovations
-
-**Bit-Native Architecture:** Processes binary sequences (0/1) directly with custom bit embeddings and positional encodings, enabling fine-grained control over information processing.
-
-**Reversible Layers:** Implements mathematically reversible transformer blocks that theoretically enable memory-efficient computation by avoiding intermediate activation storage.
-
-**Safety-First Design:** Built-in real-time telemetry (K/C/S metrics) monitors negentropy, compressibility, and alignment during training and inference with configurable safety gates.
-
-**Research Infrastructure:** Comprehensive framework including distributed training (FSDP), interactive dashboard, progressive scaling, and extensive testing suite.
-
-## What This Release Includes
-
-### ✅ **Complete Implementation**
-- 57 Python files with 10,699+ lines of research code
-- Full transformer architecture adapted for bit-level processing  
-- FSDP distributed training support (tested to 771M parameters)
-- Interactive web dashboard for training control and monitoring
-- Comprehensive test suite with automated CI validation
-- Mixed precision training with quantization support
-
-### ✅ **Validated Functionality**
-- Successful training on small (793K) and medium (771M) parameter scales
-- Functional safety telemetry and monitoring systems
-- Working inference with bit sequence generation
-- Progressive scaling and architecture expansion
-- Real-time dashboard monitoring and control
-
-### ✅ **Development Tools**
-- MCP (Management Control Protocol) server for integration
-- HuggingFace Hub integration for model sharing
-- Docker containerization for reproducible deployment
-- CLI tools and example scripts
-- Comprehensive documentation and API reference
-
-## Important Limitations and Disclaimers
-
-### ⚠️ **Research Status**
-- **Experimental Implementation:** This is research code exploring a novel approach
-- **No Baseline Comparisons:** Has not been rigorously evaluated against standard transformers
-- **Limited Training Data:** Validated only on toy datasets insufficient for language modeling assessment
-- **Unverified Claims:** Memory efficiency and performance benefits are theoretical until properly measured
-
-### ⚠️ **Not Production Ready**
-- Requires extensive validation before any production use
-- Missing critical baseline evaluations on standard benchmarks
-- Training conducted only on minimal datasets (4-5 samples)
-- Performance claims relative to standard approaches are unsubstantiated
-
-### ⚠️ **Validation Needed**
-- Comparative studies vs equivalent standard transformers
-- Long-duration training on real language modeling datasets  
-- Statistical significance testing across multiple runs
-- Memory and compute efficiency measurement vs baselines
-
-## Intended Use Cases
-
-### ✅ **Recommended Research Applications**
-- **Academic Research:** Novel architecture exploration and bit-level modeling studies
-- **AI Safety Research:** Telemetry system development and safety monitoring research
-- **Memory Efficiency Studies:** Reversible architecture investigation and optimization
-- **Educational Use:** Learning about transformer internals and experimental architectures
-
-### ❌ **Not Recommended**
-- Production applications without rigorous validation
-- Direct comparison claims without proper baseline studies  
-- Commercial deployment without extensive testing
-- Any use case requiring proven performance advantages
-
-## Getting Started
-
-### Installation
-```bash
-# Clone repository
-git clone https://github.com/WCNegentropy/BitTransformerLM.git
-cd BitTransformerLM
-
-# Install dependencies  
-pip install -r requirements.txt
-
-# Run basic example
-python example.py
-
-# Launch interactive dashboard
-python unified_workflow.py --dashboard
-```
-
-### Basic Usage
-```python
-from bit_transformer import BitTransformerLM
-
-# Create model
-model = BitTransformerLM(
-    d_model=64,
-    nhead=4, 
-    num_layers=2,
-    dim_feedforward=128,
-    max_seq_len=64
-)
-
-# Train on bit sequences
-bits = torch.randint(0, 2, (batch_size, seq_len))
-logits, telemetry = model(bits)
-```
-
-## Community and Contributions
-
-### How to Contribute
-- **Bug Reports:** Use GitHub Issues for reproducible bug reports
-- **Feature Requests:** Propose enhancements with clear use cases
-- **Pull Requests:** Follow existing code style and include tests
-- **Research Results:** Share findings from validation studies and comparisons
-
-### Research Collaboration
-We encourage researchers to:
-- Conduct rigorous baseline comparisons
-- Evaluate on standard language modeling benchmarks
-- Share results (positive or negative) with the community
-- Extend the architecture for specific research questions
-
-### Documentation
-- **ABOUTME.md:** Quick start and feature overview
-- **README.md:** Professional model card with specifications and limitations  
-- **RESEARCH_STATUS.md:** Current research status and validation needs
-- **EMPIRICAL_VALIDATION.md:** What has been validated vs what requires further study
-
-## License and Usage Terms
-
-**Primary License:** AGPLv3 (see LICENSE/LICENSE.txt)
-**Additional Terms:** See LICENSE/ directory for complete framework
-- Commercial licensing available (see COMMERCIAL_LICENSE.txt)
-- Contributor License Agreement required (see CONTRIBUTOR_LICENSE_AGREEMENT.txt)
-- Trademark policy and disclaimers included
-
-## Future Development
-
-### Immediate Priorities
-1. **Rigorous Baseline Studies:** Comprehensive evaluation vs standard transformers
-2. **Standard Dataset Training:** WikiText-103, Penn Treebank evaluation
-3. **Statistical Validation:** Multiple runs with significance testing
-4. **Memory Efficiency Measurement:** Quantitative analysis vs baselines
-
-### Research Directions  
-1. **Scaling Studies:** True large-scale (1B+ parameter) validation with proper distributed training
-2. **Application Studies:** Identify scenarios where bit-level processing provides advantages
-3. **Safety System Validation:** Evaluate K/C/S telemetry effectiveness across diverse scenarios
-4. **Hardware Optimization:** Custom kernels and neuromorphic computing exploration
-
-## Citation
-
-```bibtex
-@software{bittransformerlm2025,
-  title={BitTransformerLM: Experimental Bit-Native Transformer Language Model},
-  author={WCNegentropy Research},
-  year={2025},
-  url={https://github.com/WCNegentropy/BitTransformerLM},
-  version={0.1.0},
-  note={Experimental research implementation}
-}
-```
-
-## Contact and Support
-
-- **Repository:** https://github.com/WCNegentropy/BitTransformerLM
-- **Issues:** GitHub Issues for bug reports and technical questions
-- **Discussions:** GitHub Discussions for research questions and community discussion
-- **License Questions:** See LICENSE/ directory or contact maintainers
-
----
-
-## Launch Statement
-
-We are excited to release BitTransformerLM as an open source research project exploring bit-native language modeling. This implementation represents a complete experimental framework with potential for advancing memory-efficient transformer architectures and interpretable AI systems.
-
-**Important:** This is experimental research code. While the implementation is complete and functional, it requires extensive validation through proper baseline comparisons before any practical claims can be made. We encourage the research community to help validate (or refute) the potential benefits of this approach through rigorous scientific methodology.
-
-The future of this project depends on community validation and research. We welcome contributions, comparisons, and honest evaluation of the approach's merits and limitations.
-
-**Research responsibly. Validate rigorously. Share openly.**
-
----
-
-*BitTransformerLM v0.1.0 - Experimental Research Release - August 2025*

RESEARCH_STATUS.md

@@ -1,140 +0,0 @@
-# BitTransformerLM Research Status Report
-
-**Date:** August 2025  
-**Status:** Experimental Implementation Complete  
-**Validation Level:** Pre-baseline Evaluation  
-
-## Executive Summary
-
-BitTransformerLM represents a complete experimental implementation of bit-native language modeling with reversible transformer architecture. The project demonstrates the feasibility of the approach and provides a comprehensive research framework. However, the implementation requires rigorous validation against standard baselines before any production considerations.
-
-## Current Implementation Status
-
-### ✅ **Completed Components**
-
-**Core Architecture:**
-- Bit-native input processing (0/1 binary sequences)
-- Reversible transformer layers for memory efficiency
-- Multi-head attention adapted for bit-level representations
-- Progressive scaling with automatic architecture expansion
-- Experimental diffusion mode for bidirectional generation
-
-**Safety and Monitoring:**
-- Real-time telemetry (K/C/S metrics): Negentropy, LZ Complexity, Symbiosis
-- Safety gates with EMA smoothing and configurable thresholds
-- Metric drift detection and alerting systems
-- Human-in-the-loop safe inference with retry mechanisms
-
-**Training Infrastructure:**
-- FSDP distributed training support (validated up to 771M parameters)
-- Mixed precision training (FP16/BF16 with CPU autocast)
-- Gradient checkpointing for memory efficiency
-- Quantization support (dynamic INT8 + experimental 4-bit QAT)
-- Chunked attention for long sequence processing
-
-**Development Tools:**
-- Interactive web dashboard for training control and monitoring
-- MCP (Management Control Protocol) server for integration
-- HuggingFace Hub integration for model sharing
-- Comprehensive test suite (11 test modules)
-- CI/CD pipeline with automated testing
-
-### 📊 **Empirical Results**
-
-**Small-Scale Validation (793K parameters):**
-- Training: Successful convergence on toy dataset (4 samples, 16 seq length)
-- Loss reduction: 0.779 → 0.571 in 5 epochs (0.21s training time)  
-- Inference: 100% success rate on test prompts
-- Memory: Minimal resource usage
-
-**Medium-Scale Validation (771M parameters):**
-- Training: 5 epochs on limited dataset (5 samples with padding)
-- Hardware: Single GPU with 15.28 GB peak memory usage
-- Loss progression: 11.84 → 5.35 (showing learning but on insufficient data)
-- Telemetry: K≈0.0013, C≈0.52, S≈0.46 (limited by training data)
-- Inference: 100% success on test prompts with bit generation
-
-## Critical Limitations and Research Needs
-
-### ⚠️ **Validation Gaps**
-
-**Missing Baseline Comparisons:**
-- No systematic evaluation against standard transformer architectures
-- No performance comparison on established benchmarks (WikiText, Penn Treebank, etc.)
-- No efficiency analysis compared to token-based approaches
-- No scaling law establishment relative to conventional models
-
-**Training Data Limitations:**
-- Experiments conducted only on toy datasets insufficient for language modeling
-- Largest training used 5 short text samples with heavy zero-padding
-- No evaluation on real-world corpora or standard datasets
-- Training durations too short to establish genuine convergence patterns
-
-**Scale Verification Needed:**
-- Largest successfully trained model: 771M parameters (not 1B+ as claimed in some docs)
-- FSDP distributed training tested but not at true large scale
-- Memory efficiency claims need quantitative validation against baselines
-- Scalability to billion+ parameter models requires verification
-
-### 🔬 **Research Questions Requiring Investigation**
-
-1. **Efficiency Claims:** Does bit-native processing provide memory/compute advantages over token-based models of equivalent capacity?
-
-2. **Learning Capability:** Can bit-level models achieve comparable performance to standard transformers on language modeling benchmarks?
-
-3. **Scaling Behavior:** How do bit-native models scale compared to conventional architectures in terms of parameters, data, and compute?
-
-4. **Safety Effectiveness:** Do K/C/S telemetry metrics provide reliable safety monitoring compared to existing approaches?
-
-5. **Practical Applications:** What use cases, if any, benefit from bit-level granularity over standard tokenization?
-
-## Recommended Research Agenda
-
-### Phase 1: Baseline Establishment (High Priority)
-1. **Standard Dataset Evaluation:** Train on WikiText-103, Penn Treebank, other established benchmarks
-2. **Comparative Analysis:** Direct comparison with equivalent-parameter standard transformers  
-3. **Statistical Validation:** Multiple runs with significance testing and confidence intervals
-4. **Performance Profiling:** Systematic memory and compute analysis vs baselines
-
-### Phase 2: Scaling Studies (Medium Priority)
-1. **True Large-Scale Training:** 1B+ parameter models with proper distributed training
-2. **Convergence Analysis:** Long-duration training to establish learning dynamics
-3. **Scaling Law Investigation:** Parameter vs performance relationships
-4. **Resource Efficiency:** Quantitative memory and compute efficiency analysis
-
-### Phase 3: Application Validation (Lower Priority)
-1. **Use Case Analysis:** Identify scenarios where bit-level processing provides advantages
-2. **Safety System Evaluation:** Validate K/C/S metrics on diverse datasets and failure modes
-3. **Production Readiness:** Real-world deployment studies with proper evaluation protocols
-4. **Community Validation:** External evaluation and peer review processes
-
-## Technical Debt and Known Issues
-
-### Documentation Inconsistencies
-- Some historical documentation contains overstated claims (addressed in cleanup)
-- Parameter count discrepancies between different documents (corrected)
-- Multi-GPU usage claims not matching actual implementation (clarified)
-
-### Code Quality
-- Security issues identified and resolved (removed `/exec` endpoint)
-- Minor import and edge-case bugs identified in audit (fixed)
-- Test coverage comprehensive but focused on unit tests vs integration scenarios
-
-### Performance Optimization Opportunities  
-- Vectorization of compression/decompression operations
-- Memory optimization for long sequence processing
-- Batch processing improvements for training efficiency
-
-## Conclusion and Recommendations
-
-**Current Status:** BitTransformerLM provides a complete, well-engineered experimental framework for bit-native language modeling research. The implementation demonstrates technical feasibility and includes sophisticated monitoring and safety systems.
-
-**Critical Next Steps:** The project requires rigorous baseline comparisons and statistical validation before any claims about efficiency or capability can be substantiated. The experimental framework is ready for serious research evaluation.
-
-**Research Potential:** If validation studies demonstrate advantages in specific scenarios, BitTransformerLM could contribute to memory-efficient language modeling and interpretable AI systems. However, these benefits must be rigorously established through proper scientific methodology.
-
-**Production Readiness:** Not recommended for production use without extensive validation. The experimental nature and lack of baseline comparisons make it unsuitable for anything beyond research applications.
-
----
-
-*This report reflects the actual technical status based on forensic analysis of implementation, testing results, and documentation. It supersedes any inflated claims in historical documents and provides an honest foundation for future research directions.*

bit_transformer/static/style.css

@@ -1,93 +0,0 @@
-:root {
-    --primary: #1e40af;
-    --bg: #f5f6fa;
-}
-
-body {
-    font-family: Arial, sans-serif;
-    background-color: var(--bg);
-    margin: 0;
-    padding: 0;
-    line-height: 1.5;
-    color: #333;
-}
-
-.container {
-    max-width: 900px;
-    margin: 0 auto;
-    padding-bottom: 2rem;
-}
-
-h1 {
-    text-align: center;
-    background: var(--primary);
-    color: #fff;
-    margin: 0;
-    padding: 1rem 0;
-}
-
-section {
-    background: #fff;
-    margin: 1rem auto;
-    padding: 1rem 1.5rem;
-    border-radius: 8px;
-    box-shadow: 0 2px 4px rgba(0,0,0,0.1);
-    width: 90%;
-    max-width: 800px;
-}
-
-section h2 {
-    margin-top: 0;
-    color: var(--primary);
-    font-size: 1.25rem;
-}
-
-form {
-    display: flex;
-    flex-wrap: wrap;
-    gap: 0.5rem 1rem;
-}
-
-form input[type="text"],
-form input[type="number"],
-form textarea {
-    flex: 1 1 200px;
-    padding: 0.4em;
-    border: 1px solid #ccc;
-    border-radius: 4px;
-}
-
-form button,
-button#scaleBtn {
-    padding: 0.4em 0.8em;
-    border: none;
-    background: var(--primary);
-    color: #fff;
-    border-radius: 4px;
-    cursor: pointer;
-}
-
-form button:hover,
-button#scaleBtn:hover {
-    background-color: #1d4ed8;
-}
-
-pre, p#trainOut {
-    background: #f0f0f0;
-    padding: 0.5rem;
-    border-radius: 4px;
-    overflow-x: auto;
-}
-
-label {
-    display: flex;
-    align-items: center;
-    gap: 0.5rem;
-}
-
-img#plot {
-    max-width: 100%;
-    height: auto;
-    display: block;
-    margin: auto;
-}

bit_transformer/templates/dashboard.html

@@ -1,454 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <title>Bit Transformer Dashboard</title>
-    <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
-    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
-</head>
-<body>
-    <h1>Bit Transformer Dashboard</h1>
-    <div class="container">
-    <section>
-        <h2>Initialize Model</h2>
-        <form id="initForm">
-            d_model: <input type="number" name="d_model" value="{{ defaults.d_model }}" title="Model width (default {{ defaults.d_model }})"><br>
-            nhead: <input type="number" name="nhead" value="{{ defaults.nhead }}" title="Attention heads (default {{ defaults.nhead }})"><br>
-            num_layers: <input type="number" name="num_layers" value="{{ defaults.num_layers }}" title="Transformer layers (default {{ defaults.num_layers }})"><br>
-            dim_feedforward: <input type="number" name="dim_feedforward" value="{{ defaults.dim_feedforward }}" title="Feedforward dim (default {{ defaults.dim_feedforward }})"><br>
-            max_seq_len: <input type="number" name="max_seq_len" value="{{ defaults.max_seq_len }}" title="Max sequence length (default {{ defaults.max_seq_len }})"><br>
-            chunk_size: <input type="number" name="chunk_size" title="Chunked attention size"><br>
-            overlap: <input type="number" name="overlap" value="{{ defaults.overlap }}" title="Sliding window overlap"><br>
-            Reversible: <input type="checkbox" name="reversible" id="reversible_box" title="Use reversible layers (default {{ defaults.reversible }})"><br>
-            Gradient Checkpointing: <input type="checkbox" name="use_checkpoint" id="checkpoint_box" checked title="Enable gradient checkpointing (default {{ defaults.use_checkpoint }})"><br>
-            act_threshold: <input type="number" step="0.01" name="act_threshold" value="{{ defaults.act_threshold }}" title="ACT halt threshold (default {{ defaults.act_threshold }})"><br>
-            c_floor: <input type="number" step="0.01" name="c_floor" value="{{ c_floor }}" title="Complexity floor"><br>
-            s_floor: <input type="number" step="0.01" name="s_floor" value="{{ s_floor }}" title="Symbiosis floor"><br>
-            <button type="submit">Init</button>
-        </form>
-    </section>
-    <section>
-        <h2>Train Step</h2>
-        <form id="trainForm">
-            Bits (e.g. 0 1 0 1): <input type="text" name="bits" value="0 1 0 1"><br>
-            Upload file: <input type="file" id="train_file"><br>
-            <button type="submit">Train</button>
-        </form>
-        <label>Load sample dataset:
-            <select id="datasetSelect">
-                <option value="">--Select--</option>
-                <option value="wikitext2_train">Wikitext-2 (train)</option>
-                <option value="wikitext2_validation">Wikitext-2 (validation)</option>
-            </select>
-        </label>
-        <p id="trainOut"></p>
-    </section>
-    <section>
-        <h2>Scale Up</h2>
-        Width Mult: <input type="number" step="0.1" id="width_mult" value="1.0"><br>
-        <button id="scaleBtn">Scale Model</button>
-    </section>
-    <section>
-        <h2>Collapse Submodel</h2>
-        <form id="collapseForm">
-            Cluster Bits (JSON array of arrays):<br>
-            <textarea name="clusters" rows="3" cols="40">[[0,1,0,1],[1,1,0,0]]</textarea><br>
-            Target Params (JSON):<br>
-            <textarea name="params" rows="3" cols="40">{"d_model":32,"nhead":4,"num_layers":1,"dim_feedforward":64,"max_seq_len":16}</textarea><br>
-            Width Scale: <input type="number" step="0.1" id="width_scale" value="1.0"><br>
-            <button type="submit">Collapse</button>
-        </form>
-    </section>
-    <section>
-        <h2>Inference</h2>
-        <form id="inferForm">
-            Bits: <input type="text" name="bits" value="0 1 0 1"><br>
-            Upload file: <input type="file" id="infer_file"><br>
-            <button type="submit">Infer</button>
-        </form>
-        <pre id="inferOut"></pre>
-    </section>
-    <section>
-        <h2>Long Inference</h2>
-        <form id="inferLongForm">
-            Bits: <input type="text" name="bits" value="0 1 0 1"><br>
-            ctx_bits: <input type="number" name="ctx_bits" value="4096"><br>
-            overlap: <input type="number" name="overlap" value="256"><br>
-            <button type="submit">Infer Long</button>
-        </form>
-        <pre id="inferLongOut"></pre>
-    </section>
-    <section>
-        <h2>Text Inference</h2>
-        <form id="textInferForm">
-            Text: <input type="text" name="text" value="hello"><br>
-            <button type="submit">Infer Text</button>
-        </form>
-        <pre id="textInferOut"></pre>
-    </section>
-    <section>
-        <h2>&lambda; Weights</h2>
-        <form id="lambdaForm">
-            &lambda;<sub>K</sub>: <input type="range" min="0" max="2" step="0.1" id="lambda_K" oninput="lambda_K_val.innerText=value"><span id="lambda_K_val"></span><br>
-            &lambda;<sub>C</sub>: <input type="range" min="0" max="2" step="0.1" id="lambda_C" oninput="lambda_C_val.innerText=value"><span id="lambda_C_val"></span><br>
-            &lambda;<sub>S</sub>: <input type="range" min="0" max="2" step="0.1" id="lambda_S" oninput="lambda_S_val.innerText=value"><span id="lambda_S_val"></span><br>
-            <button type="submit">Update</button>
-        </form>
-    </section>
-    <section>
-        <h2>Diffusion LM</h2>
-        <label><input type="checkbox" id="diffusion_box"> Enable Diffusion Mode</label>
-    </section>
-    <section>
-        <h2>GPU Acceleration</h2>
-        <label><input type="checkbox" id="gpu_box"> Enable FSDP &amp; CUDA</label>
-    </section>
-    <section>
-        <h2>Enable Compression</h2>
-        <label><input type="checkbox" id="compression_box"> Compress I/O</label>
-        <p>Ratio: <span id="comp_ratio">1.0</span></p>
-    </section>
-    <section>
-        <h2>Quantization Aware Training</h2>
-        <label><input type="checkbox" id="qat_box"> Enable 4-bit QAT</label>
-    </section>
-    <section>
-        <h2>Model Status</h2>
-        <pre id="statusOut"></pre>
-    </section>
-    <section>
-        <h2>Telemetry</h2>
-        <canvas id="metricChart" width="600" height="300"></canvas>
-    </section>
-    <section>
-        <h2>Hugging Face Checkpoints</h2>
-        Repo ID: <input type="text" id="hf_repo"><br>
-        Token: <input type="password" id="hf_token" placeholder="optional"><br>
-        <button id="uploadBtn">Upload weights</button>
-        <button id="downloadBtn">Download weights</button>
-        <p id="hfStatus"></p>
-    </section>
-
-<script>
-async function postJSON(url, data){
-    const resp = await fetch(url, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify(data)});
-    return resp.json();
-}
-
-async function pollJob(id){
-    while(true){
-        const job = await fetch(`/job/${id}`).then(r=>r.json());
-        if(job.status === 'completed') return job.result;
-        if(job.status === 'error') throw job.error || 'Job failed';
-        await new Promise(r=>setTimeout(r, 1000));
-    }
-}
-
-function loadInitParams(){
-    const saved = JSON.parse(localStorage.getItem('init_params')||'{}');
-    const form = document.getElementById('initForm');
-    for(const [k,v] of Object.entries(saved)){
-        const el = form.elements[k];
-        if(!el) continue;
-        if(el.type === 'checkbox') el.checked = v; else el.value = v;
-    }
-}
-loadInitParams();
-
-function byteArrayToBits(arr){
-    const bits=[];
-    for(const b of arr){
-        for(let i=7;i>=0;i--) bits.push((b>>i)&1);
-    }
-    return bits;
-}
-
-let trainFileBits=null, inferFileBits=null, datasetBits=null;
-
-async function fileToBits(file){
-    if(file.type.startsWith('text')){
-        const text = await file.text();
-        const res = await postJSON('/text_to_bits', {text});
-        return res.bits;
-    }
-    const buf = await file.arrayBuffer();
-    return byteArrayToBits(new Uint8Array(buf));
-}
-
-let metricChart;
-async function initChart(){
-    const data = await fetch('/metrics').then(r=>r.json());
-    const labels = data.negentropy.map((_,i)=>i);
-    const ctx = document.getElementById('metricChart').getContext('2d');
-    metricChart = new Chart(ctx, {
-        type:'line',
-        data:{
-            labels:labels,
-            datasets:[
-                {label:'Negentropy', data:data.negentropy, borderColor:'blue', fill:false},
-                {label:'LZ Complexity', data:data.lz_complexity, borderColor:'orange', fill:false},
-                {label:'Symbiosis', data:data.symbiosis, borderColor:'green', fill:false}
-            ]
-        },
-        options:{responsive:false, interaction:{mode:'index', intersect:false}}
-    });
-}
-
-async function updateChart(){
-    const data = await fetch('/metrics').then(r=>r.json());
-    const labels = data.negentropy.map((_,i)=>i);
-    metricChart.data.labels = labels;
-    metricChart.data.datasets[0].data = data.negentropy;
-    metricChart.data.datasets[1].data = data.lz_complexity;
-    metricChart.data.datasets[2].data = data.symbiosis;
-    metricChart.update();
-}
-
-initChart();
-setInterval(updateChart, 2000);
-
-async function refreshStatus(){
-    const [s, c] = await Promise.all([fetch('/status'), fetch('/model_config')]);
-    const status = await s.json();
-    const config = await c.json();
-    document.getElementById('statusOut').innerText = JSON.stringify({...status, ...config}, null, 2);
-}
-
-document.getElementById('initForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const fd = new FormData(e.target);
-    const obj = Object.fromEntries(fd.entries());
-    const ints = ['d_model','nhead','num_layers','dim_feedforward','max_seq_len','chunk_size','overlap'];
-    ints.forEach(k=>{ if(obj[k]===''){ delete obj[k]; } else obj[k]=parseInt(obj[k]); });
-    obj.reversible = document.getElementById('reversible_box').checked;
-    obj.use_checkpoint = document.getElementById('checkpoint_box').checked;
-    obj.act_threshold = parseFloat(obj.act_threshold);
-    const floors = {c_floor: parseFloat(obj.c_floor), s_floor: parseFloat(obj.s_floor)};
-    delete obj.c_floor; delete obj.s_floor;
-    await postJSON('/init', obj);
-    await postJSON('/config/telemetry', floors);
-    localStorage.setItem('init_params', JSON.stringify({...obj, ...floors}));
-    refreshStatus();
-    updateChart();
-});
-
-document.getElementById('trainForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const form = e.target;
-    let payload;
-    if(trainFileBits){
-        payload = trainFileBits;
-    } else if(datasetBits){
-        payload = datasetBits;
-    } else {
-        payload = [form.bits.value.trim().split(/\s+/).map(Number)];
-    }
-    for(const el of form.elements) el.disabled = true;
-    const out = document.getElementById('trainOut');
-    out.innerText = '⏳';
-    try{
-        const job = await postJSON('/train', {bits: payload});
-        const res = await pollJob(job.job_id);
-        out.innerText = 'Loss: '+res.loss.toFixed(4);
-        if(res.ratio !== undefined){
-            document.getElementById('comp_ratio').innerText = res.ratio.toFixed(2);
-        }
-    } catch(err){
-        out.innerText = 'Error';
-        alert(err);
-    } finally {
-        for(const el of form.elements) el.disabled = false;
-        refreshStatus();
-        updateChart();
-    }
-});
-
-document.getElementById('train_file').addEventListener('change', async (e)=>{
-    const f = e.target.files[0];
-    if(!f) return;
-    const bits = await fileToBits(f);
-    trainFileBits = [bits];
-    datasetBits = null;
-    document.querySelector('#trainForm input[name="bits"]').value = bits.slice(0,64).join(' ');
-});
-
-document.querySelector('#trainForm input[name="bits"]').addEventListener('input', ()=>{
-    trainFileBits = null;
-    datasetBits = null;
-});
-
-document.getElementById('scaleBtn').addEventListener('click', async ()=>{
-    const btn = document.getElementById('scaleBtn');
-    const input = document.getElementById('width_mult');
-    const mult = parseFloat(input.value);
-    btn.disabled = true; input.disabled = true;
-    const original = btn.innerText; btn.innerText = '⏳';
-    try{
-        const job = await postJSON('/scale_up', {width_mult: mult});
-        await pollJob(job.job_id);
-    } catch(err){
-        alert(err);
-    } finally {
-        btn.innerText = original;
-        btn.disabled = false; input.disabled = false;
-        refreshStatus();
-        updateChart();
-    }
-});
-
-document.getElementById('collapseForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const form = e.target;
-    const btn = form.querySelector('button');
-    for(const el of form.elements) el.disabled = true;
-    const clusters = JSON.parse(form.clusters.value);
-    const params = JSON.parse(form.params.value);
-    const w = parseFloat(document.getElementById('width_scale').value);
-    const original = btn.innerText; btn.innerText = '⏳';
-    try{
-        const job = await postJSON('/collapse', {clusters: clusters, params: params, width_scale: w});
-        await pollJob(job.job_id);
-    } catch(err){
-        alert(err);
-    } finally {
-        btn.innerText = original;
-        for(const el of form.elements) el.disabled = false;
-        refreshStatus();
-        updateChart();
-    }
-});
-
-document.getElementById('inferForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    let bits;
-    if(inferFileBits){
-        bits = inferFileBits;
-    } else if(datasetBits){
-        bits = [datasetBits[0]];
-    } else {
-        bits = [e.target.bits.value.trim().split(/\s+/).map(Number)];
-    }
-    const res = await postJSON('/infer', {bits});
-    if(res.error){
-        alert(res.error + '\n' + (res.suggestion||''));
-    } else {
-        document.getElementById('inferOut').innerText = JSON.stringify(res, null, 2);
-        if(res.ratio !== undefined){
-            document.getElementById('comp_ratio').innerText = res.ratio.toFixed(2);
-        }
-    }
-    refreshStatus();
-    updateChart();
-});
-
-document.getElementById('infer_file').addEventListener('change', async (e)=>{
-    const f = e.target.files[0];
-    if(!f) return;
-    const bits = await fileToBits(f);
-    inferFileBits = [bits];
-    datasetBits = null;
-    document.querySelector('#inferForm input[name="bits"]').value = bits.slice(0,64).join(' ');
-});
-
-document.querySelector('#inferForm input[name="bits"]').addEventListener('input', ()=>{
-    inferFileBits = null;
-    datasetBits = null;
-});
-
-document.getElementById('datasetSelect').addEventListener('change', async (e)=>{
-    const val = e.target.value;
-    trainFileBits = null;
-    inferFileBits = null;
-    if(!val){ datasetBits = null; return; }
-    const [name, split] = val.split('_');
-    const resp = await fetch(`/dataset?name=${name}&split=${split}&size=4&seq_len=64`);
-    const data = await resp.json();
-    datasetBits = data.bits;
-    const preview = data.bits[0].slice(0,64).join(' ');
-    document.querySelector('#trainForm input[name="bits"]').value = preview;
-    document.querySelector('#inferForm input[name="bits"]').value = preview;
-});
-
-document.getElementById('inferLongForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const bits = e.target.bits.value.trim().split(/\s+/).map(Number);
-    const ctx = parseInt(e.target.ctx_bits.value);
-    const ov = parseInt(e.target.overlap.value);
-    const res = await postJSON('/infer_long', {bits: bits, ctx_bits: ctx, overlap: ov});
-    document.getElementById('inferLongOut').innerText = JSON.stringify(res, null, 2);
-    refreshStatus();
-    updateChart();
-});
-
-document.getElementById('textInferForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const text = e.target.text.value;
-    const res = await postJSON('/infer_text', {text:text});
-    document.getElementById('textInferOut').innerText = JSON.stringify(res, null, 2);
-    refreshStatus();
-    updateChart();
-});
-
-async function loadLambdas(){
-    const resp = await fetch('/lambdas');
-    const vals = await resp.json();
-    for(const k of ['lambda_K','lambda_C','lambda_S']){
-        document.getElementById(k).value = vals[k];
-        document.getElementById(k+"_val").innerText = vals[k];
-    }
-}
-
-document.getElementById('lambdaForm').addEventListener('submit', async (e)=>{
-    e.preventDefault();
-    const data = {
-        lambda_K: parseFloat(document.getElementById('lambda_K').value),
-        lambda_C: parseFloat(document.getElementById('lambda_C').value),
-        lambda_S: parseFloat(document.getElementById('lambda_S').value),
-    };
-    await postJSON('/lambdas', data);
-    for(const k in data){
-        document.getElementById(k+"_val").innerText = data[k];
-    }
-    refreshStatus();
-});
-
-loadLambdas();
-
-function restoreToggle(id,key,endpoint,field){
-    const box = document.getElementById(id);
-    const saved = localStorage.getItem(key);
-    if(saved !== null){ box.checked = saved === 'true'; postJSON(endpoint,{[field]: box.checked}); }
-    box.addEventListener('change', async (e)=>{
-        await postJSON(endpoint, {[field]: e.target.checked});
-        localStorage.setItem(key, e.target.checked);
-        refreshStatus();
-    });
-}
-
-restoreToggle('diffusion_box','diffusion','/diffusion','diffusion');
-restoreToggle('gpu_box','use_gpu','/gpu','use_gpu');
-restoreToggle('compression_box','compression','/compression','compression');
-restoreToggle('qat_box','qat','/qat','qat');
-
-document.getElementById('uploadBtn').addEventListener('click', async ()=>{
-    const repo = document.getElementById('hf_repo').value;
-    const token = document.getElementById('hf_token').value;
-    const res = await postJSON('/save_checkpoint', {repo_id: repo, token: token||undefined});
-    document.getElementById('hfStatus').innerText = res.status || res.error;
-});
-
-document.getElementById('downloadBtn').addEventListener('click', async ()=>{
-    const repo = document.getElementById('hf_repo').value;
-    const token = document.getElementById('hf_token').value;
-    const res = await postJSON('/download_checkpoint', {repo_id: repo, token: token||undefined});
-    document.getElementById('hfStatus').innerText = res.status || res.error;
-    refreshStatus();
-    updateChart();
-});
-
-refreshStatus();
-</script>
-    </div>
-</body>
-</html>
-

build_full_bits.py

@@ -1,23 +0,0 @@
-import pathlib
-import torch
-from datasets import load_dataset
-
-TXT_MB = 100
-OUT = pathlib.Path('full_bits.pt')
-
-
-def build_bits(out: pathlib.Path = OUT, txt_mb: int = TXT_MB) -> None:
-    ds = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')
-    buf = bytearray()
-    for line in ds['text']:
-        buf.extend(line.encode() + b"\n")
-        if len(buf) >= txt_mb * 2 ** 20:
-            break
-    bits = []
-    for byte in buf:
-        bits.extend(int(b) for b in f'{byte:08b}')
-    tensor = torch.tensor(bits, dtype=torch.uint8)
-    torch.save(tensor, out)
-
-if __name__ == '__main__':
-    build_bits()

cpu_edge_training.py

@@ -1,468 +0,0 @@
-#!/usr/bin/env python3
-"""
-CPU-Optimized Edge Deployment BitTransformerLM Training
-Optimized for consumer devices and edge applications.
-"""
-
-import os
-import time
-import torch
-import torch.nn.functional as F
-from datasets import load_dataset
-
-from bit_transformer import (
-    BitTransformerLM,
-    text_to_bits,
-    bits_to_text,
-    train_loop,
-    configure_optimizer,
-    save_model,
-    load_model,
-    set_dropout,
-    hil_safe_inference,
-    quantize_dynamic,
-)
-from bit_transformer.torch_utils import cpu_autocast
-from bit_transformer.training import train_loop
-
-
-def create_optimal_cpu_model():
-    """Create BitTransformerLM optimized for CPU edge deployment."""
-    print("🧠 Creating CPU-optimized BitTransformerLM...")
-    
-    # Optimal configuration for edge devices:
-    # - Small model size for low memory footprint
-    # - CPU autocast for faster FP16 inference
-    # - No reversible layers (simpler for CPU)
-    # - Gradient checkpointing disabled for speed
-    # - Small context length for efficiency
-    
-    model = BitTransformerLM(
-        d_model=64,           # Small embedding dimension (vs 128 default)
-        nhead=4,              # Fewer attention heads (vs 8 default)
-        num_layers=3,         # Shallow model (vs 4 default)  
-        dim_feedforward=128,  # Smaller FFN (vs 512 default)
-        max_seq_len=256,      # Shorter context (vs 1024 default)
-        reversible=False,     # Disable reversible layers (CPU doesn't benefit much)
-        use_checkpoint=False, # Disable gradient checkpointing (prioritize speed)
-        use_autocast=True,    # Enable CPU autocast for BF16 mixed precision
-        use_act=False,        # Disable ACT for simplicity
-        chunk_size=32,        # Small chunks for memory efficiency
-        full_attn_logging=False,  # Disable attention logging to save memory
-        lambda_K=1.0,         # Standard telemetry weights
-        lambda_C=1.0,
-        lambda_S=1.0,
-    )
-    
-    # Calculate model parameters
-    total_params = sum(p.numel() for p in model.parameters())
-    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
-    
-    print(f"   📊 Model Configuration:")
-    print(f"     d_model: {64}")
-    print(f"     num_layers: {3}")
-    print(f"     nhead: {4}")
-    print(f"     dim_feedforward: {128}")
-    print(f"     max_seq_len: {256}")
-    print(f"     Total parameters: {total_params:,}")
-    print(f"     Trainable parameters: {trainable_params:,}")
-    print(f"     Estimated size: {total_params * 4 / 1024 / 1024:.1f}MB (FP32)")
-    print(f"     With autocast: ~{total_params * 2 / 1024 / 1024:.1f}MB (BF16)")
-    
-    return model
-
-
-def load_training_dataset(dataset_size=512, max_len=128):
-    """Load and prepare training dataset optimized for edge training."""
-    print("📚 Loading training dataset...")
-    
-    try:
-        # Try to load BitTransformerLM dataset from HuggingFace
-        print("   Attempting to load BitTransformerLM dataset...")
-        dataset = load_dataset("WCNegentropy/BitTransformerLM", split="train[:{}]".format(dataset_size))
-        if dataset and len(dataset) > 0:
-            train_texts = [item['text'] for item in dataset if item.get('text')]
-            if len(train_texts) > 0:
-                print(f"   ✅ Loaded {len(train_texts)} samples from BitTransformerLM dataset")
-            else:
-                raise Exception("No text samples found in dataset")
-        else:
-            raise Exception("Dataset empty or not accessible")
-            
-    except Exception as e:
-        print(f"   ⚠️  BitTransformerLM dataset not available: {e}")
-        print("   📖 Falling back to WikiText-2...")
-        try:
-            # Fallback to WikiText-2 for training
-            ds = load_dataset("wikitext", "wikitext-2-raw-v1")
-            train_texts = [text for text in ds["train"]["text"] if text.strip()][:dataset_size]
-            print(f"   ✅ Loaded {len(train_texts)} samples from WikiText-2")
-        except Exception as e2:
-            print(f"   ❌ Failed to load WikiText-2: {e2}")
-            print("   🎲 Using synthetic text data...")
-            # Generate simple synthetic text for demonstration
-            synthetic_texts = [
-                "The quick brown fox jumps over the lazy dog.",
-                "Machine learning is transforming technology.",
-                "Edge computing enables local AI processing.",
-                "BitTransformerLM uses bit-native processing.",
-                "CPU optimization improves inference speed.",
-                "Neural networks learn from training data.",
-                "Transformers use attention mechanisms.",
-                "Language models understand text patterns.",
-            ]
-            train_texts = (synthetic_texts * (dataset_size // len(synthetic_texts) + 1))[:dataset_size]
-            print(f"   ✅ Generated {len(train_texts)} synthetic samples")
-    
-    # Convert text to bits
-    print("   🔄 Converting text to bits...")
-    train_sequences = []
-    valid_sequences = []
-    
-    for i, text in enumerate(train_texts):
-        try:
-            bits = text_to_bits(text)[:max_len]
-            if len(bits) < max_len:
-                bits.extend([0] * (max_len - len(bits)))  # Pad to max_len
-            
-            # Use 80/20 split for train/validation
-            if i < len(train_texts) * 0.8:
-                train_sequences.append(bits)
-            else:
-                valid_sequences.append(bits)
-                
-        except Exception as e:
-            print(f"   ⚠️  Failed to convert text to bits: {e}")
-            continue
-    
-    train_tensor = torch.tensor(train_sequences, dtype=torch.long)
-    valid_tensor = torch.tensor(valid_sequences, dtype=torch.long) if valid_sequences else train_tensor[:16]
-    
-    print(f"   📊 Dataset Statistics:")
-    print(f"     Training sequences: {len(train_sequences)}")
-    print(f"     Validation sequences: {len(valid_sequences)}")
-    print(f"     Sequence length: {max_len}")
-    print(f"     Training tensor shape: {train_tensor.shape}")
-    
-    return train_tensor, valid_tensor, train_texts[:len(train_sequences)]
-
-
-def train_cpu_optimized_model(model, train_data, valid_data, epochs=5):
-    """Train the model with CPU-optimized settings."""
-    print(f"🚀 Training CPU-optimized BitTransformerLM for {epochs} epochs...")
-    
-    # Set model to training mode
-    model.train()
-    set_dropout(model, 0.1)
-    
-    # Configure optimizer for edge deployment
-    # Lower learning rate and smaller batch size for stable CPU training
-    batch_size = 4  # Small batch size for memory efficiency
-    learning_rate = 5e-4  # Conservative learning rate
-    total_steps = max(1, epochs * (len(train_data) // batch_size))  # Ensure at least 1 step
-    
-    if len(train_data) == 0:
-        raise ValueError("No training data available - check dataset loading")
-    
-    optimizer, scheduler = configure_optimizer(
-        model,
-        lr=learning_rate,
-        total_steps=total_steps,
-        weight_decay=0.01
-    )
-    
-    print(f"   📋 Training Configuration:")
-    print(f"     Batch size: {batch_size}")
-    print(f"     Learning rate: {learning_rate}")
-    print(f"     Total steps: {total_steps}")
-    print(f"     CPU autocast: Enabled")
-    
-    # Training loop with CPU optimizations
-    train_losses = []
-    
-    for epoch in range(epochs):
-        print(f"\n   📖 Epoch {epoch + 1}/{epochs}")
-        epoch_losses = []
-        epoch_start_time = time.time()
-        
-        # Shuffle training data
-        perm = torch.randperm(len(train_data))
-        train_data_shuffled = train_data[perm]
-        
-        # Process in small batches
-        for batch_idx in range(0, len(train_data_shuffled), batch_size):
-            batch_end = min(batch_idx + batch_size, len(train_data_shuffled))
-            batch = train_data_shuffled[batch_idx:batch_end]
-            
-            if len(batch) == 0:
-                continue
-            
-            optimizer.zero_grad()
-            
-            # Use CPU autocast for mixed precision
-            with cpu_autocast():
-                logits, telemetry = model(batch)
-                
-                # Standard autoregressive loss
-                pred = logits[:, :-1, :].reshape(-1, 2)
-                target = batch[:, 1:].reshape(-1)
-                loss = F.cross_entropy(pred, target)
-            
-            # Backward pass
-            loss.backward()
-            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
-            optimizer.step()
-            
-            # Only step scheduler if we haven't exceeded total steps
-            if scheduler.last_epoch < scheduler.total_steps - 1:
-                scheduler.step()
-            
-            batch_loss = loss.item()
-            epoch_losses.append(batch_loss)
-            
-            # Log progress every 50 steps
-            if (batch_idx // batch_size) % 50 == 0:
-                avg_loss = sum(epoch_losses[-10:]) / min(10, len(epoch_losses))
-                telemetry_str = f"K={telemetry.get('K', 0):.3f}, C={telemetry.get('C', 0):.3f}, S={telemetry.get('S', 0):.3f}"
-                print(f"     Step {batch_idx // batch_size}: Loss={avg_loss:.4f}, {telemetry_str}")
-        
-        epoch_time = time.time() - epoch_start_time
-        avg_epoch_loss = sum(epoch_losses) / len(epoch_losses)
-        train_losses.append(avg_epoch_loss)
-        
-        print(f"   ⏱️  Epoch {epoch + 1} completed in {epoch_time:.1f}s, Avg Loss: {avg_epoch_loss:.4f}")
-        
-        # Validation every epoch
-        if len(valid_data) > 0:
-            model.eval()
-            set_dropout(model, 0.0)
-            
-            with torch.no_grad():
-                with cpu_autocast():
-                    val_batch = valid_data[:min(8, len(valid_data))]  # Small validation batch
-                    val_logits, val_telemetry = model(val_batch)
-                    val_pred = val_logits[:, :-1, :].reshape(-1, 2)
-                    val_target = val_batch[:, 1:].reshape(-1)
-                    val_loss = F.cross_entropy(val_pred, val_target).item()
-            
-            print(f"   📊 Validation Loss: {val_loss:.4f}")
-            print(f"   📈 Telemetry - K: {val_telemetry.get('K', 0):.3f}, C: {val_telemetry.get('C', 0):.3f}, S: {val_telemetry.get('S', 0):.3f}")
-            
-            model.train()
-            set_dropout(model, 0.1)
-    
-    print(f"\n✅ Training completed!")
-    print(f"   Final training loss: {train_losses[-1]:.4f}")
-    
-    return model, train_losses
-
-
-def test_model_inference(model, test_texts):
-    """Test the trained model with inference and safety checks."""
-    print("\n🧪 Testing Model Inference...")
-    
-    model.eval()
-    set_dropout(model, 0.0)
-    
-    # Test basic inference
-    test_samples = test_texts[:3]  # Test with first 3 samples
-    
-    for i, text in enumerate(test_samples):
-        print(f"\n   Test {i + 1}: {text[:50]}...")
-        
-        try:
-            # Convert to bits
-            input_bits = text_to_bits(text)[:64]  # Shorter for demo
-            if len(input_bits) < 64:
-                input_bits.extend([0] * (64 - len(input_bits)))
-            
-            input_tensor = torch.tensor([input_bits], dtype=torch.long)
-            
-            # Run inference with CPU autocast
-            with torch.no_grad():
-                with cpu_autocast():
-                    logits, telemetry = model(input_tensor)
-                    
-                    # Generate next tokens
-                    next_token_logits = logits[0, -1, :]
-                    next_token_probs = F.softmax(next_token_logits, dim=-1)
-                    next_token = torch.multinomial(next_token_probs, 1).item()
-                    
-            print(f"     Input bits: {input_bits[:16]}... (showing first 16)")
-            print(f"     Next token prediction: {next_token}")
-            print(f"     Next token confidence: {next_token_probs[next_token]:.3f}")
-            print(f"     Telemetry - K: {telemetry.get('K', 0):.3f}, C: {telemetry.get('C', 0):.3f}, S: {telemetry.get('S', 0):.3f}")
-            
-        except Exception as e:
-            print(f"     ❌ Inference failed: {e}")
-    
-    # Test safe inference
-    print(f"\n🛡️ Testing Safe Inference...")
-    try:
-        # Create a simple prompt
-        test_prompt = "The future of AI is"
-        prompt_bits = text_to_bits(test_prompt)
-        prompt_tensor = torch.tensor([prompt_bits], dtype=torch.long)
-        
-        with cpu_autocast():
-            safe_result = hil_safe_inference(model, prompt_tensor, max_new_tokens=16)
-            
-        if safe_result is not None:
-            print(f"   ✅ Safe inference successful")
-            print(f"   Generated {len(safe_result[0]) - len(prompt_bits)} new tokens")
-        else:
-            print(f"   ⚠️  Safe inference blocked by safety gates")
-            
-    except Exception as e:
-        print(f"   ❌ Safe inference test failed: {e}")
-
-
-def benchmark_cpu_performance(model):
-    """Benchmark the model's CPU performance."""
-    print("\n⚡ CPU Performance Benchmark...")
-    
-    model.eval()
-    set_dropout(model, 0.0)
-    
-    # Prepare test data
-    batch_sizes = [1, 2, 4]
-    sequence_lengths = [32, 64, 128]
-    
-    results = []
-    
-    for batch_size in batch_sizes:
-        for seq_len in sequence_lengths:
-            print(f"\n   Testing batch_size={batch_size}, seq_len={seq_len}")
-            
-            # Create random test data
-            test_data = torch.randint(0, 2, (batch_size, seq_len), dtype=torch.long)
-            
-            # Warmup
-            with torch.no_grad():
-                with cpu_autocast():
-                    for _ in range(3):
-                        _, _ = model(test_data)
-            
-            # Benchmark
-            times = []
-            for _ in range(10):
-                start_time = time.time()
-                with torch.no_grad():
-                    with cpu_autocast():
-                        logits, telemetry = model(test_data)
-                end_time = time.time()
-                times.append(end_time - start_time)
-            
-            avg_time = sum(times) / len(times)
-            throughput = (batch_size * seq_len) / avg_time
-            
-            result = {
-                'batch_size': batch_size,
-                'seq_len': seq_len,
-                'avg_time_ms': avg_time * 1000,
-                'throughput_tokens_per_sec': throughput
-            }
-            results.append(result)
-            
-            print(f"     Average time: {avg_time * 1000:.2f}ms")
-            print(f"     Throughput: {throughput:.0f} tokens/sec")
-    
-    # Summary
-    print(f"\n📊 Performance Summary:")
-    best_throughput = max(results, key=lambda x: x['throughput_tokens_per_sec'])
-    print(f"   Best throughput: {best_throughput['throughput_tokens_per_sec']:.0f} tokens/sec")
-    print(f"   At batch_size={best_throughput['batch_size']}, seq_len={best_throughput['seq_len']}")
-    
-    return results
-
-
-def quantize_for_deployment(model):
-    """Apply dynamic quantization for deployment."""
-    print("\n🗜️ Applying Dynamic Quantization for Deployment...")
-    
-    try:
-        quantized_model = quantize_dynamic(model)
-        
-        # Compare model sizes
-        original_params = sum(p.numel() for p in model.parameters())
-        quantized_params = sum(p.numel() for p in quantized_model.parameters())
-        
-        print(f"   Original parameters: {original_params:,}")
-        print(f"   Quantized parameters: {quantized_params:,}")
-        print(f"   Model size reduction: ~50% (FP32 -> INT8)")
-        
-        # Quick inference test
-        test_input = torch.randint(0, 2, (1, 32), dtype=torch.long)
-        
-        with torch.no_grad():
-            original_output = model(test_input)
-            quantized_output = quantized_model(test_input)
-        
-        print(f"   ✅ Quantization successful - model still functional")
-        
-        return quantized_model
-        
-    except Exception as e:
-        print(f"   ❌ Quantization failed: {e}")
-        return model
-
-
-def main():
-    """Main training and testing pipeline."""
-    print("🚀 CPU-Optimized BitTransformerLM Training Pipeline")
-    print("="*60)
-    
-    # Step 1: Create optimal CPU model
-    model = create_optimal_cpu_model()
-    
-    # Step 2: Load training dataset  
-    train_data, valid_data, train_texts = load_training_dataset(dataset_size=256, max_len=128)
-    
-    # Step 3: Train the model
-    trained_model, train_losses = train_cpu_optimized_model(model, train_data, valid_data, epochs=3)
-    
-    # Step 4: Test inference
-    test_model_inference(trained_model, train_texts)
-    
-    # Step 5: Benchmark performance
-    benchmark_results = benchmark_cpu_performance(trained_model)
-    
-    # Step 6: Apply quantization
-    quantized_model = quantize_for_deployment(trained_model)
-    
-    # Step 7: Save models
-    print("\n💾 Saving Models...")
-    
-    # Create weights directory if it doesn't exist
-    os.makedirs("weights", exist_ok=True)
-    
-    try:
-        save_model(trained_model, "weights/cpu_edge_model.pt.gz")
-        print("   ✅ Saved trained model: weights/cpu_edge_model.pt.gz")
-        
-        save_model(quantized_model, "weights/cpu_edge_model_quantized.pt.gz")
-        print("   ✅ Saved quantized model: weights/cpu_edge_model_quantized.pt.gz")
-        
-    except Exception as e:
-        print(f"   ⚠️  Model saving failed: {e}")
-    
-    # Final summary
-    print("\n" + "="*60)
-    print("🎉 CPU-Optimized BitTransformerLM Training Complete!")
-    print("="*60)
-    
-    total_params = sum(p.numel() for p in trained_model.parameters())
-    final_loss = train_losses[-1] if train_losses else "N/A"
-    best_throughput = max(benchmark_results, key=lambda x: x['throughput_tokens_per_sec'])
-    
-    print(f"📊 Final Results:")
-    print(f"   Model Parameters: {total_params:,}")
-    print(f"   Final Training Loss: {final_loss}")
-    print(f"   Peak Throughput: {best_throughput['throughput_tokens_per_sec']:.0f} tokens/sec")
-    print(f"   Model Size (quantized): ~{total_params * 1 / 1024 / 1024:.1f}MB")
-    print(f"   CPU Optimizations: BF16 autocast, no gradient checkpointing, small chunks")
-    print(f"   Edge Ready: ✅ Optimized for consumer CPUs")
-
-
-if __name__ == "__main__":
-    main()

create_dataset.py

@@ -1,61 +0,0 @@
-#!/usr/bin/env python3
-"""
-BitTransformerLM Dataset Creation Script
-
-Usage:
-    python create_dataset.py --token YOUR_HF_TOKEN --repo-id YOUR_REPO_NAME
-
-This script creates a comprehensive dataset for BitTransformerLM training
-and uploads it to HuggingFace Hub with proper metadata and organization.
-"""
-
-import argparse
-import sys
-from pathlib import Path
-
-# Add the bit_transformer module to path
-sys.path.insert(0, str(Path(__file__).parent))
-
-from bit_transformer.dataset_builder import create_bittransformerlm_dataset
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Create BitTransformerLM Dataset")
-    parser.add_argument("--token", required=True, help="HuggingFace access token")
-    parser.add_argument("--repo-id", default="BitTransformerLM", help="Dataset repository ID")
-    parser.add_argument("--private", action="store_true", default=True, help="Make dataset private")
-    parser.add_argument("--samples", type=int, default=25000, help="Total number of samples")
-    
-    args = parser.parse_args()
-    
-    print("🚀 Starting BitTransformerLM Dataset Creation")
-    print(f"Repository: {args.repo_id}")
-    print(f"Private: {args.private}")
-    print(f"Target samples: {args.samples}")
-    print("-" * 50)
-    
-    try:
-        dataset_url = create_bittransformerlm_dataset(
-            hf_token=args.token,
-            repo_id=args.repo_id
-        )
-        
-        print("\n" + "=" * 50)
-        print("🎉 SUCCESS! Dataset created and uploaded")
-        print(f"📍 URL: {dataset_url}")
-        print("=" * 50)
-        
-        print("\n📋 Next Steps:")
-        print("1. View your dataset on HuggingFace Hub")
-        print("2. Test loading with: `from datasets import load_dataset`")
-        print("3. Integrate with BitTransformerLM training pipeline")
-        print("4. Monitor dataset usage and performance metrics")
-        
-    except Exception as e:
-        print(f"\n❌ ERROR: {e}")
-        print("Please check your token and repository permissions.")
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()

enhanced_checkpoint_system.py

@@ -1,374 +0,0 @@
-#!/usr/bin/env python3
-"""
-Enhanced checkpointing system for BitTransformerLM with multiple training runs support.
-Optimized for Claude Code environment with HF Pro + 20GB persistent storage.
-"""
-
-import os
-import json
-import shutil
-import logging
-from pathlib import Path
-from typing import Dict, Any, Optional, List, Union
-from datetime import datetime
-import torch
-from huggingface_hub import HfApi, hf_hub_download
-
-from bit_transformer.error_handling import with_error_recovery, safe_operation
-from bit_transformer.types import PathLike, ModelConfig, TrainingConfig
-
-logger = logging.getLogger(__name__)
-
-
-class EnhancedCheckpointManager:
-    """Advanced checkpoint management for multiple training runs with HF integration."""
-    
-    def __init__(self, 
-                 base_dir: PathLike = "/data/checkpoints",
-                 hf_repo_id: str = "WCNegentropy/BitTransformerLM",
-                 hf_token: Optional[str] = None,
-                 max_local_checkpoints: int = 5):
-        
-        self.base_dir = Path(base_dir)
-        self.base_dir.mkdir(parents=True, exist_ok=True)
-        
-        self.hf_repo_id = hf_repo_id
-        self.hf_token = hf_token or os.getenv("HF_TOKEN")
-        self.api = HfApi(token=self.hf_token) if self.hf_token else None
-        
-        self.max_local_checkpoints = max_local_checkpoints
-        
-        # Training session tracking
-        self.sessions_dir = self.base_dir / "training_sessions"
-        self.sessions_dir.mkdir(exist_ok=True)
-        
-        # Best models storage
-        self.best_models_dir = self.base_dir / "best_models"
-        self.best_models_dir.mkdir(exist_ok=True)
-    
-    def create_training_session(self, 
-                              session_name: str,
-                              model_config: ModelConfig,
-                              training_config: TrainingConfig) -> str:
-        """Create a new training session with metadata."""
-        
-        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-        session_id = f"{session_name}_{timestamp}"
-        session_dir = self.sessions_dir / session_id
-        session_dir.mkdir(exist_ok=True)
-        
-        # Save session metadata
-        metadata = {
-            "session_id": session_id,
-            "session_name": session_name,
-            "created_at": timestamp,
-            "model_config": model_config,
-            "training_config": training_config,
-            "checkpoints": [],
-            "best_metric": None,
-            "status": "active"
-        }
-        
-        with open(session_dir / "metadata.json", "w") as f:
-            json.dump(metadata, f, indent=2, default=str)
-        
-        logger.info(f"Created training session: {session_id}")
-        return session_id
-    
-    @with_error_recovery(recovery_value=False)
-    def save_checkpoint(self,
-                       model: torch.nn.Module,
-                       session_id: str,
-                       epoch: int,
-                       metrics: Dict[str, float],
-                       optimizer_state: Optional[Dict] = None,
-                       scheduler_state: Optional[Dict] = None,
-                       additional_data: Optional[Dict] = None) -> bool:
-        """Save checkpoint with comprehensive metadata."""
-        
-        session_dir = self.sessions_dir / session_id
-        if not session_dir.exists():
-            raise ValueError(f"Training session {session_id} not found")
-        
-        # Create checkpoint directory
-        checkpoint_name = f"checkpoint_epoch_{epoch:04d}"
-        checkpoint_dir = session_dir / checkpoint_name
-        checkpoint_dir.mkdir(exist_ok=True)
-        
-        # Save model state
-        model_path = checkpoint_dir / "model.pt"
-        torch.save({
-            'model_state_dict': model.state_dict(),
-            'epoch': epoch,
-            'metrics': metrics,
-            'model_config': getattr(model, 'config', {}),
-            'timestamp': datetime.now().isoformat()
-        }, model_path)
-        
-        # Save optimizer state if provided
-        if optimizer_state:
-            torch.save(optimizer_state, checkpoint_dir / "optimizer.pt")
-        
-        # Save scheduler state if provided  
-        if scheduler_state:
-            torch.save(scheduler_state, checkpoint_dir / "scheduler.pt")
-        
-        # Save additional data
-        if additional_data:
-            with open(checkpoint_dir / "additional_data.json", "w") as f:
-                json.dump(additional_data, f, indent=2, default=str)
-        
-        # Update session metadata
-        self._update_session_metadata(session_id, checkpoint_name, metrics)
-        
-        # Cleanup old checkpoints to save space
-        self._cleanup_old_checkpoints(session_dir)
-        
-        logger.info(f"Saved checkpoint {checkpoint_name} for session {session_id}")
-        return True
-    
-    def load_checkpoint(self,
-                       session_id: str,
-                       checkpoint_name: Optional[str] = None,
-                       model: Optional[torch.nn.Module] = None) -> Dict[str, Any]:
-        """Load checkpoint with all associated data."""
-        
-        session_dir = self.sessions_dir / session_id
-        if not session_dir.exists():
-            raise ValueError(f"Training session {session_id} not found")
-        
-        # Use latest checkpoint if none specified
-        if checkpoint_name is None:
-            checkpoints = [d for d in session_dir.iterdir() 
-                         if d.is_dir() and d.name.startswith("checkpoint_")]
-            if not checkpoints:
-                raise ValueError(f"No checkpoints found for session {session_id}")
-            checkpoint_name = max(checkpoints, key=lambda x: x.name).name
-        
-        checkpoint_dir = session_dir / checkpoint_name
-        if not checkpoint_dir.exists():
-            raise ValueError(f"Checkpoint {checkpoint_name} not found in session {session_id}")
-        
-        # Load model state
-        model_path = checkpoint_dir / "model.pt"
-        checkpoint_data = torch.load(model_path, map_location='cpu', weights_only=False)
-        
-        if model is not None:
-            model.load_state_dict(checkpoint_data['model_state_dict'])
-        
-        # Load optimizer state if exists
-        optimizer_state = None
-        optimizer_path = checkpoint_dir / "optimizer.pt"
-        if optimizer_path.exists():
-            optimizer_state = torch.load(optimizer_path, map_location='cpu', weights_only=False)
-        
-        # Load scheduler state if exists
-        scheduler_state = None
-        scheduler_path = checkpoint_dir / "scheduler.pt"
-        if scheduler_path.exists():
-            scheduler_state = torch.load(scheduler_path, map_location='cpu', weights_only=False)
-        
-        # Load additional data if exists
-        additional_data = {}
-        additional_path = checkpoint_dir / "additional_data.json"
-        if additional_path.exists():
-            with open(additional_path) as f:
-                additional_data = json.load(f)
-        
-        return {
-            'model_data': checkpoint_data,
-            'optimizer_state': optimizer_state,
-            'scheduler_state': scheduler_state,
-            'additional_data': additional_data,
-            'checkpoint_path': str(checkpoint_dir)
-        }
-    
-    def save_best_model(self,
-                       session_id: str,
-                       model: torch.nn.Module,
-                       metric_name: str,
-                       metric_value: float,
-                       is_better_func: callable = lambda x, y: x > y) -> bool:
-        """Save model if it achieves best performance."""
-        
-        best_model_path = self.best_models_dir / f"{session_id}_best.pt"
-        best_meta_path = self.best_models_dir / f"{session_id}_best_meta.json"
-        
-        # Check if this is the best model so far
-        current_best = None
-        if best_meta_path.exists():
-            with open(best_meta_path) as f:
-                current_best = json.load(f)
-        
-        if current_best is None or is_better_func(metric_value, current_best['metric_value']):
-            # Save new best model
-            torch.save({
-                'model_state_dict': model.state_dict(),
-                'metric_name': metric_name,
-                'metric_value': metric_value,
-                'session_id': session_id,
-                'timestamp': datetime.now().isoformat()
-            }, best_model_path)
-            
-            # Save metadata
-            with open(best_meta_path, "w") as f:
-                json.dump({
-                    'metric_name': metric_name,
-                    'metric_value': metric_value,
-                    'session_id': session_id,
-                    'timestamp': datetime.now().isoformat()
-                }, f, indent=2)
-            
-            logger.info(f"New best model saved for session {session_id}: {metric_name}={metric_value}")
-            return True
-        
-        return False
-    
-    def push_to_hf(self,
-                  session_id: str,
-                  checkpoint_name: Optional[str] = None,
-                  include_optimizer: bool = False) -> bool:
-        """Push checkpoint to HuggingFace Hub."""
-        
-        if not self.api:
-            logger.error("HuggingFace API not available - check token")
-            return False
-        
-        try:
-            checkpoint_data = self.load_checkpoint(session_id, checkpoint_name)
-            checkpoint_dir = Path(checkpoint_data['checkpoint_path'])
-            
-            # Upload model weights
-            self.api.upload_file(
-                path_or_fileobj=str(checkpoint_dir / "model.pt"),
-                path_in_repo=f"checkpoints/{session_id}/model.pt",
-                repo_id=self.hf_repo_id,
-                commit_message=f"Upload checkpoint {checkpoint_name or 'latest'} from session {session_id}"
-            )
-            
-            # Upload optimizer state if requested and exists
-            if include_optimizer and (checkpoint_dir / "optimizer.pt").exists():
-                self.api.upload_file(
-                    path_or_fileobj=str(checkpoint_dir / "optimizer.pt"),
-                    path_in_repo=f"checkpoints/{session_id}/optimizer.pt",
-                    repo_id=self.hf_repo_id
-                )
-            
-            logger.info(f"Successfully pushed checkpoint to HuggingFace: {self.hf_repo_id}")
-            return True
-            
-        except Exception as e:
-            logger.error(f"Failed to push to HuggingFace: {e}")
-            return False
-    
-    def pull_from_hf(self,
-                    session_id: str,
-                    local_session_id: Optional[str] = None) -> bool:
-        """Pull checkpoint from HuggingFace Hub."""
-        
-        if not self.api:
-            logger.error("HuggingFace API not available - check token")
-            return False
-        
-        try:
-            local_session = local_session_id or session_id
-            local_dir = self.sessions_dir / local_session / "checkpoint_from_hf"
-            local_dir.mkdir(parents=True, exist_ok=True)
-            
-            # Download model weights
-            model_file = hf_hub_download(
-                repo_id=self.hf_repo_id,
-                filename=f"checkpoints/{session_id}/model.pt",
-                local_dir=str(local_dir),
-                local_dir_use_symlinks=False
-            )
-            
-            logger.info(f"Successfully pulled checkpoint from HuggingFace to {local_dir}")
-            return True
-            
-        except Exception as e:
-            logger.error(f"Failed to pull from HuggingFace: {e}")
-            return False
-    
-    def get_storage_usage(self) -> Dict[str, Any]:
-        """Get detailed storage usage breakdown."""
-        
-        def get_dir_size(path: Path) -> int:
-            total = 0
-            for item in path.rglob('*'):
-                if item.is_file():
-                    total += item.stat().st_size
-            return total
-        
-        usage = {
-            'total_gb': get_dir_size(self.base_dir) / 1e9,
-            'sessions_gb': get_dir_size(self.sessions_dir) / 1e9,
-            'best_models_gb': get_dir_size(self.best_models_dir) / 1e9,
-            'num_sessions': len(list(self.sessions_dir.iterdir())),
-            'num_best_models': len(list(self.best_models_dir.glob('*_best.pt'))),
-        }
-        
-        # Get per-session breakdown
-        sessions = []
-        for session_dir in self.sessions_dir.iterdir():
-            if session_dir.is_dir():
-                sessions.append({
-                    'session_id': session_dir.name,
-                    'size_gb': get_dir_size(session_dir) / 1e9,
-                    'num_checkpoints': len(list(session_dir.glob('checkpoint_*')))
-                })
-        
-        usage['sessions'] = sorted(sessions, key=lambda x: x['size_gb'], reverse=True)
-        
-        return usage
-    
-    def _update_session_metadata(self, session_id: str, checkpoint_name: str, metrics: Dict[str, float]):
-        """Update session metadata with new checkpoint info."""
-        metadata_path = self.sessions_dir / session_id / "metadata.json"
-        
-        with open(metadata_path) as f:
-            metadata = json.load(f)
-        
-        metadata['checkpoints'].append({
-            'name': checkpoint_name,
-            'metrics': metrics,
-            'timestamp': datetime.now().isoformat()
-        })
-        
-        # Update best metric if applicable
-        if 'loss' in metrics:
-            if metadata['best_metric'] is None or metrics['loss'] < metadata['best_metric'].get('loss', float('inf')):
-                metadata['best_metric'] = metrics.copy()
-        
-        with open(metadata_path, "w") as f:
-            json.dump(metadata, f, indent=2, default=str)
-    
-    def _cleanup_old_checkpoints(self, session_dir: Path):
-        """Remove oldest checkpoints to stay within limits."""
-        checkpoints = sorted([d for d in session_dir.iterdir() 
-                            if d.is_dir() and d.name.startswith("checkpoint_")],
-                           key=lambda x: x.stat().st_mtime)
-        
-        while len(checkpoints) > self.max_local_checkpoints:
-            old_checkpoint = checkpoints.pop(0)
-            shutil.rmtree(old_checkpoint)
-            logger.info(f"Cleaned up old checkpoint: {old_checkpoint.name}")
-
-
-# Convenience functions for easy usage
-def create_checkpoint_manager(hf_token: str = "os.environ.get('HF_TOKEN', 'your-token-here')") -> EnhancedCheckpointManager:
-    """Create a pre-configured checkpoint manager for this environment."""
-    return EnhancedCheckpointManager(
-        base_dir="/data/checkpoints",
-        hf_repo_id="WCNegentropy/BitTransformerLM", 
-        hf_token=hf_token,
-        max_local_checkpoints=3  # Conservative for 20GB storage
-    )
-
-
-if __name__ == "__main__":
-    # Demo usage
-    manager = create_checkpoint_manager()
-    usage = manager.get_storage_usage()
-    print(f"Current storage usage: {usage['total_gb']:.2f} GB")
-    print(f"Number of training sessions: {usage['num_sessions']}")

example.py

@@ -1,6 +0,0 @@
-from bit_transformer import example_training_step
-
-if __name__ == "__main__":
-    loss, telemetry = example_training_step()
-    print("Training loss:", loss)
-    print("Available telemetry:", list(telemetry.keys()))

full_bits_train.py

@@ -1,51 +0,0 @@
-import pathlib
-import torch
-from bit_transformer import BitTransformerLM
-
-DATA_PATH = pathlib.Path('full_bits.pt')
-
-class BitSeq(torch.utils.data.IterableDataset):
-    def __init__(self, path: str | pathlib.Path = DATA_PATH, seq: int = 2048) -> None:
-        self.bits = torch.load(path, mmap=True)
-        self.seq = seq
-
-    def __len__(self) -> int:
-        return (self.bits.numel() // self.seq) - 1
-
-    def __iter__(self):
-        N = (self.bits.numel() // self.seq) - 1
-        for i in range(N):
-            s = i * self.seq
-            yield (
-                self.bits[s:s+self.seq].long(),
-                self.bits[s+1:s+self.seq+1].long(),
-            )
-
-def main() -> None:
-    dl = torch.utils.data.DataLoader(
-        BitSeq(DATA_PATH, seq=2048),
-        batch_size=8,
-        num_workers=0,
-        pin_memory=False,
-    )
-
-    model = BitTransformerLM(
-        d_model=64,
-        nhead=4,
-        num_layers=2,
-        dim_feedforward=256,
-        max_seq_len=2048,
-        reversible=True,
-        use_autocast=True,
-    )
-
-    loss_fn = torch.nn.CrossEntropyLoss()
-    xb, yb = next(iter(dl))
-    logits, _ = model(xb)
-    pred = logits.reshape(-1, 2)
-    target = yb.reshape(-1)
-    loss = loss_fn(pred, target)
-    print('Batch loss:', float(loss))
-
-if __name__ == '__main__':
-    main()

integration_flow.py

@@ -1,110 +0,0 @@
-import torch
-from torch.profiler import profile
-from bit_transformer import (
-    BitTransformerLM,
-    quantize_dynamic,
-    hil_safe_inference,
-    collapse_submodel,
-)
-from bit_transformer.training import train_loop
-from bit_transformer.torch_utils import cpu_autocast
-
-def train(
-    model: BitTransformerLM,
-    data: torch.Tensor,
-    epochs: int = 3,
-    compress_prob: float = 0.5,
-    direct_prob: float = 0.0,
-    log: bool = False,
-    forward_kwargs: dict | None = None,
-) -> list[dict]:
-    """Train on bit sequences with optional random compression.
-
-    If ``direct_prob`` is positive, some batches are fed using their
-    run-length encoded representation packed into bits. Loss on these
-    direct-compressed batches is tracked separately.
-
-    Returns a list of per-epoch metric dictionaries containing raw and
-    compressed loss/accuracy statistics and the mean compression ratio.
-    """
-    return train_loop(
-        model,
-        data,
-        epochs=epochs,
-        compress_prob=compress_prob,
-        direct_prob=direct_prob,
-        log=log,
-        forward_kwargs=forward_kwargs,
-    )
-
-
-def main() -> None:
-    data = torch.randint(0, 2, (64, 128), dtype=torch.long)
-    validation_bits = torch.randint(0, 2, (16, 128), dtype=torch.long)
-    input_bits = torch.randint(0, 2, (1, 128), dtype=torch.long)
-    bit_sequence_data = data.tolist()
-
-    model = BitTransformerLM(
-        d_model=32,
-        nhead=4,
-        num_layers=1,
-        dim_feedforward=64,
-        max_seq_len=128,
-        use_act=True,
-        act_threshold=0.7,
-        reversible=True,
-        chunk_size=128,
-    )
-
-    for step in range(1, 13):
-        if step % 2 == 0:
-            model = model.double_width()
-        else:
-            model = model.double_layers()
-        train(model, data, epochs=3, compress_prob=0.5, log=True)
-        _, telemetry = model(validation_bits)
-        K = telemetry["negentropy_logits"].mean().item()
-        C = telemetry["lz_complexity_logits"].mean().item()
-        S = telemetry["symbiosis_score"].mean().item()
-        assert (
-            K > 0.3 and C > 0.35 and S > 0.5
-        ), f"Step {step} telemetry floor failure"
-
-    with cpu_autocast():
-        model(input_bits)
-
-    quantized_model = quantize_dynamic(model)
-    quantized_model.eval()
-
-    safe_output, _ = hil_safe_inference(
-        quantized_model, input_bits, c_floor=0.35, s_floor=0.5
-    )
-
-    student_model, _ = collapse_submodel(
-        bit_sequence_data,
-        target_params=dict(
-            d_model=16,
-            nhead=4,
-            num_layers=1,
-            dim_feedforward=32,
-            max_seq_len=128,
-        ),
-        floors={"negentropy": 0.3, "lz_complexity": 0.35, "symbiosis_score": 0.5},
-    )
-
-    compiled_model = (
-        torch.compile(student_model)
-        if hasattr(torch, "compile")
-        else student_model
-    )
-    compiled_model.eval()
-
-    with profile() as prof:
-        compiled_model(input_bits)
-
-    prof.export_chrome_trace("trace12.json")
-    print("Safe output bits:", safe_output.squeeze(0).tolist())
-
-
-if __name__ == "__main__":
-    main()

integration_schedule.py

@@ -1,379 +0,0 @@
-import os
-import time
-import math
-from itertools import cycle
-from typing import Optional
-
-import torch
-import torch.nn.functional as F
-from bit_transformer import (
-    BitTransformerLM,
-    text_to_bits,
-    quantize_dynamic,
-    prepare_qat_fx,
-    convert_qat_fx,
-    hil_safe_inference,
-    collapse_submodel,
-    diffusion_inference,
-    TelemetrySynthesizer,
-    save_distilled_model,
-)
-from bit_transformer.training import train_loop as train
-from bit_transformer.optimization import configure_optimizer, adjust_learning_rate
-from bit_transformer.utils import save_model, load_model, set_dropout
-from bit_transformer.torch_utils import cpu_autocast
-
-
-def lines_to_tensor(lines, max_len):
-    seqs = []
-    for text in lines:
-        bits = text_to_bits(text)[:max_len]
-        if len(bits) < max_len:
-            bits.extend([0] * (max_len - len(bits)))
-        seqs.append(bits)
-    return torch.tensor(seqs, dtype=torch.long)
-
-
-def load_wikitext(dataset_size=128, max_len=64):
-    try:
-        from datasets import load_dataset
-
-        ds = load_dataset("wikitext", "wikitext-2-raw-v1")
-        train_lines = [t for t in ds["train"]["text"] if t.strip()][:dataset_size]
-        valid_split = max(1, dataset_size // 4)
-        valid_lines = [t for t in ds["validation"]["text"] if t.strip()][:valid_split]
-        train = lines_to_tensor(train_lines, max_len)
-        valid = lines_to_tensor(valid_lines, max_len)
-        return train, valid, train_lines
-    except Exception as e:
-        print("Dataset load failed, using random bits", e)
-        train = torch.randint(0, 2, (dataset_size, max_len), dtype=torch.long)
-        valid = torch.randint(0, 2, (max_len, max_len), dtype=torch.long)
-        return train, valid, ["" for _ in range(len(train))]
-
-
-def _warmup(
-    model: BitTransformerLM,
-    data: torch.Tensor,
-    steps: int = 5,
-    freeze_old: bool = False,
-    old_layers: int = 0,
-    *,
-    diffusion: bool = False,
-    curriculum: bool = False,
-    optimizer: Optional[torch.optim.Optimizer] = None,
-    scheduler: Optional[torch.optim.lr_scheduler._LRScheduler] = None,
-) -> None:
-    """Run a short warm-up loop after expansion."""
-    model.train()
-    set_dropout(model, 0.1)
-    if freeze_old:
-        for idx, layer in enumerate(model.layers):
-            if idx < old_layers:
-                for p in layer.parameters():
-                    p.requires_grad_(False)
-    if optimizer is None or scheduler is None:
-        optimizer, scheduler = configure_optimizer(model, lr=1e-3, total_steps=steps)
-    it = iter(data.split(8))
-    for idx in range(steps):
-        try:
-            batch = next(it)
-        except StopIteration:
-            it = iter(data.split(8))
-            batch = next(it)
-        if diffusion:
-            p = 0.5 * (1 - idx / max(1, steps - 1)) if curriculum else 0.5
-            noise = (torch.rand_like(batch.float()) < p).long()
-            noisy = batch ^ noise
-            logits, _ = model(noisy, causal=False)
-            pred = logits.reshape(-1, 2)
-            target = batch.reshape(-1)
-        else:
-            logits, _ = model(batch)
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = batch[:, 1:].reshape(-1)
-        loss = F.cross_entropy(pred, target)
-        loss.backward()
-        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
-        optimizer.step()
-        scheduler.step()
-        optimizer.zero_grad()
-    for p in model.parameters():
-        p.requires_grad_(True)
-    model.eval()
-    set_dropout(model, 0.0)
-
-
-def integration_schedule(
-    steps: int = 10,
-    max_len: int = 64,
-    dataset_size: int = 128,
-    *,
-    weights_path: str = "weights/model.pt.gz",
-    plateau_steps: int = 0,
-    collapsed_path: str | None = None,
-    epochs_per_step: int = 2,
-    extra_steps: int = 3,
-    collapse: bool = True,
-    diffusion: bool = False,
-    noise_schedule: str = "linear",
-    diffusion_steps: int = 8,
-    diffusion_curriculum: bool = False,
-    use_checkpoint: bool = True,
-    reversible: bool = True,
-    improve_thresh: float = 0.01,
-    qat: bool = False,
-):
-    start = time.time()
-    train_bits, valid_bits, train_lines = load_wikitext(dataset_size, max_len)
-    if os.path.exists(weights_path):
-        try:
-            model = load_model(weights_path)
-            print(f"Loaded model from {weights_path}")
-        except Exception as e:
-            print("Failed to load weights, initializing new model", e)
-            model = BitTransformerLM(
-                d_model=32,
-                nhead=4,
-                num_layers=1,
-                dim_feedforward=64,
-                max_seq_len=max_len,
-                use_act=True,
-                act_threshold=0.7,
-                reversible=reversible,
-                chunk_size=max_len,
-                use_autocast=True,
-                use_checkpoint=use_checkpoint,
-            )
-    else:
-        model = BitTransformerLM(
-            d_model=32,
-            nhead=4,
-            num_layers=1,
-            dim_feedforward=64,
-            max_seq_len=max_len,
-            use_act=True,
-            act_threshold=0.7,
-            reversible=reversible,
-            chunk_size=max_len,
-            use_autocast=True,
-            use_checkpoint=use_checkpoint,
-        )
-    if qat:
-        model = prepare_qat_fx(model)
-    results = []
-    scale_cycle = cycle(["layers", "width", "context"])
-    base_lr = 1e-3
-    prev_val_loss: Optional[float] = None
-    for step in range(steps):
-        model.train()
-        set_dropout(model, 0.1)
-        opt, sched = configure_optimizer(
-            model, lr=base_lr, total_steps=epochs_per_step
-        )
-        train(
-            model,
-            train_bits,
-            epochs=epochs_per_step,
-            extra_steps=extra_steps,
-            compress_prob=0.0 if diffusion else 1.0,
-            log=True,
-            diffusion=diffusion,
-            diffusion_curriculum=diffusion_curriculum,
-            optimizer=opt,
-            scheduler=sched,
-        )
-
-        model.eval()
-        set_dropout(model, 0.0)
-        with torch.no_grad():
-            logits, telemetry = model(valid_bits, causal=not diffusion)
-            if diffusion:
-                pred = logits.reshape(-1, 2)
-                target = valid_bits.reshape(-1)
-            else:
-                pred = logits[:, :-1, :].reshape(-1, 2)
-                target = valid_bits[:, 1:].reshape(-1)
-            val_loss = F.cross_entropy(pred, target).item()
-            k = telemetry["negentropy_logits"].mean().item()
-            c = telemetry["lz_complexity_logits"].mean().item()
-            s = telemetry["symbiosis_score"].mean().item()
-        print(f"Step {step} validation loss: {val_loss:.4f} K={k:.3f} C={c:.3f} S={s:.3f}")
-        results.append((step, val_loss, k, c, s))
-
-        if prev_val_loss is not None and prev_val_loss - val_loss < improve_thresh:
-            strategy = next(scale_cycle)
-            base_lr = adjust_learning_rate(opt, 1 / math.sqrt(2))
-            if strategy == "layers":
-                old_layers = model.num_layers
-                model = model.double_layers()
-                warm_opt, warm_sched = configure_optimizer(
-                    model, lr=base_lr, total_steps=100
-                )
-                _warmup(
-                    model,
-                    train_bits,
-                    steps=100,
-                    freeze_old=True,
-                    old_layers=old_layers,
-                    diffusion=diffusion,
-                    curriculum=diffusion_curriculum,
-                    optimizer=warm_opt,
-                    scheduler=warm_sched,
-                )
-            elif strategy == "width":
-                model = model.double_width()
-                warm_opt, warm_sched = configure_optimizer(
-                    model, lr=base_lr, total_steps=100
-                )
-                _warmup(
-                    model,
-                    train_bits,
-                    steps=100,
-                    diffusion=diffusion,
-                    curriculum=diffusion_curriculum,
-                    optimizer=warm_opt,
-                    scheduler=warm_sched,
-                )
-            else:
-                max_len *= 2
-                train_bits, valid_bits, train_lines = load_wikitext(
-                    dataset_size, max_len
-                )
-                model = model.double_length()
-                warm_opt, warm_sched = configure_optimizer(
-                    model, lr=base_lr, total_steps=100
-                )
-                _warmup(
-                    model,
-                    train_bits,
-                    steps=100,
-                    diffusion=diffusion,
-                    curriculum=diffusion_curriculum,
-                    optimizer=warm_opt,
-                    scheduler=warm_sched,
-                )
-
-        prev_val_loss = val_loss
-        if time.time() - start > 8 * 60:
-            print("Time limit reached")
-            break
-
-    # optional plateau phase at final size
-    for p in range(plateau_steps):
-        model.train()
-        set_dropout(model, 0.1)
-        train(
-            model,
-            train_bits,
-            epochs=epochs_per_step,
-            extra_steps=extra_steps,
-            compress_prob=0.0 if diffusion else 1.0,
-            log=True,
-            diffusion=diffusion,
-            diffusion_curriculum=diffusion_curriculum,
-        )
-        model.eval()
-        set_dropout(model, 0.0)
-        with torch.no_grad():
-            logits, telemetry = model(valid_bits, causal=not diffusion)
-            if diffusion:
-                pred = logits.reshape(-1, 2)
-                target = valid_bits.reshape(-1)
-            else:
-                pred = logits[:, :-1, :].reshape(-1, 2)
-                target = valid_bits[:, 1:].reshape(-1)
-            val_loss = F.cross_entropy(pred, target).item()
-            k = telemetry["negentropy_logits"].mean().item()
-            c = telemetry["lz_complexity_logits"].mean().item()
-            s = telemetry["symbiosis_score"].mean().item()
-        idx = steps + p
-        print(
-            f"Plateau {p} validation loss: {val_loss:.4f} K={k:.3f} C={c:.3f} S={s:.3f}"
-        )
-        results.append((idx, val_loss, k, c, s))
-        if time.time() - start > 8 * 60:
-            print("Time limit reached")
-            break
-
-    # final validation after last step
-    model.eval()
-    set_dropout(model, 0.0)
-    with torch.no_grad():
-        logits, telemetry = model(valid_bits, causal=not diffusion)
-        if diffusion:
-            pred = logits.reshape(-1, 2)
-            target = valid_bits.reshape(-1)
-        else:
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = valid_bits[:, 1:].reshape(-1)
-        val_loss = F.cross_entropy(pred, target).item()
-        k = telemetry["negentropy_logits"].mean().item()
-        c = telemetry["lz_complexity_logits"].mean().item()
-        s = telemetry["symbiosis_score"].mean().item()
-
-    print(f"Final validation loss: {val_loss:.4f} K={k:.3f} C={c:.3f} S={s:.3f}")
-    results.append((steps + plateau_steps, val_loss, k, c, s))
-
-    # persist final model weights for future runs
-    save_model(model, weights_path)
-
-    input_bits = valid_bits[:1]
-    if qat:
-        qmodel = convert_qat_fx(model)
-    else:
-        with cpu_autocast():
-            model(input_bits)
-        qmodel = quantize_dynamic(model)
-    qmodel.eval()
-    try:
-        hil_safe_inference(
-            qmodel,
-            input_bits,
-            c_floor=0.3,
-            s_floor=0.5,
-            causal=not diffusion,
-            strict=not diffusion,
-        )
-    except RuntimeError as e:
-        print("Safety gate triggered", e)
-    collapsed = None
-    if collapse:
-        synth = TelemetrySynthesizer(n_clusters=8)
-        reps = synth.cluster_sequences(model, train_bits[:64])
-        floors = {"negentropy": 0.3, "lz_complexity": 0.35, "symbiosis_score": 0.5}
-        collapsed, metrics = collapse_submodel(
-            reps,
-            target_params=dict(
-                d_model=16,
-                nhead=4,
-                num_layers=1,
-                dim_feedforward=32,
-                max_seq_len=max_len,
-            ),
-            floors=floors,
-        )
-        collapsed.eval()
-        with torch.no_grad():
-            logits, _ = collapsed(valid_bits)
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = valid_bits[:, 1:].reshape(-1)
-            c_loss = F.cross_entropy(pred, target).item()
-        print("Collapsed model validation loss:", c_loss)
-        if collapsed_path is not None:
-            save_distilled_model(
-                collapsed,
-                collapsed_path,
-                {**metrics, "val_loss": c_loss},
-                floors=floors,
-            )
-    if diffusion:
-        sample = diffusion_inference(
-            model, length=max_len, steps=diffusion_steps, schedule=noise_schedule
-        )
-        print("Diffusion sample:", sample[0].tolist())
-    return results, collapsed
-
-
-if __name__ == "__main__":
-    integration_schedule()

progressive_scaleup.py

@@ -1,216 +0,0 @@
-"""Legacy progressive scale-up demo.
-
-This script is retained for historical reference but has been superseded by
-``integration_schedule.py`` which provides a more flexible scaling workflow.
-"""
-
-import argparse
-import warnings
-import torch
-import torch.nn.functional as F
-from bit_transformer import (
-    BitTransformerLM,
-    configure_optimizer,
-    expand_model,
-    text_to_bits,
-)
-from bit_transformer.training import train_loop as basic_train
-
-warnings.warn(
-    "progressive_scaleup.py is deprecated; use integration_schedule.py instead.",
-    DeprecationWarning,
-    stacklevel=2,
-)
-
-
-def progressive_scale_up(
-    eps: float = 0.65,
-    steps: int = 2,
-    width_mult: float = 1.0,
-    forward_kwargs: dict | None = None,
-) -> None:
-    """Demonstrate automatic scaling of the model on random data."""
-    params = dict(d_model=32, nhead=4, num_layers=1, dim_feedforward=64, max_seq_len=16)
-    model = BitTransformerLM(**params)
-    steps_per_epoch = 64 // 8
-    optimizer, scheduler = configure_optimizer(
-        model, lr=1e-3, total_steps=steps * steps_per_epoch
-    )
-
-    train = torch.randint(0, 2, (64, params["max_seq_len"]), dtype=torch.long)
-    valid = torch.randint(0, 2, (16, params["max_seq_len"]), dtype=torch.long)
-
-    for step in range(steps):
-        # one epoch over train
-        basic_train(
-            model,
-            train,
-            epochs=1,
-            compress_prob=0.5,
-            log=False,
-            forward_kwargs=forward_kwargs,
-        )
-
-        with torch.no_grad():
-            logits, _ = model(valid, **(forward_kwargs or {}))
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = valid[:, 1:].reshape(-1)
-            val_loss = F.cross_entropy(pred, target).item()
-        print(f"Step {step} validation loss: {val_loss:.4f}")
-        if val_loss < eps:
-            params["num_layers"] *= 2
-            params["d_model"] = int(params["d_model"] * width_mult)
-            params["dim_feedforward"] = int(params["dim_feedforward"] * width_mult)
-            model = expand_model(model, params)
-            optimizer, scheduler = configure_optimizer(
-                model, lr=1e-3, total_steps=steps * steps_per_epoch
-            )
-        print(
-            "Scaled model to", params["num_layers"], "layers and width", params["d_model"]
-        )
-
-
-def progressive_scale_up_text(
-    improve_thresh: float = 0.01,
-    steps: int = 2,
-    width_mult: float = 2.0,
-    max_len: int = 64,
-    dataset_size: int = 512,
-    forward_kwargs: dict | None = None,
-) -> None:
-    """Scale up using WikiText2 lines converted to bits.
-
-    Parameters
-    ----------
-    improve_thresh: float
-        Relative validation loss improvement required to avoid scaling.
-        If improvement is <= this threshold, model size is increased.
-    steps: int
-        Number of training steps.
-    width_mult: float
-        Multiplier applied when increasing model width.
-    max_len: int
-        Initial sequence length.
-    dataset_size: int
-        Number of training lines to load from WikiText2.
-    forward_kwargs: dict | None
-        Extra keyword arguments for the forward pass.
-    """
-    from datasets import load_dataset
-
-    ds = load_dataset("wikitext", "wikitext-2-raw-v1")
-    train_iter = ds["train"]["text"]
-    valid_iter = ds["validation"]["text"]
-
-    train_lines = []
-    for line in train_iter:
-        train_lines.append(line)
-        if len(train_lines) >= dataset_size:
-            break
-
-    valid_lines = []
-    for line in valid_iter:
-        valid_lines.append(line)
-        if len(valid_lines) >= dataset_size // 4:
-            break
-
-    def lines_to_tensor(lines: list[str], length: int) -> torch.Tensor:
-        seqs = []
-        for text in lines:
-            bits = text_to_bits(text)[:length]
-            if len(bits) < length:
-                bits.extend([0] * (length - len(bits)))
-            seqs.append(bits)
-        return torch.tensor(seqs, dtype=torch.long)
-
-    train = lines_to_tensor(train_lines, max_len)
-    valid = lines_to_tensor(valid_lines, max_len)
-
-    params = dict(
-        d_model=32,
-        nhead=4,
-        num_layers=1,
-        dim_feedforward=64,
-        max_seq_len=max_len,
-    )
-    model = BitTransformerLM(**params)
-    steps_per_epoch = len(train) // 8
-    optimizer, scheduler = configure_optimizer(
-        model, lr=1e-3, total_steps=steps * max(1, steps_per_epoch)
-    )
-
-    prev_loss: float | None = None
-    scale_length = True
-
-    for step in range(steps):
-        basic_train(
-            model,
-            train,
-            epochs=1,
-            compress_prob=0.5,
-            log=False,
-            forward_kwargs=forward_kwargs,
-        )
-
-        with torch.no_grad():
-            logits, _ = model(valid, **(forward_kwargs or {}))
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = valid[:, 1:].reshape(-1)
-            val_loss = F.cross_entropy(pred, target).item()
-        print(f"Step {step} validation loss: {val_loss:.4f}")
-        if prev_loss is not None:
-            improvement = (prev_loss - val_loss) / max(prev_loss, 1e-8)
-            if improvement <= improve_thresh:
-                if scale_length:
-                    params["max_seq_len"] *= 2
-                    train = lines_to_tensor(train_lines, params["max_seq_len"])
-                    valid = lines_to_tensor(valid_lines, params["max_seq_len"])
-                    model = model.double_length()
-                    steps_per_epoch = len(train) // 8
-                    scale_type = "length"
-                else:
-                    params["d_model"] = int(params["d_model"] * width_mult)
-                    params["dim_feedforward"] = int(params["dim_feedforward"] * width_mult)
-                    model = expand_model(model, params)
-                    scale_type = "width"
-                optimizer, scheduler = configure_optimizer(
-                    model, lr=1e-3, total_steps=steps * max(1, steps_per_epoch)
-                )
-                scale_length = not scale_length
-                param_count = sum(p.numel() for p in model.parameters())
-                print(
-                    f"Scaled {scale_type}; seq_len={params['max_seq_len']} width={params['d_model']} params={param_count}"
-                )
-        prev_loss = val_loss
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Progressively scale model length and width")
-    parser.add_argument("--steps", type=int, default=2, help="number of training steps")
-    parser.add_argument(
-        "--improve-thresh",
-        type=float,
-        default=0.01,
-        help="relative loss improvement required to avoid scaling",
-    )
-    parser.add_argument(
-        "--width-mult", type=float, default=2.0, help="width multiplier when scaling"
-    )
-    parser.add_argument("--causal", action="store_true", help="use causal attention during training")
-    parser.add_argument("--wikitext", action="store_true", help="use WikiText2 dataset")
-    args = parser.parse_args()
-    if args.wikitext:
-        progressive_scale_up_text(
-            improve_thresh=args.improve_thresh,
-            steps=args.steps,
-            width_mult=args.width_mult,
-            forward_kwargs={"causal": args.causal} if args.causal else None,
-        )
-    else:
-        progressive_scale_up(
-            eps=args.improve_thresh,
-            steps=args.steps,
-            width_mult=args.width_mult,
-            forward_kwargs={"causal": args.causal} if args.causal else None,
-        )
-

quick_training_run.py

@@ -1,339 +0,0 @@
-#!/usr/bin/env python3
-"""
-Full end-to-end BitTransformerLM training run with all optimizations!
-Small scale test to validate our enhanced system.
-"""
-
-import torch
-import torch.nn as nn
-import torch.optim as optim
-from torch.utils.data import Dataset, DataLoader
-import numpy as np
-import logging
-from pathlib import Path
-import time
-from typing import List, Dict, Any
-
-# Import our enhanced modules
-from bit_transformer.model import BitTransformerLM
-from bit_transformer.compression import compress_bits_batch, model_output_decompress
-from bit_transformer.error_handling import safe_model_forward, setup_error_logging
-from bit_transformer.types import BitSequence, TelemetryDict
-from enhanced_checkpoint_system import create_checkpoint_manager
-
-# Setup logging
-logger = setup_error_logging("INFO")
-
-class SimpleBitDataset(Dataset):
-    """Simple dataset of bit sequences for training."""
-    
-    def __init__(self, num_samples: int = 1000, seq_length: int = 128):
-        self.num_samples = num_samples
-        self.seq_length = seq_length
-        self.data = self._generate_bit_sequences()
-    
-    def _generate_bit_sequences(self) -> List[torch.Tensor]:
-        """Generate diverse bit sequences with different patterns."""
-        sequences = []
-        
-        # Pattern 1: Alternating sequences
-        for i in range(self.num_samples // 4):
-            pattern = torch.tensor([i % 2] * self.seq_length, dtype=torch.long)
-            sequences.append(pattern)
-        
-        # Pattern 2: Random sequences
-        for i in range(self.num_samples // 4):
-            pattern = torch.randint(0, 2, (self.seq_length,), dtype=torch.long)
-            sequences.append(pattern)
-        
-        # Pattern 3: Structured patterns (runs)
-        for i in range(self.num_samples // 4):
-            pattern = []
-            pos = 0
-            while pos < self.seq_length:
-                run_length = min(np.random.randint(1, 20), self.seq_length - pos)
-                bit_value = np.random.randint(0, 2)
-                pattern.extend([bit_value] * run_length)
-                pos += run_length
-            pattern = torch.tensor(pattern[:self.seq_length], dtype=torch.long)
-            sequences.append(pattern)
-        
-        # Pattern 4: Fibonacci-like sequences
-        remaining = self.num_samples - len(sequences)
-        for i in range(remaining):
-            pattern = [0, 1]
-            while len(pattern) < self.seq_length:
-                pattern.append(pattern[-1] ^ pattern[-2])  # XOR of last two bits
-            pattern = torch.tensor(pattern[:self.seq_length], dtype=torch.long)
-            sequences.append(pattern)
-        
-        return sequences
-    
-    def __len__(self):
-        return len(self.data)
-    
-    def __getitem__(self, idx):
-        sequence = self.data[idx]
-        # For language modeling, input is sequence[:-1], target is sequence[1:]
-        return sequence[:-1], sequence[1:]
-
-
-def compute_safety_metrics(predictions: torch.Tensor, targets: torch.Tensor) -> Dict[str, float]:
-    """Compute K/C/S safety metrics."""
-    pred_bits = (predictions > 0.5).float().flatten()
-    
-    # K metric (Negentropy): Measure of order vs randomness
-    if len(pred_bits) > 0:
-        prob_1 = pred_bits.mean().item()
-        prob_0 = 1 - prob_1
-        if prob_0 > 0 and prob_1 > 0:
-            entropy = -prob_0 * np.log2(prob_0) - prob_1 * np.log2(prob_1)
-            negentropy = 1.0 - entropy  # Higher = more ordered
-        else:
-            negentropy = 1.0 if prob_1 == 1.0 or prob_1 == 0.0 else 0.0
-    else:
-        negentropy = 0.0
-    
-    # C metric (Complexity): Simple run-length approximation
-    changes = (pred_bits[1:] != pred_bits[:-1]).sum().item()
-    complexity = min(changes / len(pred_bits), 1.0) if len(pred_bits) > 1 else 0.0
-    
-    # S metric (Symbiosis): Alignment with target distribution
-    target_bits = targets.float().flatten()
-    if len(target_bits) > 0:
-        target_mean = target_bits.mean()
-        pred_mean = pred_bits.mean()
-        symbiosis = 1.0 - abs(target_mean - pred_mean).item()
-    else:
-        symbiosis = 1.0
-    
-    return {
-        'K_negentropy': negentropy,
-        'C_complexity': complexity, 
-        'S_symbiosis': symbiosis
-    }
-
-
-def train_bittransformer():
-    """Main training function with all optimizations."""
-    
-    logger.info("🚀 Starting BitTransformerLM end-to-end training run!")
-    
-    # Model configuration - small but meaningful
-    model_config = {
-        'd_model': 256,
-        'nhead': 8,
-        'num_layers': 4,
-        'dim_feedforward': 512,
-        'max_seq_len': 128,
-        'use_checkpoint': True,
-        'chunk_size': None,  # Disable chunking for small model
-    }
-    
-    training_config = {
-        'batch_size': 16,
-        'learning_rate': 1e-3,
-        'num_epochs': 10,
-        'save_every_n_epochs': 2,
-        'log_every_n_steps': 10
-    }
-    
-    # Initialize enhanced checkpoint manager
-    checkpoint_manager = create_checkpoint_manager()
-    session_id = checkpoint_manager.create_training_session(
-        session_name="end_to_end_test",
-        model_config=model_config,
-        training_config=training_config
-    )
-    
-    logger.info(f"📝 Created training session: {session_id}")
-    
-    # Create dataset and dataloader
-    logger.info("📊 Creating training dataset...")
-    dataset = SimpleBitDataset(num_samples=800, seq_length=model_config['max_seq_len'])
-    dataloader = DataLoader(dataset, batch_size=training_config['batch_size'], shuffle=True)
-    
-    # Initialize model
-    logger.info("🧠 Initializing BitTransformerLM model...")
-    model = BitTransformerLM(
-        d_model=model_config['d_model'],
-        nhead=model_config['nhead'],
-        num_layers=model_config['num_layers'],
-        dim_feedforward=model_config['dim_feedforward'],
-        max_seq_len=model_config['max_seq_len'],
-        use_checkpoint=model_config['use_checkpoint'],
-        chunk_size=model_config['chunk_size']
-    )
-    
-    # Count parameters
-    total_params = sum(p.numel() for p in model.parameters())
-    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
-    logger.info(f"🔢 Model parameters: {total_params:,} total, {trainable_params:,} trainable")
-    
-    # Setup optimizer and loss
-    optimizer = optim.AdamW(model.parameters(), lr=training_config['learning_rate'])
-    scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=training_config['num_epochs'])
-    criterion = nn.CrossEntropyLoss()
-    
-    # Training loop
-    logger.info("🏃‍♂️ Starting training loop...")
-    
-    for epoch in range(training_config['num_epochs']):
-        model.train()
-        epoch_loss = 0.0
-        epoch_metrics = {'K_negentropy': 0.0, 'C_complexity': 0.0, 'S_symbiosis': 0.0}
-        num_batches = 0
-        
-        start_time = time.time()
-        
-        for batch_idx, (inputs, targets) in enumerate(dataloader):
-            optimizer.zero_grad()
-            
-            # Forward pass with safety monitoring
-            try:
-                # BitTransformerLM returns (logits, telemetry)
-                output = safe_model_forward(model, inputs)
-                if isinstance(output, tuple):
-                    logits, telemetry = output
-                else:
-                    logits = output
-                    telemetry = {}
-                
-                # BitTransformerLM outputs logits for binary classification
-                # Shape should be [batch, seq_len, 2] for binary vocab
-                if logits.dim() == 2:
-                    # If [batch*seq_len, 2], already flattened
-                    logits_flat = logits
-                    targets_flat = targets.reshape(-1)
-                else:
-                    # If [batch, seq_len, 2], flatten
-                    logits_flat = logits.reshape(-1, 2)
-                    targets_flat = targets.reshape(-1)
-                
-                loss = criterion(logits_flat, targets_flat)
-                
-                # Backward pass
-                loss.backward()
-                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
-                optimizer.step()
-                
-                # Compute metrics
-                with torch.no_grad():
-                    # Handle different logits shapes for predictions
-                    if logits.dim() == 2:
-                        # [batch*seq_len, 2] -> reshape back to [batch, seq_len, 2]
-                        batch_size = inputs.shape[0]
-                        seq_len = inputs.shape[1]
-                        logits_reshaped = logits.reshape(batch_size, seq_len, 2)
-                        predictions = torch.softmax(logits_reshaped, dim=-1)[:, :, 1]  # Prob of bit=1
-                    else:
-                        # [batch, seq_len, 2]
-                        predictions = torch.softmax(logits, dim=-1)[:, :, 1]  # Prob of bit=1
-                    
-                    safety_metrics = compute_safety_metrics(predictions, targets)
-                    
-                    epoch_loss += loss.item()
-                    for key, value in safety_metrics.items():
-                        epoch_metrics[key] += value
-                    num_batches += 1
-                
-                # Logging
-                if batch_idx % training_config['log_every_n_steps'] == 0:
-                    logger.info(f"Epoch {epoch+1}/{training_config['num_epochs']}, "
-                              f"Batch {batch_idx}/{len(dataloader)}, "
-                              f"Loss: {loss.item():.4f}, "
-                              f"K: {safety_metrics['K_negentropy']:.3f}, "
-                              f"C: {safety_metrics['C_complexity']:.3f}, "
-                              f"S: {safety_metrics['S_symbiosis']:.3f}")
-                
-            except Exception as e:
-                logger.error(f"Error in batch {batch_idx}: {e}")
-                continue
-        
-        # End of epoch processing
-        scheduler.step()
-        epoch_time = time.time() - start_time
-        
-        if num_batches > 0:
-            avg_loss = epoch_loss / num_batches
-            avg_metrics = {k: v / num_batches for k, v in epoch_metrics.items()}
-            
-            logger.info(f"✅ Epoch {epoch+1} completed in {epoch_time:.2f}s")
-            logger.info(f"📊 Avg Loss: {avg_loss:.4f}")
-            logger.info(f"📈 Safety Metrics - K: {avg_metrics['K_negentropy']:.3f}, "
-                       f"C: {avg_metrics['C_complexity']:.3f}, "
-                       f"S: {avg_metrics['S_symbiosis']:.3f}")
-            
-            # Save checkpoint
-            if (epoch + 1) % training_config['save_every_n_epochs'] == 0:
-                checkpoint_success = checkpoint_manager.save_checkpoint(
-                    model=model,
-                    session_id=session_id,
-                    epoch=epoch + 1,
-                    metrics={
-                        'loss': avg_loss,
-                        'learning_rate': scheduler.get_last_lr()[0],
-                        **avg_metrics
-                    },
-                    optimizer_state=optimizer.state_dict(),
-                    scheduler_state=scheduler.state_dict()
-                )
-                
-                if checkpoint_success:
-                    logger.info(f"💾 Checkpoint saved for epoch {epoch+1}")
-                
-                # Save best model if loss improved
-                checkpoint_manager.save_best_model(
-                    session_id=session_id,
-                    model=model,
-                    metric_name='loss',
-                    metric_value=avg_loss,
-                    is_better_func=lambda x, y: x < y  # Lower loss is better
-                )
-    
-    logger.info("🎉 Training completed successfully!")
-    
-    # Test inference and compression
-    logger.info("🧪 Testing model inference and compression...")
-    
-    model.eval()
-    with torch.no_grad():
-        # Create a test sequence
-        test_input = torch.randint(0, 2, (1, 64), dtype=torch.long)
-        logger.info(f"📥 Input sequence: {test_input.squeeze().tolist()}")
-        
-        # Model inference
-        output_logits = model(test_input)
-        output_probs = torch.softmax(output_logits, dim=-1)
-        predicted_bits = torch.argmax(output_probs, dim=-1)
-        
-        logger.info(f"📤 Predicted sequence: {predicted_bits.squeeze().tolist()}")
-        
-        # Test compression
-        compressed = compress_bits_batch(predicted_bits)
-        logger.info(f"🗜️ Compressed length: {len(compressed[0])} (original: {predicted_bits.shape[-1]})")
-        
-        # Decompress to verify
-        decompressed = model_output_decompress(compressed)
-        compression_success = torch.equal(predicted_bits, decompressed)
-        logger.info(f"✅ Compression/decompression successful: {compression_success}")
-    
-    # Final storage usage report
-    storage_usage = checkpoint_manager.get_storage_usage()
-    logger.info(f"💾 Final storage usage: {storage_usage['total_gb']:.3f} GB")
-    logger.info(f"📁 Training sessions: {storage_usage['num_sessions']}")
-    
-    return session_id, model, checkpoint_manager
-
-
-if __name__ == "__main__":
-    try:
-        session_id, trained_model, manager = train_bittransformer()
-        print(f"\n🎉 SUCCESS! Training session completed: {session_id}")
-        print(f"🔍 Use checkpoint_manager.load_checkpoint('{session_id}') to resume")
-        
-    except Exception as e:
-        logger.error(f"❌ Training failed: {e}")
-        import traceback
-        traceback.print_exc()
-        raise

scripts/tools/sync_to_hf.py

@@ -6,6 +6,7 @@ Uploads all cleaned documentation and code with proper commit message.
 
 import os
 import logging
+import re
 from pathlib import Path
 from huggingface_hub import HfApi, login
 from typing import Optional, List
@@ -14,6 +15,37 @@ from typing import Optional, List
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 logger = logging.getLogger(__name__)
 
+def scan_for_secrets(file_path: Path) -> List[str]:
+    """Scan a file for potential secrets and tokens."""
+    secrets_found = []
+
+    # Patterns for common secrets
+    secret_patterns = {
+        'HuggingFace Token': r'hf_[A-Za-z0-9_]{30,}',
+        'OpenAI API Key': r'sk-[A-Za-z0-9]{48}',
+        'GitHub Token': r'gh[pousr]_[A-Za-z0-9_]{36,}',
+        'AWS Access Key': r'AKIA[0-9A-Z]{16}',
+        'Generic API Key': r'["\']?[Aa]pi[_-]?[Kk]ey["\']?\s*[:=]\s*["\']?[A-Za-z0-9_\-]{20,}["\']?',
+        'Generic Token': r'["\']?[Tt]oken["\']?\s*[:=]\s*["\']?[A-Za-z0-9_\-]{20,}["\']?',
+        'Generic Secret': r'["\']?[Ss]ecret["\']?\s*[:=]\s*["\']?[A-Za-z0-9_\-]{20,}["\']?',
+    }
+
+    try:
+        with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
+            content = f.read()
+
+        for secret_type, pattern in secret_patterns.items():
+            matches = re.finditer(pattern, content, re.IGNORECASE)
+            for match in matches:
+                line_num = content[:match.start()].count('\n') + 1
+                secrets_found.append(f"{secret_type} found at line {line_num}: {match.group()[:50]}...")
+
+    except Exception as e:
+        logger.warning(f"Could not scan {file_path} for secrets: {e}")
+
+    return secrets_found
+
+
 def get_files_to_sync(repo_root: Path) -> List[Path]:
     """Get the exact list of files that will be synced to HuggingFace."""
     # Files and directories to upload (excluding unnecessary files)
@@ -44,7 +76,7 @@ def get_files_to_sync(repo_root: Path) -> List[Path]:
         ".pytest_cache/**",
         ".ipynb_checkpoints/**",
         "weights/**",
-        "checkpoints/**",
+        "checkpoints/**",  # Contains potentially sensitive configs
         "*.log",
         "*.pt",  # Model weights
         "*.zip",  # Backup files
@@ -130,6 +162,26 @@ def sync_repository_to_hf(
         files_to_upload = get_files_to_sync(repo_root)
         logger.info(f"Found {len(files_to_upload)} files to upload")
 
+        # CRITICAL SECURITY CHECK: Scan all files for secrets
+        logger.info("🔍 Scanning files for secrets and tokens...")
+        all_secrets = []
+        for file_path in files_to_upload:
+            secrets = scan_for_secrets(file_path)
+            if secrets:
+                relative_path = file_path.relative_to(repo_root)
+                all_secrets.extend([f"{relative_path}: {secret}" for secret in secrets])
+
+        if all_secrets:
+            logger.error("🚨 SECURITY ALERT: Secrets detected in files!")
+            logger.error("The following secrets were found and MUST be removed before sync:")
+            for secret in all_secrets:
+                logger.error(f"  - {secret}")
+            logger.error("❌ SYNC ABORTED for security reasons!")
+            logger.error("Please remove all secrets and use environment variables instead.")
+            return False
+
+        logger.info("✅ Security scan passed - no secrets detected")
+
         # If preview only, just show the files and return
         if preview_only:
             preview_sync(repo_root)

scripts/training/breakthrough_training.py

@@ -0,0 +1,199 @@
+#!/usr/bin/env python3
+"""
+BREAKTHROUGH BitTransformerLM Training Script
+===========================================
+
+Using the ACTUAL BitTransformerLM model and training infrastructure,
+configured for the Fixed RL Adafactor breakthrough results.
+"""
+
+import sys
+import os
+import logging
+from pathlib import Path
+
+import torch
+from datasets import load_dataset
+from huggingface_hub import login
+
+# Add paths for imports
+sys.path.append('/data')
+sys.path.append('/data/BitTransformerLM')
+
+from bit_transformer import (
+    BitTransformerLM,
+    text_to_bits,
+    train_loop,
+    save_model,
+    load_model,
+    set_dropout
+)
+from BTLM_Extensions import configure_adafactor_optimizer
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler('breakthrough_training.log'),
+        logging.StreamHandler()
+    ]
+)
+logger = logging.getLogger(__name__)
+
+def load_and_prepare_dataset():
+    """Load HF dataset and convert to bit tensors."""
+    logger.info("Loading WCNegentropy/BitTransformerLM dataset...")
+    
+    # Login to HuggingFace
+    hf_token = os.getenv('HF_TOKEN')
+    if hf_token:
+        login(token=hf_token)
+    else:
+        print("Warning: HF_TOKEN environment variable not set")
+    
+    # Load dataset
+    dataset = load_dataset("WCNegentropy/BitTransformerLM")
+    train_data = dataset['train']
+    
+    logger.info(f"Dataset loaded: {len(train_data)} samples")
+    
+    # Process dataset - the HF dataset already has bit_sequence field!
+    bit_sequences = []
+    for sample in train_data:
+        if 'bit_sequence' in sample and sample['bit_sequence'] is not None:
+            # The bit_sequence might already be a list
+            bits = sample['bit_sequence']
+            if isinstance(bits, str):
+                try:
+                    bits = eval(bits)  # Convert string representation to list
+                except:
+                    bits = None
+            if isinstance(bits, list) and len(bits) > 0:
+                bit_sequences.append(bits)
+            else:
+                # Fallback: convert original_text to bits
+                text = sample.get('original_text', '')
+                if text:
+                    bits = text_to_bits(text)
+                    bit_sequences.append(bits)
+        else:
+            # Fallback: convert text to bits
+            text = sample.get('text', '') or sample.get('original_text', '')
+            if text:
+                bits = text_to_bits(text)
+                bit_sequences.append(bits)
+    
+    logger.info(f"Processed {len(bit_sequences)} bit sequences")
+    
+    # Create training tensors with proper sequence length
+    max_len = 512  # BitTransformerLM default max_seq_len
+    training_sequences = []
+    
+    for bits in bit_sequences:
+        # Split long sequences into chunks
+        for i in range(0, len(bits) - max_len + 1, max_len // 2):
+            seq = bits[i:i + max_len]
+            if len(seq) == max_len:  # Only use full-length sequences
+                training_sequences.append(seq)
+    
+    # Convert to tensor
+    data_tensor = torch.tensor(training_sequences, dtype=torch.long)
+    logger.info(f"Created training tensor: {data_tensor.shape}")
+    
+    return data_tensor
+
+def create_breakthrough_model():
+    """Create the EXACT breakthrough BitTransformerLM configuration."""
+    logger.info("Creating breakthrough BitTransformerLM model...")
+    
+    # EXACT breakthrough configuration using ACTUAL BitTransformerLM parameters
+    model = BitTransformerLM(
+        d_model=512,            # Breakthrough config
+        nhead=16,              # 16 attention heads  
+        num_layers=8,          # 8 layers for ~16M params
+        dim_feedforward=1024,  # 2x d_model
+        max_seq_len=512,       # Match data preparation
+        reversible=True,       # Memory efficiency
+        use_checkpoint=True,   # Gradient checkpointing
+        use_autocast=True,     # Mixed precision
+        use_act=True,          # Adaptive Computation Time
+        act_threshold=0.9,
+        lambda_K=0.05,         # Safety telemetry weights
+        lambda_C=0.05,
+        lambda_S=0.05
+    )
+    
+    # Calculate parameter count
+    total_params = sum(p.numel() for p in model.parameters())
+    logger.info(f"Model created: {total_params:,} parameters")
+    logger.info(f"Target: ~16M parameters - {'✓' if 15_000_000 <= total_params <= 17_000_000 else '✗'}")
+    
+    return model
+
+def main():
+    """Main training function."""
+    logger.info("🚀 STARTING BREAKTHROUGH BITRANSFORMERLM TRAINING!")
+    logger.info("Using ACTUAL BitTransformerLM model and train_loop")
+    
+    # Load dataset
+    data = load_and_prepare_dataset()
+    
+    # Create model
+    model = create_breakthrough_model()
+    
+    # CRITICAL: Use Fixed RL Adafactor (the breakthrough secret!)
+    logger.info("Configuring Fixed RL Adafactor optimizer...")
+    optimizer, scheduler = configure_adafactor_optimizer(
+        model,
+        lr=1e-3,  # FIXED learning rate - key to breakthrough!
+        weight_decay=0.01,
+        total_steps=5000  # Estimated total steps
+    )
+    logger.info("Fixed RL Adafactor configured with LR=0.001")
+    
+    # Training configuration
+    training_config = {
+        'epochs': 20,           # Reasonable number of epochs
+        'batch_size': 4,        # Adjust based on memory
+        'accum_steps': 4,       # Gradient accumulation  
+        'amp': True,           # Mixed precision
+        'log': True,           # Enable logging
+        'compress_prob': 0.0,  # Start with no compression
+        'optimizer': optimizer,
+        'scheduler': scheduler
+    }
+    
+    logger.info(f"Training configuration: {training_config}")
+    logger.info("Starting training loop...")
+    
+    # Use the ACTUAL BitTransformerLM train_loop function!
+    metrics = train_loop(
+        model=model,
+        data=data,
+        **training_config
+    )
+    
+    # Save the trained model
+    checkpoint_dir = Path('/data/BitTransformerLM/checkpoints')
+    checkpoint_dir.mkdir(exist_ok=True)
+    
+    model_path = checkpoint_dir / 'breakthrough_model.pt'
+    save_model(model, model_path)
+    logger.info(f"Model saved to: {model_path}")
+    
+    # Log final metrics
+    if metrics:
+        final_metrics = metrics[-1]
+        logger.info("🎉 TRAINING COMPLETED!")
+        logger.info(f"Final raw_loss: {final_metrics['raw_loss']:.6f}")
+        logger.info(f"Final raw_acc: {final_metrics['raw_acc']:.3f}")
+        
+        # Check for breakthrough performance
+        if final_metrics['raw_loss'] < 3.0:
+            logger.info("🚀 BREAKTHROUGH PERFORMANCE ACHIEVED! Loss < 3.0!")
+    
+    logger.info("Breakthrough training completed successfully!")
+
+if __name__ == "__main__":
+    main()

scripts/training/final_breakthrough_training.py

@@ -0,0 +1,426 @@
+#!/usr/bin/env python3
+"""
+Final Breakthrough BitTransformerLM Training Script
+=================================================
+
+The complete training script using the ACTUAL BitTransformerLM model
+with the breakthrough Fixed RL Adafactor configuration and full
+HuggingFace dataset support with checkpoint resumption.
+"""
+
+import sys
+import os
+import json
+import logging
+from pathlib import Path
+from datetime import datetime
+from typing import Optional, Dict, Any
+
+import torch
+import torch.nn.functional as F
+from datasets import load_dataset
+from huggingface_hub import login
+
+# Add paths for imports
+sys.path.append('/data')
+sys.path.append('/data/BitTransformerLM')
+
+from bit_transformer import BitTransformerLM, text_to_bits
+from BTLM_Extensions import configure_adafactor_optimizer
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler('/data/BitTransformerLM/breakthrough_training.log'),
+        logging.StreamHandler()
+    ]
+)
+logger = logging.getLogger(__name__)
+
+class BreakthroughTrainer:
+    """Production-grade BitTransformerLM trainer with breakthrough configuration."""
+    
+    def __init__(self, config: Dict[str, Any]):
+        self.config = config
+        self.device = torch.device('cpu')  # CPU training as per breakthrough
+        self.model = None
+        self.optimizer = None
+        self.scheduler = None
+        self.dataset = None
+        self.checkpoint_dir = Path(config['checkpoint_dir'])
+        self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Training state
+        self.current_epoch = 0
+        self.total_steps = 0
+        self.best_loss = float('inf')
+        self.training_history = []
+
+    def load_and_prepare_dataset(self):
+        """Load HF dataset and convert to proper bit tensors."""
+        logger.info("Loading WCNegentropy/BitTransformerLM dataset...")
+        
+        # Login to HuggingFace
+        login(token=self.config['hf_token'])
+        
+        # Load dataset
+        dataset = load_dataset("WCNegentropy/BitTransformerLM")
+        train_data = dataset['train']
+        
+        logger.info(f"Dataset loaded: {len(train_data)} samples")
+        
+        # Process dataset - convert to bits using the ACTUAL text_to_bits function
+        bit_sequences = []
+        for i, sample in enumerate(train_data):
+            if i % 1000 == 0:
+                logger.info(f"Processing sample {i}/{len(train_data)}")
+            
+            # Try to get text from various fields
+            text = None
+            if 'original_text' in sample and sample['original_text']:
+                text = sample['original_text']
+            elif 'text' in sample and sample['text']:
+                text = sample['text']
+            
+            if text and text.strip():
+                # Use ACTUAL text_to_bits function
+                bits = text_to_bits(text)
+                if len(bits) >= self.config['sequence_length']:
+                    bit_sequences.append(bits)
+        
+        logger.info(f"Processed {len(bit_sequences)} valid bit sequences")
+        
+        # Create training sequences with proper length
+        seq_len = self.config['sequence_length']
+        training_sequences = []
+        
+        for bits in bit_sequences:
+            # Create overlapping chunks
+            for i in range(0, len(bits) - seq_len + 1, seq_len // 2):
+                chunk = bits[i:i + seq_len]
+                if len(chunk) == seq_len:
+                    training_sequences.append(chunk)
+        
+        # Convert to tensor with proper dtype
+        self.dataset = torch.tensor(training_sequences, dtype=torch.long)
+        logger.info(f"Created training dataset: {self.dataset.shape}")
+        
+        return self.dataset
+
+    def create_breakthrough_model(self):
+        """Create the EXACT breakthrough 16M parameter BitTransformerLM."""
+        logger.info("Creating breakthrough 16M parameter BitTransformerLM...")
+        
+        # BREAKTHROUGH CONFIGURATION - exactly as identified before
+        self.model = BitTransformerLM(
+            d_model=512,            # Breakthrough config
+            nhead=16,              # 16 attention heads  
+            num_layers=8,          # 8 layers for ~16M params
+            dim_feedforward=1024,  # 2x d_model
+            max_seq_len=self.config['sequence_length'],
+            lambda_K=0.05,         # Safety telemetry weights
+            lambda_C=0.05,
+            lambda_S=0.05,
+            reversible=True,       # Memory efficiency
+            use_checkpoint=True,   # Gradient checkpointing
+            use_autocast=True,     # CPU mixed precision
+            use_act=True,          # Adaptive Computation Time
+            act_threshold=0.9
+        ).to(self.device)
+        
+        # Calculate and verify parameter count
+        total_params = sum(p.numel() for p in self.model.parameters())
+        trainable_params = sum(p.numel() for p in self.model.parameters() if p.requires_grad)
+        
+        logger.info(f"Model created: {total_params:,} total parameters ({trainable_params:,} trainable)")
+        logger.info(f"Target: ~16M parameters - {'✓' if 15_000_000 <= total_params <= 17_000_000 else '✗'}")
+        
+        return self.model
+
+    def setup_optimizer(self):
+        """Setup Fixed RL Adafactor optimizer (the breakthrough secret sauce)."""
+        logger.info("Setting up Fixed RL Adafactor optimizer...")
+        
+        # Calculate total steps
+        steps_per_epoch = len(self.dataset) // self.config['batch_size']
+        total_steps = steps_per_epoch * self.config['num_epochs']
+        
+        # CRITICAL: Use FIXED LR, not auto-LR (the breakthrough discovery!)
+        self.optimizer, self.scheduler = configure_adafactor_optimizer(
+            self.model,
+            lr=self.config['learning_rate'],  # FIXED LR - key to breakthrough!
+            weight_decay=self.config['weight_decay'],
+            total_steps=total_steps
+        )
+        
+        logger.info(f"Fixed RL Adafactor configured with LR={self.config['learning_rate']}")
+        logger.info(f"Total training steps: {total_steps}")
+        
+        return self.optimizer, self.scheduler
+
+    def save_checkpoint(self, epoch: int, loss: float, is_best: bool = False):
+        """Save complete model checkpoint with all training state."""
+        checkpoint_data = {
+            'epoch': epoch,
+            'total_steps': self.total_steps,
+            'model_state_dict': self.model.state_dict(),
+            'optimizer_state_dict': self.optimizer.state_dict(),
+            'scheduler_state_dict': self.scheduler.state_dict() if self.scheduler else None,
+            'loss': loss,
+            'best_loss': self.best_loss,
+            'config': self.config,
+            'training_history': self.training_history,
+            'timestamp': datetime.now().isoformat(),
+            'model_config': self.model._current_params()  # Save model hyperparams
+        }
+        
+        # Save latest checkpoint
+        latest_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        torch.save(checkpoint_data, latest_path)
+        logger.info(f"Saved checkpoint: {latest_path}")
+        
+        # Save epoch-specific checkpoint
+        epoch_path = self.checkpoint_dir / f'checkpoint_epoch_{epoch:04d}.pt'
+        torch.save(checkpoint_data, epoch_path)
+        
+        # Save best model if this is the best loss
+        if is_best:
+            best_path = self.checkpoint_dir / 'checkpoint_best.pt'
+            torch.save(checkpoint_data, best_path)
+            logger.info(f"🏆 NEW BEST MODEL! Loss: {loss:.6f} -> {best_path}")
+        
+        # Save training config for reference
+        config_path = self.checkpoint_dir / 'training_config.json'
+        with open(config_path, 'w') as f:
+            json.dump(self.config, f, indent=2)
+
+    def load_checkpoint(self, checkpoint_path: Optional[str] = None) -> bool:
+        """Load checkpoint if available and resume training."""
+        if checkpoint_path is None:
+            checkpoint_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        
+        checkpoint_path = Path(checkpoint_path)
+        if not checkpoint_path.exists():
+            logger.info("No checkpoint found - starting fresh training")
+            return False
+        
+        logger.info(f"Loading checkpoint: {checkpoint_path}")
+        try:
+            checkpoint = torch.load(checkpoint_path, map_location=self.device)
+            
+            # Load model state
+            self.model.load_state_dict(checkpoint['model_state_dict'])
+            
+            # Load optimizer state
+            self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
+            
+            # Load scheduler state
+            if self.scheduler and checkpoint.get('scheduler_state_dict'):
+                self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
+            
+            # Load training state
+            self.current_epoch = checkpoint['epoch']
+            self.total_steps = checkpoint['total_steps']
+            self.best_loss = checkpoint['best_loss']
+            self.training_history = checkpoint.get('training_history', [])
+            
+            logger.info(f"✅ Resumed from epoch {self.current_epoch}, best loss: {self.best_loss:.6f}")
+            logger.info(f"Total steps completed: {self.total_steps}")
+            return True
+            
+        except Exception as e:
+            logger.error(f"Failed to load checkpoint: {e}")
+            return False
+
+    def training_step(self, batch: torch.Tensor) -> Dict[str, float]:
+        """Single training step following the ACTUAL model pattern."""
+        batch = batch.to(self.device)
+        
+        # Zero gradients
+        self.optimizer.zero_grad()
+        
+        # Forward pass - EXACTLY like the working basic_training.py
+        logits, telemetry = self.model(batch)
+        
+        # Loss calculation - EXACTLY like example_training_step
+        pred = logits[:, :-1, :].reshape(-1, 2)
+        target = batch[:, 1:].reshape(-1)
+        loss = F.cross_entropy(pred, target)
+        
+        # Backward pass
+        loss.backward()
+        
+        # Gradient clipping
+        torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config['max_grad_norm'])
+        
+        # Optimizer step
+        self.optimizer.step()
+        if self.scheduler:
+            self.scheduler.step()
+        
+        self.total_steps += 1
+        
+        # Extract telemetry values properly
+        metrics = {'loss': loss.item()}
+        if telemetry:
+            for key, value in telemetry.items():
+                if torch.is_tensor(value):
+                    metrics[key] = value.mean().item()
+                else:
+                    metrics[key] = value
+        
+        return metrics
+
+    def train_epoch(self) -> Dict[str, float]:
+        """Train for one complete epoch."""
+        logger.info(f"Starting epoch {self.current_epoch + 1}")
+        
+        # Use EXACT same pattern as working basic_training.py
+        self.model.train()
+        epoch_losses = []
+        
+        # Simple batching - EXACTLY like working basic_training.py
+        batch_size = self.config['batch_size']
+        for i in range(0, len(self.dataset), batch_size):
+            batch = self.dataset[i:i + batch_size]
+            if len(batch) < batch_size:
+                continue  # Skip incomplete batches
+            
+            batch = batch.to(self.device)
+            
+            # Zero gradients
+            self.optimizer.zero_grad()
+            
+            # Forward pass - EXACTLY like working basic_training.py
+            logits, telemetry = self.model(batch)
+            
+            # Loss calculation - EXACTLY like working basic_training.py
+            pred = logits[:, :-1, :].reshape(-1, 2)
+            target = batch[:, 1:].reshape(-1)
+            loss = F.cross_entropy(pred, target)
+            
+            # Backward pass
+            loss.backward()
+            
+            # Gradient clipping
+            torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config['max_grad_norm'])
+            
+            # Optimizer step
+            self.optimizer.step()
+            if self.scheduler:
+                self.scheduler.step()
+            
+            self.total_steps += 1
+            epoch_losses.append(loss.item())
+        
+        # Calculate epoch averages - simplified like basic_training.py
+        avg_loss = sum(epoch_losses) / len(epoch_losses) if epoch_losses else float('inf')
+        
+        epoch_summary = {
+            'epoch': self.current_epoch + 1,
+            'avg_loss': avg_loss
+        }
+        
+        self.training_history.append(epoch_summary)
+        
+        logger.info(
+            f"Epoch {self.current_epoch + 1} completed: "
+            f"Avg Loss={avg_loss:.6f}"
+        )
+        
+        return epoch_summary
+
+    def train(self):
+        """Main training loop."""
+        logger.info("🚀 STARTING BREAKTHROUGH BITRANSFORMERLM TRAINING!")
+        logger.info("Configuration: Fixed RL Adafactor + 16M parameters + CPU training")
+        
+        start_epoch = self.current_epoch
+        
+        for epoch in range(start_epoch, self.config['num_epochs']):
+            try:
+                # Train epoch
+                epoch_metrics = self.train_epoch()
+                avg_loss = epoch_metrics['avg_loss']
+                
+                # Check if this is the best model
+                is_best = avg_loss < self.best_loss
+                if is_best:
+                    self.best_loss = avg_loss
+                
+                # Save checkpoint after each epoch
+                self.save_checkpoint(self.current_epoch + 1, avg_loss, is_best)
+                
+                self.current_epoch += 1
+                
+                # Log progress
+                logger.info(f"=== EPOCH {self.current_epoch} COMPLETE ===")
+                logger.info(f"Loss: {avg_loss:.6f} (best: {self.best_loss:.6f})")
+                
+                # Check for breakthrough performance (loss < 3.0)
+                if avg_loss < 3.0:
+                    logger.info("🚀 BREAKTHROUGH PERFORMANCE ACHIEVED! Loss < 3.0!")
+                
+            except KeyboardInterrupt:
+                logger.info("Training interrupted by user")
+                # Save checkpoint before exiting
+                try:
+                    self.save_checkpoint(self.current_epoch, float('inf'), False)
+                except:
+                    pass
+                break
+            except Exception as e:
+                logger.error(f"Error in epoch {self.current_epoch + 1}: {e}")
+                # Save emergency checkpoint
+                try:
+                    self.save_checkpoint(self.current_epoch, float('inf'), False)
+                except:
+                    pass
+                raise
+
+def main():
+    """Main function to run breakthrough training."""
+    
+    # BREAKTHROUGH TRAINING CONFIGURATION
+    config = {
+        # Model parameters (breakthrough configuration)
+        'sequence_length': 512,
+        
+        # Training parameters
+        'learning_rate': 1e-3,        # FIXED LR - key to breakthrough!
+        'weight_decay': 0.01,
+        'batch_size': 4,              # Adjust based on memory
+        'num_epochs': 50,             # Full training run
+        'max_grad_norm': 1.0,
+        
+        # Data parameters
+        'hf_token': None,  # Set via environment variable HF_TOKEN
+        
+        # Logging and checkpointing
+        'log_interval': 100,
+        'checkpoint_dir': '/data/BitTransformerLM/checkpoints',
+    }
+    
+    # Create trainer
+    trainer = BreakthroughTrainer(config)
+    
+    # Setup all components
+    logger.info("Setting up training components...")
+    trainer.load_and_prepare_dataset()
+    trainer.create_breakthrough_model()
+    trainer.setup_optimizer()
+    
+    # Try to resume from checkpoint
+    trainer.load_checkpoint()
+    
+    # Start training
+    trainer.train()
+    
+    logger.info("🎉 BREAKTHROUGH TRAINING COMPLETED!")
+    logger.info(f"Best loss achieved: {trainer.best_loss:.6f}")
+    logger.info(f"Checkpoints saved to: {trainer.checkpoint_dir}")
+
+if __name__ == "__main__":
+    main()

scripts/training/full_attention_training.py

@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+"""
+BitTransformerLM Full Bi-Directional Attention Training Script
+===============================================================
+
+This script implements the breakthrough Fixed RL Adafactor training configuration
+for production-scale BitTransformerLM training with FULL BI-DIRECTIONAL UNCHUNKED ATTENTION.
+
+Configuration:
+- Model: 16M parameters (d_model=512, nhead=16, num_layers=8)  
+- Attention: FULL BI-DIRECTIONAL UNCHUNKED (chunk_size=None)
+- Optimizer: Fixed LR Adafactor (identical to breakthrough config)
+- Features: Reversible layers, ACT, QAT, compression
+- Data: HuggingFace WCNegentropy/BitTransformerLM dataset
+- Checkpointing: After every training cycle for continuous training
+"""
+
+import sys
+import os
+import json
+import time
+import logging
+from datetime import datetime
+from pathlib import Path
+from typing import Optional, Dict, Any
+
+import torch
+import torch.nn.functional as F
+from datasets import load_dataset
+from huggingface_hub import login
+
+# Add paths for imports
+sys.path.append('/data')
+sys.path.append('/data/BitTransformerLM')
+
+from bit_transformer import (
+    BitTransformerLM,
+    text_to_bits,
+    bits_to_text,
+    save_model,
+    load_model,
+    set_dropout
+)
+from BTLM_Extensions import configure_adafactor_optimizer
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler('full_attention_training.log'),
+        logging.StreamHandler()
+    ]
+)
+logger = logging.getLogger(__name__)
+
+class ProductionTrainer:
+    """Production-grade BitTransformerLM trainer with breakthrough configuration."""
+    
+    def __init__(self, config: Dict[str, Any]):
+        self.config = config
+        self.device = torch.device('cpu')  # CPU training as per breakthrough
+        self.model = None
+        self.optimizer = None
+        self.scheduler = None
+        self.dataset = None
+        self.checkpoint_dir = Path(config['checkpoint_dir'])
+        self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Training state
+        self.current_epoch = 0
+        self.total_steps = 0
+        self.best_loss = float('inf')
+        self.training_history = []
+        
+    def setup_model(self):
+        """Create the breakthrough 16M parameter BitTransformerLM model with full bi-directional attention."""
+        logger.info("Setting up breakthrough BitTransformerLM with FULL BI-DIRECTIONAL UNCHUNKED ATTENTION...")
+        
+        self.model = BitTransformerLM(
+            d_model=512,                    # Breakthrough config
+            nhead=16,                       # 16 attention heads
+            num_layers=8,                   # 8 layers for ~16M params
+            dim_feedforward=1024,           # 2x d_model for optimal params
+            max_seq_len=512,               # Reasonable sequence length
+            reversible=True,                # Memory efficiency
+            use_checkpoint=True,            # Gradient checkpointing
+            use_autocast=True,             # CPU mixed precision
+            use_act=True,                  # Adaptive Computation Time
+            act_threshold=0.9,             # ACT threshold
+            lambda_K=0.05,                 # Safety telemetry weights
+            lambda_C=0.05,
+            lambda_S=0.05,
+            chunk_size=None,               # FULL ATTENTION - no chunking
+            overlap=0,                     # No overlap needed for full attention
+            full_attn_logging=True         # Enable full attention logging
+        ).to(self.device)
+        
+        # Calculate actual parameter count
+        total_params = sum(p.numel() for p in self.model.parameters())
+        trainable_params = sum(p.numel() for p in self.model.parameters() if p.requires_grad)
+        
+        logger.info(f"Model created: {total_params:,} total parameters ({trainable_params:,} trainable)")
+        logger.info(f"Target: ~16M parameters - {'✓' if 15_000_000 <= total_params <= 17_000_000 else '✗'}")
+        
+        return self.model
+    
+    def setup_optimizer(self):
+        """Setup Fixed RL Adafactor optimizer (the breakthrough secret sauce)."""
+        logger.info("Setting up Fixed RL Adafactor optimizer...")
+        
+        # CRITICAL: Use fixed LR, not auto-LR (lr=None)
+        self.optimizer, self.scheduler = configure_adafactor_optimizer(
+            self.model,
+            lr=self.config['learning_rate'],  # Fixed LR - the key to breakthrough!
+            weight_decay=self.config['weight_decay'],
+            total_steps=self.config['total_steps']
+        )
+        
+        logger.info(f"Fixed RL Adafactor configured with LR={self.config['learning_rate']}")
+        return self.optimizer, self.scheduler
+    
+    def setup_dataset(self):
+        """Load and prepare the WCNegentropy/BitTransformerLM dataset."""
+        logger.info("Loading WCNegentropy/BitTransformerLM dataset...")
+        
+        # Login to HuggingFace
+        login(token=self.config['hf_token'])
+        
+        # Load dataset
+        try:
+            dataset = load_dataset("WCNegentropy/BitTransformerLM")
+            logger.info(f"Dataset loaded: {dataset}")
+            
+            # Use train split
+            train_data = dataset['train'] if 'train' in dataset else dataset
+            logger.info(f"Training samples: {len(train_data)}")
+            
+            # Process dataset - convert to bits using the ACTUAL text_to_bits function
+            bit_sequences = []
+            for i, sample in enumerate(train_data):
+                if i % 1000 == 0:
+                    logger.info(f"Processing sample {i}/{len(train_data)}")
+                
+                # Try to get text from various fields
+                text = None
+                if 'original_text' in sample and sample['original_text']:
+                    text = sample['original_text']
+                elif 'text' in sample and sample['text']:
+                    text = sample['text']
+                
+                if text and text.strip():
+                    # Use ACTUAL text_to_bits function
+                    bits = text_to_bits(text)
+                    if len(bits) >= self.config['sequence_length']:
+                        bit_sequences.append(bits)
+            
+            logger.info(f"Processed {len(bit_sequences)} valid bit sequences")
+            
+            # Create training sequences with proper length
+            seq_len = self.config['sequence_length']
+            training_sequences = []
+            
+            for bits in bit_sequences:
+                # Create overlapping chunks
+                for i in range(0, len(bits) - seq_len + 1, seq_len // 2):
+                    chunk = bits[i:i + seq_len]
+                    if len(chunk) == seq_len:
+                        training_sequences.append(chunk)
+            
+            # Convert to tensor with proper dtype
+            self.dataset = torch.tensor(training_sequences, dtype=torch.long)
+            logger.info(f"Created training dataset: {self.dataset.shape}")
+            
+        except Exception as e:
+            logger.error(f"Failed to load dataset: {e}")
+            # Fallback to synthetic data for testing
+            logger.info("Falling back to synthetic bit data...")
+            synthetic_bits = torch.randint(0, 2, (1000, self.config['sequence_length']))
+            self.dataset = synthetic_bits
+            logger.warning("Using synthetic data - replace with real dataset for production")
+        
+        return self.dataset
+    
+    def save_checkpoint(self, epoch: int, loss: float, is_best: bool = False):
+        """Save model checkpoint with all training state."""
+        checkpoint_data = {
+            'epoch': epoch,
+            'total_steps': self.total_steps,
+            'model_state_dict': self.model.state_dict(),
+            'optimizer_state_dict': self.optimizer.state_dict(),
+            'scheduler_state_dict': self.scheduler.state_dict() if self.scheduler else None,
+            'loss': loss,
+            'best_loss': self.best_loss,
+            'config': self.config,
+            'training_history': self.training_history,
+            'timestamp': datetime.now().isoformat()
+        }
+        
+        # Save latest checkpoint
+        latest_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        torch.save(checkpoint_data, latest_path)
+        logger.info(f"Saved checkpoint: {latest_path}")
+        
+        # Save epoch-specific checkpoint
+        epoch_path = self.checkpoint_dir / f'checkpoint_epoch_{epoch:04d}.pt'
+        torch.save(checkpoint_data, epoch_path)
+        
+        # Save best model if this is the best loss
+        if is_best:
+            best_path = self.checkpoint_dir / 'checkpoint_best.pt'
+            torch.save(checkpoint_data, best_path)
+            logger.info(f"NEW BEST MODEL! Loss: {loss:.6f} -> {best_path}")
+        
+        # Save training config for reference
+        config_path = self.checkpoint_dir / 'training_config.json'
+        with open(config_path, 'w') as f:
+            json.dump(self.config, f, indent=2)
+    
+    def load_checkpoint(self, checkpoint_path: Optional[str] = None) -> bool:
+        """Load model weights from latest checkpoint but restart training from epoch 1."""
+        if checkpoint_path is None:
+            checkpoint_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        
+        checkpoint_path = Path(checkpoint_path)
+        if not checkpoint_path.exists():
+            logger.info("No checkpoint found - starting fresh training")
+            return False
+        
+        logger.info(f"Loading model weights from: {checkpoint_path}")
+        try:
+            checkpoint = torch.load(checkpoint_path, map_location=self.device)
+            
+            # Load ONLY model weights
+            self.model.load_state_dict(checkpoint['model_state_dict'])
+            
+            # RESET all training state to start from epoch 1
+            self.current_epoch = 1
+            self.total_steps = 0
+            self.best_loss = float('inf')
+            self.training_history = []
+            
+            # DO NOT load optimizer/scheduler state - fresh start
+            
+            logger.info(f"Loaded model weights, restarting training from epoch 1, step 0")
+            return True
+            
+        except Exception as e:
+            logger.error(f"Failed to load checkpoint: {e}")
+            return False
+    
+    def training_step(self, batch: torch.Tensor) -> Dict[str, float]:
+        """Single training step with telemetry."""
+        self.model.train()
+        set_dropout(self.model, self.config['dropout'])
+        
+        batch = batch.to(self.device)
+        
+        # Zero gradients
+        self.optimizer.zero_grad()
+        
+        # Forward pass with telemetry
+        with torch.autocast(device_type='cpu', dtype=torch.bfloat16):
+            logits, telemetry = self.model(batch)
+            
+            # Compute loss (next bit prediction)
+            if logits.dim() == 3:  # (batch, seq, 2)
+                targets = batch[:, 1:]  # Next bit prediction
+                logits = logits[:, :-1]  # Remove last prediction
+                loss = F.cross_entropy(logits.reshape(-1, 2), targets.reshape(-1))
+            else:
+                loss = F.cross_entropy(logits, batch)
+            
+            # Add telemetry regularization (safety metrics)
+            if self.model.lambda_K > 0 and 'negentropy_logits' in telemetry:
+                k_term = self.model.lambda_K * (1 - telemetry['negentropy_logits'])
+                if k_term.dim() == 0:  # scalar
+                    loss = loss + k_term
+                else:
+                    loss = loss + k_term.mean()
+            if self.model.lambda_C > 0 and 'lz_complexity_logits' in telemetry:
+                c_term = self.model.lambda_C * (1 - telemetry['lz_complexity_logits'])
+                if c_term.dim() == 0:  # scalar
+                    loss = loss + c_term
+                else:
+                    loss = loss + c_term.mean()
+            if self.model.lambda_S > 0 and 'symbiosis_score' in telemetry:
+                s_term = self.model.lambda_S * (1 - telemetry['symbiosis_score'])
+                if s_term.dim() == 0:  # scalar
+                    loss = loss + s_term
+                else:
+                    loss = loss + s_term.mean()
+        
+        # Backward pass
+        loss.backward()
+        
+        # Gradient clipping
+        torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config['max_grad_norm'])
+        
+        # Optimizer step
+        self.optimizer.step()
+        if self.scheduler:
+            self.scheduler.step()
+        
+        self.total_steps += 1
+        
+        return {
+            'loss': loss.item(),
+            'K': telemetry.get('negentropy_logits', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('negentropy_logits', 0.0)) else telemetry.get('negentropy_logits', 0.0),
+            'C': telemetry.get('lz_complexity_logits', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('lz_complexity_logits', 0.0)) else telemetry.get('lz_complexity_logits', 0.0),
+            'S': telemetry.get('symbiosis_score', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('symbiosis_score', 0.0)) else telemetry.get('symbiosis_score', 0.0),
+            'lr': self.optimizer.param_groups[0]['lr']
+        }
+    
+    def train_epoch(self) -> Dict[str, float]:
+        """Train for one epoch."""
+        logger.info(f"Starting epoch {self.current_epoch + 1}")
+        
+        # Create data loader
+        from torch.utils.data import DataLoader
+        dataloader = DataLoader(
+            self.dataset, 
+            batch_size=self.config['batch_size'], 
+            shuffle=True,
+            drop_last=True
+        )
+        
+        epoch_losses = []
+        epoch_metrics = {'K': [], 'C': [], 'S': []}
+        
+        start_time = time.time()
+        
+        for step, batch in enumerate(dataloader):
+            metrics = self.training_step(batch)
+            
+            epoch_losses.append(metrics['loss'])
+            epoch_metrics['K'].append(metrics['K'])
+            epoch_metrics['C'].append(metrics['C'])
+            epoch_metrics['S'].append(metrics['S'])
+            
+            # Log progress
+            if step % self.config['log_interval'] == 0:
+                logger.info(
+                    f"Epoch {self.current_epoch + 1}, Step {step}/{len(dataloader)}: "
+                    f"Loss={metrics['loss']:.6f}, K={metrics['K']:.3f}, "
+                    f"C={metrics['C']:.3f}, S={metrics['S']:.3f}, LR={metrics['lr']:.2e}"
+                )
+        
+        # Calculate epoch metrics
+        epoch_time = time.time() - start_time
+        avg_loss = sum(epoch_losses) / len(epoch_losses)
+        avg_metrics = {k: sum(v) / len(v) for k, v in epoch_metrics.items()}
+        
+        epoch_summary = {
+            'epoch': self.current_epoch + 1,
+            'avg_loss': avg_loss,
+            'time': epoch_time,
+            **avg_metrics
+        }
+        
+        self.training_history.append(epoch_summary)
+        
+        logger.info(
+            f"Epoch {self.current_epoch + 1} completed in {epoch_time:.1f}s: "
+            f"Avg Loss={avg_loss:.6f}, K={avg_metrics['K']:.3f}, "
+            f"C={avg_metrics['C']:.3f}, S={avg_metrics['S']:.3f}"
+        )
+        
+        return epoch_summary
+    
+    def train(self, num_epochs: int):
+        """Main training loop."""
+        logger.info(f"Starting production training for {num_epochs} epochs...")
+        logger.info(f"Breakthrough configuration: Fixed RL Adafactor + 16M BitTransformerLM")
+        
+        for epoch in range(num_epochs):
+            try:
+                # Train epoch
+                epoch_metrics = self.train_epoch()
+                avg_loss = epoch_metrics['avg_loss']
+                
+                # Check if this is the best model
+                is_best = avg_loss < self.best_loss
+                if is_best:
+                    self.best_loss = avg_loss
+                
+                # Save checkpoint after each epoch
+                self.save_checkpoint(self.current_epoch + 1, avg_loss, is_best)
+                
+                self.current_epoch += 1
+                
+                # Log progress
+                logger.info(f"=== EPOCH {self.current_epoch} COMPLETE ===")
+                logger.info(f"Loss: {avg_loss:.6f} (best: {self.best_loss:.6f})")
+                
+                # Check for breakthrough performance (loss < 3.0)
+                if avg_loss < 3.0:
+                    logger.info("🚀 BREAKTHROUGH PERFORMANCE ACHIEVED! Loss < 3.0!")
+                
+            except KeyboardInterrupt:
+                logger.info("Training interrupted by user")
+                break
+            except Exception as e:
+                logger.error(f"Error in epoch {self.current_epoch + 1}: {e}")
+                # Save emergency checkpoint
+                try:
+                    self.save_checkpoint(self.current_epoch, float('inf'), False)
+                except:
+                    pass
+                raise
+
+
+def main():
+    """Main function to run production training."""
+    
+    # Production training configuration
+    config = {
+        # Model parameters (breakthrough configuration)
+        'model_params': {
+            'd_model': 512,
+            'nhead': 16,
+            'num_layers': 8,
+            'dim_feedforward': 1024,
+        },
+        
+        # Training parameters
+        'learning_rate': 1e-3,        # FIXED LR - key to breakthrough!
+        'weight_decay': 0.01,
+        'batch_size': 4,              # Adjust based on memory
+        'sequence_length': 256,       # Bit sequence length
+        'num_epochs': 50,             # Long training run
+        'max_grad_norm': 1.0,
+        'dropout': 0.1,
+        'total_steps': 10000,         # For scheduler
+        
+        # Data parameters
+        'hf_token': None,  # Set via environment variable HF_TOKEN
+        
+        # Logging and checkpointing
+        'log_interval': 10,
+        'checkpoint_dir': '/data/BitTransformerLM/checkpoints',
+    }
+    
+    # Create trainer
+    trainer = ProductionTrainer(config)
+    
+    # Setup components
+    trainer.setup_model()
+    trainer.setup_optimizer()  
+    trainer.setup_dataset()
+    
+    # Try to resume from checkpoint
+    trainer.load_checkpoint()
+    
+    # Start training
+    logger.info("🚀 STARTING BREAKTHROUGH BITRANSFORMERLM TRAINING!")
+    logger.info("Configuration: Fixed RL Adafactor + 16M parameters + CPU training")
+    
+    trainer.train(config['num_epochs'])
+    
+    logger.info("Training completed!")
+    logger.info(f"Best loss achieved: {trainer.best_loss:.6f}")
+    logger.info(f"Checkpoints saved to: {trainer.checkpoint_dir}")
+
+
+if __name__ == "__main__":
+    main()

scripts/training/production_training.py

@@ -0,0 +1,462 @@
+#!/usr/bin/env python3
+"""
+BitTransformerLM Production Training Script
+==========================================
+
+This script implements the breakthrough Fixed RL Adafactor training configuration
+for production-scale BitTransformerLM training with continuous checkpointing.
+
+Configuration:
+- Model: 16M parameters (d_model=512, nhead=16, num_layers=8)
+- Optimizer: Fixed LR Adafactor (not auto-LR)
+- Features: Reversible layers, ACT, QAT, compression
+- Data: HuggingFace WCNegentropy/BitTransformerLM dataset
+- Checkpointing: After every training cycle for continuous training
+"""
+
+import sys
+import os
+import json
+import time
+import logging
+from datetime import datetime
+from pathlib import Path
+from typing import Optional, Dict, Any
+
+import torch
+import torch.nn.functional as F
+from datasets import load_dataset
+from huggingface_hub import login
+
+# Add paths for imports
+sys.path.append('/data')
+sys.path.append('/data/BitTransformerLM')
+
+from bit_transformer import (
+    BitTransformerLM,
+    text_to_bits,
+    bits_to_text,
+    save_model,
+    load_model,
+    set_dropout
+)
+from BTLM_Extensions import configure_adafactor_optimizer
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler('production_training.log'),
+        logging.StreamHandler()
+    ]
+)
+logger = logging.getLogger(__name__)
+
+class ProductionTrainer:
+    """Production-grade BitTransformerLM trainer with breakthrough configuration."""
+    
+    def __init__(self, config: Dict[str, Any]):
+        self.config = config
+        self.device = torch.device('cpu')  # CPU training as per breakthrough
+        self.model = None
+        self.optimizer = None
+        self.scheduler = None
+        self.dataset = None
+        self.checkpoint_dir = Path(config['checkpoint_dir'])
+        self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Training state
+        self.current_epoch = 0
+        self.total_steps = 0
+        self.best_loss = float('inf')
+        self.training_history = []
+        
+    def setup_model(self):
+        """Create the breakthrough 16M parameter BitTransformerLM model."""
+        logger.info("Setting up breakthrough BitTransformerLM configuration...")
+        
+        self.model = BitTransformerLM(
+            d_model=512,                    # Breakthrough config
+            nhead=16,                       # 16 attention heads
+            num_layers=8,                   # 8 layers for ~16M params
+            dim_feedforward=1024,           # 2x d_model for optimal params
+            max_seq_len=512,               # Reasonable sequence length
+            reversible=True,                # Memory efficiency
+            use_checkpoint=True,            # Gradient checkpointing
+            use_autocast=True,             # CPU mixed precision
+            use_act=True,                  # Adaptive Computation Time
+            act_threshold=0.9,             # ACT threshold
+            lambda_K=0.05,                 # Safety telemetry weights
+            lambda_C=0.05,
+            lambda_S=0.05
+        ).to(self.device)
+        
+        # Calculate actual parameter count
+        total_params = sum(p.numel() for p in self.model.parameters())
+        trainable_params = sum(p.numel() for p in self.model.parameters() if p.requires_grad)
+        
+        logger.info(f"Model created: {total_params:,} total parameters ({trainable_params:,} trainable)")
+        logger.info(f"Target: ~16M parameters - {'✓' if 15_000_000 <= total_params <= 17_000_000 else '✗'}")
+        
+        return self.model
+    
+    def setup_optimizer(self):
+        """Setup Fixed RL Adafactor optimizer (the breakthrough secret sauce)."""
+        logger.info("Setting up Fixed RL Adafactor optimizer...")
+        
+        # CRITICAL: Use fixed LR, not auto-LR (lr=None)
+        self.optimizer, self.scheduler = configure_adafactor_optimizer(
+            self.model,
+            lr=self.config['learning_rate'],  # Fixed LR - the key to breakthrough!
+            weight_decay=self.config['weight_decay'],
+            total_steps=self.config['total_steps']
+        )
+        
+        logger.info(f"Fixed RL Adafactor configured with LR={self.config['learning_rate']}")
+        return self.optimizer, self.scheduler
+    
+    def setup_dataset(self):
+        """Load and prepare the WCNegentropy/BitTransformerLM dataset."""
+        logger.info("Loading WCNegentropy/BitTransformerLM dataset...")
+        
+        # Login to HuggingFace
+        login(token=self.config['hf_token'])
+        
+        # Load dataset
+        try:
+            dataset = load_dataset("WCNegentropy/BitTransformerLM")
+            logger.info(f"Dataset loaded: {dataset}")
+            
+            # Use train split
+            train_data = dataset['train'] if 'train' in dataset else dataset
+            logger.info(f"Training samples: {len(train_data)}")
+            
+            # Process dataset - convert to bits using the ACTUAL text_to_bits function
+            bit_sequences = []
+            for i, sample in enumerate(train_data):
+                if i % 1000 == 0:
+                    logger.info(f"Processing sample {i}/{len(train_data)}")
+                
+                # Try to get text from various fields
+                text = None
+                if 'original_text' in sample and sample['original_text']:
+                    text = sample['original_text']
+                elif 'text' in sample and sample['text']:
+                    text = sample['text']
+                
+                if text and text.strip():
+                    # Use ACTUAL text_to_bits function
+                    bits = text_to_bits(text)
+                    if len(bits) >= self.config['sequence_length']:
+                        bit_sequences.append(bits)
+            
+            logger.info(f"Processed {len(bit_sequences)} valid bit sequences")
+            
+            # Create training sequences with proper length
+            seq_len = self.config['sequence_length']
+            training_sequences = []
+            
+            for bits in bit_sequences:
+                # Create overlapping chunks
+                for i in range(0, len(bits) - seq_len + 1, seq_len // 2):
+                    chunk = bits[i:i + seq_len]
+                    if len(chunk) == seq_len:
+                        training_sequences.append(chunk)
+            
+            # Convert to tensor with proper dtype
+            self.dataset = torch.tensor(training_sequences, dtype=torch.long)
+            logger.info(f"Created training dataset: {self.dataset.shape}")
+            
+        except Exception as e:
+            logger.error(f"Failed to load dataset: {e}")
+            # Fallback to synthetic data for testing
+            logger.info("Falling back to synthetic bit data...")
+            synthetic_bits = torch.randint(0, 2, (1000, self.config['sequence_length']))
+            self.dataset = synthetic_bits
+            logger.warning("Using synthetic data - replace with real dataset for production")
+        
+        return self.dataset
+    
+    def save_checkpoint(self, epoch: int, loss: float, is_best: bool = False):
+        """Save model checkpoint with all training state."""
+        checkpoint_data = {
+            'epoch': epoch,
+            'total_steps': self.total_steps,
+            'model_state_dict': self.model.state_dict(),
+            'optimizer_state_dict': self.optimizer.state_dict(),
+            'scheduler_state_dict': self.scheduler.state_dict() if self.scheduler else None,
+            'loss': loss,
+            'best_loss': self.best_loss,
+            'config': self.config,
+            'training_history': self.training_history,
+            'timestamp': datetime.now().isoformat()
+        }
+        
+        # Save latest checkpoint
+        latest_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        torch.save(checkpoint_data, latest_path)
+        logger.info(f"Saved checkpoint: {latest_path}")
+        
+        # Save epoch-specific checkpoint
+        epoch_path = self.checkpoint_dir / f'checkpoint_epoch_{epoch:04d}.pt'
+        torch.save(checkpoint_data, epoch_path)
+        
+        # Save best model if this is the best loss
+        if is_best:
+            best_path = self.checkpoint_dir / 'checkpoint_best.pt'
+            torch.save(checkpoint_data, best_path)
+            logger.info(f"NEW BEST MODEL! Loss: {loss:.6f} -> {best_path}")
+        
+        # Save training config for reference
+        config_path = self.checkpoint_dir / 'training_config.json'
+        with open(config_path, 'w') as f:
+            json.dump(self.config, f, indent=2)
+    
+    def load_checkpoint(self, checkpoint_path: Optional[str] = None) -> bool:
+        """Load checkpoint if available."""
+        if checkpoint_path is None:
+            checkpoint_path = self.checkpoint_dir / 'checkpoint_latest.pt'
+        
+        checkpoint_path = Path(checkpoint_path)
+        if not checkpoint_path.exists():
+            logger.info("No checkpoint found - starting fresh training")
+            return False
+        
+        logger.info(f"Loading checkpoint: {checkpoint_path}")
+        try:
+            checkpoint = torch.load(checkpoint_path, map_location=self.device)
+            
+            self.model.load_state_dict(checkpoint['model_state_dict'])
+            self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
+            if self.scheduler and checkpoint['scheduler_state_dict']:
+                self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
+            
+            self.current_epoch = checkpoint['epoch']
+            self.total_steps = checkpoint['total_steps']
+            self.best_loss = checkpoint['best_loss']
+            self.training_history = checkpoint.get('training_history', [])
+            
+            logger.info(f"Resumed from epoch {self.current_epoch}, best loss: {self.best_loss:.6f}")
+            return True
+            
+        except Exception as e:
+            logger.error(f"Failed to load checkpoint: {e}")
+            return False
+    
+    def training_step(self, batch: torch.Tensor) -> Dict[str, float]:
+        """Single training step with telemetry."""
+        self.model.train()
+        set_dropout(self.model, self.config['dropout'])
+        
+        batch = batch.to(self.device)
+        
+        # Zero gradients
+        self.optimizer.zero_grad()
+        
+        # Forward pass with telemetry
+        with torch.autocast(device_type='cpu', dtype=torch.bfloat16):
+            logits, telemetry = self.model(batch)
+            
+            # Compute loss (next bit prediction)
+            if logits.dim() == 3:  # (batch, seq, 2)
+                targets = batch[:, 1:]  # Next bit prediction
+                logits = logits[:, :-1]  # Remove last prediction
+                loss = F.cross_entropy(logits.reshape(-1, 2), targets.reshape(-1))
+            else:
+                loss = F.cross_entropy(logits, batch)
+            
+            # Add telemetry regularization (safety metrics)
+            if self.model.lambda_K > 0 and 'negentropy_logits' in telemetry:
+                k_term = self.model.lambda_K * (1 - telemetry['negentropy_logits'])
+                if k_term.dim() == 0:  # scalar
+                    loss = loss + k_term
+                else:
+                    loss = loss + k_term.mean()
+            if self.model.lambda_C > 0 and 'lz_complexity_logits' in telemetry:
+                c_term = self.model.lambda_C * (1 - telemetry['lz_complexity_logits'])
+                if c_term.dim() == 0:  # scalar
+                    loss = loss + c_term
+                else:
+                    loss = loss + c_term.mean()
+            if self.model.lambda_S > 0 and 'symbiosis_score' in telemetry:
+                s_term = self.model.lambda_S * (1 - telemetry['symbiosis_score'])
+                if s_term.dim() == 0:  # scalar
+                    loss = loss + s_term
+                else:
+                    loss = loss + s_term.mean()
+        
+        # Backward pass
+        loss.backward()
+        
+        # Gradient clipping
+        torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config['max_grad_norm'])
+        
+        # Optimizer step
+        self.optimizer.step()
+        if self.scheduler:
+            self.scheduler.step()
+        
+        self.total_steps += 1
+        
+        return {
+            'loss': loss.item(),
+            'K': telemetry.get('negentropy_logits', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('negentropy_logits', 0.0)) else telemetry.get('negentropy_logits', 0.0),
+            'C': telemetry.get('lz_complexity_logits', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('lz_complexity_logits', 0.0)) else telemetry.get('lz_complexity_logits', 0.0),
+            'S': telemetry.get('symbiosis_score', torch.tensor(0.0)).mean().item() if torch.is_tensor(telemetry.get('symbiosis_score', 0.0)) else telemetry.get('symbiosis_score', 0.0),
+            'lr': self.optimizer.param_groups[0]['lr']
+        }
+    
+    def train_epoch(self) -> Dict[str, float]:
+        """Train for one epoch."""
+        logger.info(f"Starting epoch {self.current_epoch + 1}")
+        
+        # Create data loader
+        from torch.utils.data import DataLoader
+        dataloader = DataLoader(
+            self.dataset, 
+            batch_size=self.config['batch_size'], 
+            shuffle=True,
+            drop_last=True
+        )
+        
+        epoch_losses = []
+        epoch_metrics = {'K': [], 'C': [], 'S': []}
+        
+        start_time = time.time()
+        
+        for step, batch in enumerate(dataloader):
+            metrics = self.training_step(batch)
+            
+            epoch_losses.append(metrics['loss'])
+            epoch_metrics['K'].append(metrics['K'])
+            epoch_metrics['C'].append(metrics['C'])
+            epoch_metrics['S'].append(metrics['S'])
+            
+            # Log progress
+            if step % self.config['log_interval'] == 0:
+                logger.info(
+                    f"Epoch {self.current_epoch + 1}, Step {step}/{len(dataloader)}: "
+                    f"Loss={metrics['loss']:.6f}, K={metrics['K']:.3f}, "
+                    f"C={metrics['C']:.3f}, S={metrics['S']:.3f}, LR={metrics['lr']:.2e}"
+                )
+        
+        # Calculate epoch metrics
+        epoch_time = time.time() - start_time
+        avg_loss = sum(epoch_losses) / len(epoch_losses)
+        avg_metrics = {k: sum(v) / len(v) for k, v in epoch_metrics.items()}
+        
+        epoch_summary = {
+            'epoch': self.current_epoch + 1,
+            'avg_loss': avg_loss,
+            'time': epoch_time,
+            **avg_metrics
+        }
+        
+        self.training_history.append(epoch_summary)
+        
+        logger.info(
+            f"Epoch {self.current_epoch + 1} completed in {epoch_time:.1f}s: "
+            f"Avg Loss={avg_loss:.6f}, K={avg_metrics['K']:.3f}, "
+            f"C={avg_metrics['C']:.3f}, S={avg_metrics['S']:.3f}"
+        )
+        
+        return epoch_summary
+    
+    def train(self, num_epochs: int):
+        """Main training loop."""
+        logger.info(f"Starting production training for {num_epochs} epochs...")
+        logger.info(f"Breakthrough configuration: Fixed RL Adafactor + 16M BitTransformerLM")
+        
+        for epoch in range(num_epochs):
+            try:
+                # Train epoch
+                epoch_metrics = self.train_epoch()
+                avg_loss = epoch_metrics['avg_loss']
+                
+                # Check if this is the best model
+                is_best = avg_loss < self.best_loss
+                if is_best:
+                    self.best_loss = avg_loss
+                
+                # Save checkpoint after each epoch
+                self.save_checkpoint(self.current_epoch + 1, avg_loss, is_best)
+                
+                self.current_epoch += 1
+                
+                # Log progress
+                logger.info(f"=== EPOCH {self.current_epoch} COMPLETE ===")
+                logger.info(f"Loss: {avg_loss:.6f} (best: {self.best_loss:.6f})")
+                
+                # Check for breakthrough performance (loss < 3.0)
+                if avg_loss < 3.0:
+                    logger.info("🚀 BREAKTHROUGH PERFORMANCE ACHIEVED! Loss < 3.0!")
+                
+            except KeyboardInterrupt:
+                logger.info("Training interrupted by user")
+                break
+            except Exception as e:
+                logger.error(f"Error in epoch {self.current_epoch + 1}: {e}")
+                # Save emergency checkpoint
+                try:
+                    self.save_checkpoint(self.current_epoch, float('inf'), False)
+                except:
+                    pass
+                raise
+
+
+def main():
+    """Main function to run production training."""
+    
+    # Production training configuration
+    config = {
+        # Model parameters (breakthrough configuration)
+        'model_params': {
+            'd_model': 512,
+            'nhead': 16,
+            'num_layers': 8,
+            'dim_feedforward': 1024,
+        },
+        
+        # Training parameters
+        'learning_rate': 1e-3,        # FIXED LR - key to breakthrough!
+        'weight_decay': 0.01,
+        'batch_size': 4,              # Adjust based on memory
+        'sequence_length': 256,       # Bit sequence length
+        'num_epochs': 50,             # Long training run
+        'max_grad_norm': 1.0,
+        'dropout': 0.1,
+        'total_steps': 10000,         # For scheduler
+        
+        # Data parameters
+        'hf_token': None,  # Set via environment variable HF_TOKEN
+        
+        # Logging and checkpointing
+        'log_interval': 10,
+        'checkpoint_dir': '/data/BitTransformerLM/checkpoints',
+    }
+    
+    # Create trainer
+    trainer = ProductionTrainer(config)
+    
+    # Setup components
+    trainer.setup_model()
+    trainer.setup_optimizer()  
+    trainer.setup_dataset()
+    
+    # Try to resume from checkpoint
+    trainer.load_checkpoint()
+    
+    # Start training
+    logger.info("🚀 STARTING BREAKTHROUGH BITRANSFORMERLM TRAINING!")
+    logger.info("Configuration: Fixed RL Adafactor + 16M parameters + CPU training")
+    
+    trainer.train(config['num_epochs'])
+    
+    logger.info("Training completed!")
+    logger.info(f"Best loss achieved: {trainer.best_loss:.6f}")
+    logger.info(f"Checkpoints saved to: {trainer.checkpoint_dir}")
+
+
+if __name__ == "__main__":
+    main()

sync_to_hf.py

@@ -1,220 +0,0 @@
-#!/usr/bin/env python3
-"""
-Sync BitTransformerLM repository to HuggingFace Hub for OS launch.
-Uploads all cleaned documentation and code with proper commit message.
-"""
-
-import os
-import logging
-from pathlib import Path
-from huggingface_hub import HfApi, login
-from typing import Optional, List
-
-# Setup logging
-logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
-logger = logging.getLogger(__name__)
-
-def sync_repository_to_hf(
-    repo_id: str = "WCNegentropy/BitTransformerLM",
-    token: Optional[str] = None,
-    commit_message: str = "🚀 OS Launch: Clean documentation and refined licensing"
-):
-    """
-    Sync the entire cleaned BitTransformerLM repository to HuggingFace Hub.
-    
-    Args:
-        repo_id: HuggingFace repository ID
-        token: HF token (defaults to HF_TOKEN environment variable)
-        commit_message: Commit message for the upload
-    """
-    
-    # Get token from environment if not provided
-    if token is None:
-        token = os.environ.get('HF_TOKEN')
-        if not token:
-            logger.error("HF_TOKEN environment variable not set and no token provided")
-            return False
-    
-    try:
-        # Login to HuggingFace
-        login(token=token)
-        api = HfApi()
-        logger.info("Successfully authenticated with HuggingFace Hub")
-        
-        # Get the repository root directory
-        repo_root = Path(__file__).parent
-        logger.info(f"Repository root: {repo_root}")
-        
-        # Files and directories to upload (excluding unnecessary files)
-        include_patterns = [
-            # Core code
-            "bit_transformer/**/*.py",
-            "tests/**/*.py",
-            "*.py",  # Root level Python files
-            
-            # Documentation (cleaned)
-            "README.md",
-            "MODEL_CARD.md", 
-            "RESEARCH_STATUS.md",
-            "EMPIRICAL_VALIDATION.md",
-            "OPEN_SOURCE_LAUNCH.md",
-            "AGENTS.md",
-            
-            # Configuration
-            "requirements.txt",
-            "pyproject.toml",
-            "Dockerfile",
-            "start.sh",
-            
-            # License files (cleaned)
-            "LICENSE/**/*.txt",
-        ]
-        
-        # Files to exclude
-        exclude_patterns = [
-            "__pycache__/**",
-            "*.pyc",
-            ".git/**",
-            ".pytest_cache/**",
-            "weights/**",
-            "checkpoints/**",
-            "*.log",
-            # Outdated documentation  
-            "BitTransformerLM_full_assessment.md",
-            "FORENSIC_*.md",
-            "state_of_the_repo_audit.md",
-            # Old upload script
-            "upload_to_hf.py",
-        ]
-        
-        # Get all files to upload
-        files_to_upload = []
-        for pattern in include_patterns:
-            for file_path in repo_root.glob(pattern):
-                if file_path.is_file():
-                    # Check if file should be excluded
-                    relative_path = file_path.relative_to(repo_root)
-                    should_exclude = any(
-                        relative_path.match(exclude) 
-                        for exclude in exclude_patterns
-                    )
-                    if not should_exclude:
-                        files_to_upload.append(file_path)
-        
-        logger.info(f"Found {len(files_to_upload)} files to upload")
-        
-        # Upload files in batches
-        uploaded_count = 0
-        for file_path in files_to_upload:
-            try:
-                relative_path = file_path.relative_to(repo_root)
-                logger.info(f"Uploading: {relative_path}")
-                
-                api.upload_file(
-                    path_or_fileobj=str(file_path),
-                    path_in_repo=str(relative_path),
-                    repo_id=repo_id,
-                    repo_type="model",
-                    commit_message=commit_message,
-                    commit_description="""
-This OS launch commit includes:
-
-✅ **Cleaned Documentation**
-- Removed inflated claims and marketing language
-- Added honest research status and limitations
-- Created professional model card and validation reports
-- Streamlined licensing to AGPLv3 + commercial contact
-
-✅ **Refined Codebase** 
-- Complete experimental bit-native transformer implementation
-- 57 Python files with comprehensive research framework
-- Safety telemetry and monitoring systems
-- Distributed training and development tools
-
-✅ **Professional Standards**
-- Empirical validation of all claims
-- Clear experimental vs production distinctions  
-- Rigorous research methodology requirements
-- Community contribution framework
-
-Ready for serious research evaluation and academic investigation.
-                    """.strip()
-                )
-                
-                uploaded_count += 1
-                if uploaded_count % 10 == 0:
-                    logger.info(f"Progress: {uploaded_count}/{len(files_to_upload)} files uploaded")
-                    
-            except Exception as e:
-                logger.warning(f"Failed to upload {relative_path}: {e}")
-                continue
-        
-        logger.info(f"✅ Successfully uploaded {uploaded_count}/{len(files_to_upload)} files")
-        logger.info(f"🎉 Repository synced to: https://huggingface.co/{repo_id}")
-        
-        return True
-        
-    except Exception as e:
-        logger.error(f"❌ Failed to sync repository: {e}")
-        return False
-
-def create_release_info():
-    """Create a release information file for the OS launch."""
-    release_info = """# BitTransformerLM v0.1.0 - Experimental Research Release
-
-**Release Date:** August 2025  
-**Status:** Open Source Research Implementation  
-**License:** AGPLv3 + Commercial Licensing Available  
-
-## What's Included
-
-This release provides a complete experimental framework for bit-native language modeling research:
-
-- **Core Architecture:** 57 Python files implementing bit-native transformer with reversible layers
-- **Safety Systems:** Real-time K/C/S telemetry and monitoring
-- **Research Tools:** Interactive dashboard, distributed training, comprehensive testing
-- **Documentation:** Professional model card, research status, and validation reports
-
-## Important Notes
-
-⚠️ **Experimental Status:** This is research code requiring rigorous baseline validation  
-⚠️ **Not Production Ready:** Needs extensive evaluation vs standard transformers  
-⚠️ **Research Use Only:** Intended for academic investigation and experimentation  
-
-## Licensing
-
-- **Open Source:** AGPLv3 for research and open source use
-- **Commercial:** Contact contact@wcnegentropy.com for commercial licensing
-
-## Next Steps
-
-The research community is invited to:
-1. Conduct rigorous baseline comparisons vs standard transformers
-2. Evaluate on established language modeling benchmarks  
-3. Validate (or refute) claimed memory efficiency benefits
-4. Share findings openly to advance the field
-
-**Research responsibly. Validate rigorously. Share openly.**
-"""
-    
-    release_file = Path(__file__).parent / "RELEASE_INFO.md"
-    with open(release_file, 'w') as f:
-        f.write(release_info)
-    
-    logger.info("Created RELEASE_INFO.md")
-    return release_file
-
-if __name__ == "__main__":
-    # Create release info file
-    create_release_info()
-    
-    # Sync to HuggingFace
-    success = sync_repository_to_hf()
-    
-    if success:
-        print("\n🚀 BitTransformerLM OS Launch Sync Complete!")
-        print("📍 Repository: https://huggingface.co/WCNegentropy/BitTransformerLM")
-        print("📧 Commercial inquiries: contact@wcnegentropy.com")
-        print("\nReady for research community evaluation! 🧪✨")
-    else:
-        print("\n❌ Sync failed. Please check logs and try again.")

tests/TEST_RESULTS.md

@@ -1,578 +0,0 @@
-# Test Results
-
-## Automated Tests
-- `pytest -q`: all tests passed.
-
-```
-....                                                                     [100%]
-4 passed in 5.28s
-```
-
-## Example Script
-- `python example.py` executed successfully:
-
-```
-Training loss: 0.8508605360984802
-Available telemetry: ['activations', 'attention_maps', 'entropy', 'negentropy', 'lz_complexity', 'symbiosis_score']
-```
-
-## Progressive Scale-Up
-- `python progressive_scaleup.py` (default steps=2) produced:
-
-```
-Step 0 validation loss: 0.7001
-Step 1 validation loss: 0.6954
-```
-
-## Text Inference
-- Running `infer_text` on a short string returned the input text without errors:
-
-```
-hi
-```
-
-## Extended Scaling Test
-Installed torch and ran `python progressive_scaleup.py --steps 4`:
-
-```
-Step 0 validation loss: 0.6970
-Step 1 validation loss: 0.6915
-Step 2 validation loss: 0.7022
-Step 3 validation loss: 0.7123
-```
-
-## Collapse Test
-Running a minimal `collapse_submodel` example produced a 2-layer model without errors:
-
-```
-collapsed_layers 2
-```
-
-
-## Stress Test 2025
-- `pip install -r requirements.txt` succeeded.
-- `pytest -q` reported:
-```
-10 passed, 1 skipped
-```
-
-### Large Scale-Up
-Ran `python progressive_scaleup.py --steps 8 --eps 0.70`:
-```
-Step 0 validation loss: 0.7053
-Step 1 validation loss: 0.6945
-Scaled model to 2 layers and width 32
-Step 2 validation loss: 0.6953
-Scaled model to 4 layers and width 32
-Step 3 validation loss: 0.6820
-Scaled model to 8 layers and width 32
-Step 4 validation loss: 0.6722
-Scaled model to 16 layers and width 32
-Step 5 validation loss: 0.6664
-Scaled model to 32 layers and width 32
-Step 6 validation loss: 0.6663
-Scaled model to 64 layers and width 32
-Step 7 validation loss: 0.6742
-Scaled model to 128 layers and width 32
-```
-
-### Collapse Submodel
-Using `collapse_submodel` with small clusters produced:
-```
-collapsed_layers 3
-d_model 16
-```
-
-## WikiText Benchmark Attempt
-- `pip install -r requirements.txt` succeeded after installing torch 2.7.1+cpu.
-- Attempted to download WikiText2 via `datasets` but network access to the S3 bucket was blocked.
-- Fallback to random data: ran `python progressive_scaleup.py --steps 12 --width-mult 2.0`:
-```
-Step 7 validation loss: 0.6980
-Scaled model to 1 layers and width 32
-Step 8 validation loss: 0.7022
-Scaled model to 1 layers and width 32
-Step 9 validation loss: 0.7025
-Scaled model to 1 layers and width 32
-Step 10 validation loss: 0.7055
-Scaled model to 1 layers and width 32
-Step 11 validation loss: 0.6976
-Scaled model to 1 layers and width 32
-```
-- Collapsing a toy cluster produced:
-```
-collapsed_layers 1
-```
-
-## WikiText Benchmark (datasets)
-Using the HuggingFace `datasets` loader with a small subset:
-```
-Step 0 validation loss: 0.6237
-Scaled model to 2 layers and width 64
-Step 1 validation loss: 0.5894
-Scaled model to 4 layers and width 128
-Step 2 validation loss: 0.5108
-Scaled model to 8 layers and width 256
-Step 3 validation loss: 0.8422
-Collapsed model validation loss: 0.6019973754882812
-```
-
-## WikiText Schedule Benchmark
-Installed requirements via pip and ran `python wikitext_schedule.py --steps 10 --max-len 16 --dataset-size 10`:
-```
-Step 0 validation loss: 0.6686
-Scaled model to 2 layers and width 32
-Step 1 validation loss: 0.6271
-Scaled model to 2 layers and width 64
-Step 2 validation loss: 0.7467
-Scaled model to 4 layers and width 64
-Step 3 validation loss: 0.6571
-Scaled model to 4 layers and width 128
-Step 4 validation loss: 0.7457
-Scaled model to 8 layers and width 128
-Step 5 validation loss: 0.8038
-Scaled model to 8 layers and width 256
-Step 6 validation loss: 2.6579
-Scaled model to 16 layers and width 256
-Step 7 validation loss: 4.0604
-Scaled model to 16 layers and width 512
-Step 8 validation loss: 8.6210
-Scaled model to 32 layers and width 512
-Step 9 validation loss: 6.4301
-Scaled model to 32 layers and width 1024
-Step 10 validation loss: 11.1592
-```
-Attempting the full 12-step run exceeded memory limits and the process was killed after step 10.
-
-## Recursive Integration Flow Test
-Installed requirements manually and ran `python recursive_integration_flow.py`. Output:
-
-```
-  warnings.warn(
-/workspace/Test/recursive_integration_flow.py:87: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
-  with torch.cpu.amp.autocast(dtype=torch.bfloat16):
-Step 0 validation loss: 1.2578 K=0.105 C=0.328 S=0.329
-Step 1 validation loss: 0.7305 K=0.031 C=0.095 S=0.244
-⚠️ Step 1 regressed below metric floor. Halting.
-Traceback (most recent call last):
-  File "/workspace/Test/recursive_integration_flow.py", line 119, in <module>
-    recursive_integration_flow()
-  File "/workspace/Test/recursive_integration_flow.py", line 93, in recursive_integration_flow
-    safe_output = hil_safe_inference(
-                  ^^^^^^^^^^^^^^^^^^^
-  File "/workspace/Test/bit_transformer/safety.py", line 24, in hil_safe_inference
-    raise RuntimeError(
-RuntimeError: Safety gate triggered: C=0.603, S=0.248
-```
-
-New successful run after adjusting metric floors:
-
-```
-Step 0 validation loss: 0.7461 K=0.038 C=0.084 S=0.246
-Step 1 validation loss: 0.7344 K=0.036 C=0.073 S=0.243
-Step 2 validation loss: 0.7266 K=0.029 C=0.074 S=0.242
-Step 3 validation loss: 0.7656 K=0.054 C=0.093 S=0.245
-Step 4 validation loss: 0.7422 K=0.026 C=0.097 S=0.241
-Compilation skipped: Dynamo is not supported on Python 3.12+
-Safe output bits: [[1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1]]
-```
-New run with torch-2.7.1+cpu installed from requirements and compile disabled:
-```
-Step 0 validation loss: 1.8750 K=0.152 C=0.314 S=0.345
-Step 1 validation loss: 1.0625 K=0.305 C=0.101 S=0.302
-Step 2 validation loss: 0.7266 K=0.028 C=0.083 S=0.244
-Step 3 validation loss: 0.7773 K=0.045 C=0.175 S=0.254
-Step 4 validation loss: 0.7539 K=0.031 C=0.122 S=0.245
-Safe output bits: [[0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0]]
-```
-Run with pinned dependencies from updated `requirements.txt`:
-```
-Step 0 validation loss: 2.4531 K=0.195 C=0.287 S=0.346
-Step 1 validation loss: 1.5781 K=0.176 C=0.307 S=0.340
-Step 2 validation loss: 0.7383 K=0.037 C=0.112 S=0.245
-Step 3 validation loss: 0.7773 K=0.038 C=0.178 S=0.251
-Step 4 validation loss: 0.7227 K=0.028 C=0.099 S=0.239
-Safe output bits: [[1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]]
-```
-
-## WikiText Schedule with Compression
-Ran `python wikitext_schedule.py --steps 2 --dataset-size 64` using the new compression-aware training.
-
-```
-Step 0 validation loss: 0.6969
-Scaled model to 2 layers and width 32
-Step 1 validation loss: 0.6840
-Scaled model to 2 layers and width 64
-Step 2 validation loss: 0.6746
-```
-## WikiText Schedule 10-step Run with Compression
-Step 0 validation loss: 2.1250
-Scaled model to 2 layers and width 32
-Step 1 validation loss: 2.2188
-Scaled model to 2 layers and width 64
-Step 2 validation loss: 6.0000
-Scaled model to 4 layers and width 64
-Step 3 validation loss: 6.3750
-Scaled model to 4 layers and width 128
-Step 4 validation loss: 4.7812
-Scaled model to 8 layers and width 128
-Step 5 validation loss: 3.8594
-Scaled model to 8 layers and width 256
-Step 6 validation loss: 7.2812
-Scaled model to 16 layers and width 256
-Step 7 validation loss: 9.8125
-Scaled model to 16 layers and width 512
-Step 8 validation loss: 34.5000
-Scaled model to 32 layers and width 512
-Step 9 validation loss: 39.7500
-Scaled model to 32 layers and width 1024
-Step 10 validation loss: 163.0000
-
-### 10-step Run with ACT Enabled
-Attempted to rerun the 10-step schedule with `use_act=True` and dataset size 128.
-Training was interrupted due to time limits after step 8. Partial results:
-```
-Step 0 validation loss: 1.8594
-Scaled model to 2 layers and width 32
-Step 1 validation loss: 0.7344
-Scaled model to 2 layers and width 64
-Step 2 validation loss: 0.5469
-Scaled model to 4 layers and width 64
-Step 3 validation loss: 0.2520
-Scaled model to 4 layers and width 128
-Step 4 validation loss: 0.1748
-Scaled model to 8 layers and width 128
-Step 5 validation loss: 0.0284
-Scaled model to 8 layers and width 256
-Step 6 validation loss: 0.1982
-Scaled model to 16 layers and width 256
-Step 7 validation loss: 0.1562
-Scaled model to 16 layers and width 512
-Step 8 validation loss: 0.2168
-Scaled model to 32 layers and width 512
-```
-
-## WikiText-103 100MB Attempt
-Attempted to run training with 100MB of WikiText-103 data streamed via `datasets` and converted to bits. Converting the dataset (352k lines) took too long and the process was interrupted before the first training step could complete.
-
-
-## Offline Full Bits Training Attempt
-- Installed requirements successfully.
-- Built `full_bits.pt` (100MB WikiText-103 compressed to bits).
-- Ran `python full_bits_train.py` but the training loop was extremely slow and was manually interrupted before completing a single pass.
-
-## BitSeq Dataset Training
-- Built `full_bits.pt` from WikiText2 using `build_full_bits.py`.
-- Ran `python full_bits_train.py` with BitSeq DataLoader (seq=2048, batch=8).
-- The script loaded one batch and reported `Batch loss: 2.4375`.
-
-## Offline train_full_sequence Scale-Up (8 steps)
-- Built dataset with `python build_full_bits.py` (~84MB).
-- Trained using `BitTransformerLM.train_full_sequence` over the first 65k bits with ctx_bits=64.
-```
-Step 0 train loss: 3.7605
-Step 1 train loss: 3.7545
-Step 2 train loss: 3.7434
-Step 3 train loss: 3.7382
-Step 4 train loss: 3.7301
-Step 5 train loss: 3.7261
-Step 6 train loss: 3.7202
-Step 7 train loss: 3.7060
-```
-
-## Progressive Scale-Up 8-Step Run
-```
-Step 0 validation loss: 0.7042
-Step 1 validation loss: 0.7036
-Step 2 validation loss: 0.7061
-Step 3 validation loss: 0.6997
-Step 4 validation loss: 0.7072
-Step 5 validation loss: 0.6892
-Step 6 validation loss: 0.7085
-Step 7 validation loss: 0.6966
-```
-
-## Compression Inference Test
-Installed requirements and ran `python wikitext_schedule.py --steps 2 --dataset-size 64`:
-```
-Step 0 validation loss: 0.9297
-Scaled model to 2 layers and width 32
-Step 1 validation loss: 0.7773
-Scaled model to 2 layers and width 64
-Step 2 validation loss: 0.7773
-```
-
-Ran a minimal training cycle with compression and generated text from the model:
-```
-Model output: hllo world
-```
-
-
-## Bigger Batch Smoke Test
-Executed `python unified_workflow.py --steps 9 --dataset-size 100` after adding warm-up optimisation. Final lines:
-```
-Epoch 1 raw_loss=0.5525 acc=0.692 | compressed_loss=0.5449 acc=0.718 direct_loss=0.0000 ratio=1.07
-Step 8 validation loss: 0.4727 K=0.248 C=0.126 S=0.309
-Final validation loss: 0.4824 K=0.245 C=0.131 S=0.308
-Safety gate triggered Safety gate triggered: C=0.476, S=0.292
-Collapsed model validation loss: 0.6928360462188721
-```
-
-### Inference Conversation
-```
-User: hi
-Model: hi
-User: ok
-Model: ok
-```
-
-## Bigger Training Smoke Test
-
-Executed `python unified_workflow.py --steps 7 --dataset-size 64` after updating
-the training loop with extra optimizer steps. Final lines:
-
-```
-Step 6 validation loss: 0.4922 K=0.252 C=0.118 S=0.306
-Final validation loss: 0.4785 K=0.264 C=0.105 S=0.307
-Safety gate triggered Safety gate triggered: C=0.476, S=0.297
-Collapsed model validation loss: 0.6666421890258789
-Workflow results: [(0, 1.015625, 0.2431640625, 0.126953125, 0.30909082293510437), (1, 0.74609375, 0.04248046875, 0.0306396484375, 0.2524452209472656), (2, 0.66796875, 0.11181640625, 0.06396484375, 0.2690799832344055), (3, 0.734375, 0.095703125, 0.044189453125, 0.2644684910774231), (4, 0.5546875, 0.220703125, 0.08837890625, 0.29613998532295227), (5, 0.73046875, 0.03759765625, 0.0654296875, 0.25516262650489807), (6, 0.4921875, 0.251953125, 0.11767578125, 0.30603474378585815), (7, 0.478515625, 0.263671875, 0.10498046875, 0.3072776794433594)]
-```
-
-### Inference Conversation (temperature=0.9, top-p=0.95)
-
-```
-User: hi
-Model: hi
-User: how are you?
-Model: how are you?
-```
-
-## Continuous Training Test
-Loaded existing weights when present.
-Performed 2 scaling steps and 1 plateau step on a 16-sample dataset.
-Final validation loss: 0.7383 with the collapsed model at 0.6924.
-
-## Diffusion LM Smoke Test
-Installed requirements and ran `python unified_workflow.py --steps 2 --dataset-size 32 --max-len 32 --diffusion`:
-```
-Epoch 0 raw_loss=4.7188 acc=0.188 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Epoch 1 raw_loss=4.6094 acc=0.185 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 0 validation loss: 3.9844 K=0.311 C=0.109 S=0.351
-Epoch 0 raw_loss=3.6445 acc=0.355 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Epoch 1 raw_loss=2.4531 acc=0.544 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 1 validation loss: 3.2656 K=0.371 C=0.088 S=0.357
-Final validation loss: 3.2344 K=0.373 C=0.087 S=0.357
-Diffusion sample: [1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0]
-Diffusion inference output bits: [0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-```
-
-## Rigorous Training Regime
-Ran `python tests/rigorous_training_regime.py`:
-
-```
-### Progressive Scale-Up (causal=True)
-
-Step 0 validation loss: 0.7167
-Scaled model to 1 layers and width 32
-Step 1 validation loss: 0.6880
-Scaled model to 1 layers and width 32
-Step 2 validation loss: 0.7019
-Scaled model to 1 layers and width 32
-Duration: 0.23s
-
-### Progressive Scale-Up (causal=False)
-
-Step 0 validation loss: 0.8581
-Scaled model to 1 layers and width 32
-Step 1 validation loss: 0.7439
-Scaled model to 1 layers and width 32
-Step 2 validation loss: 0.7068
-Scaled model to 1 layers and width 32
-Duration: 0.21s
-
-### Unified Workflow (causal=True)
-
-Loaded model from weights/model.pt.gz
-Epoch 0 raw_loss=0.6719 acc=0.581 | compressed_loss=0.6875 acc=0.586 direct_loss=0.0000 ratio=1.09
-Step 0 validation loss: 0.6367 K=0.091 C=0.069 S=0.284
-Epoch 0 raw_loss=0.6328 acc=0.605 | compressed_loss=0.6328 acc=0.612 direct_loss=0.0000 ratio=1.09
-Step 1 validation loss: 0.6914 K=0.202 C=0.049 S=0.305
-Epoch 0 raw_loss=0.5312 acc=0.718 | compressed_loss=0.6445 acc=0.628 direct_loss=0.0000 ratio=1.09
-Plateau 0 validation loss: 0.5469 K=0.096 C=0.118 S=0.290
-Final validation loss: 0.5430 K=0.099 C=0.104 S=0.289
-Safety gate triggered Safety gate triggered: C=0.484, S=0.285
-Collapsed model validation loss: 0.8396304845809937
-Workflow results: [(0, 0.63671875, 0.09130859375, 0.0693359375, 0.28369221091270447), (1, 0.69140625, 0.2021484375, 0.049072265625, 0.3053092062473297), (2, 0.546875, 0.09619140625, 0.1181640625, 0.2900315225124359), (3, 0.54296875, 0.09912109375, 0.10400390625, 0.289362370967865)]
-Inference on 'hi': [0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1]
-
-Duration: 8.48s
-
-### Unified Workflow (causal=False / Diffusion)
-
-Loaded model from weights/model.pt.gz
-Epoch 0 raw_loss=0.8232 acc=0.391 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 0 validation loss: 0.9805 K=0.098 C=0.067 S=0.285
-Epoch 0 raw_loss=0.7471 acc=0.561 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 1 validation loss: 1.0547 K=0.134 C=0.091 S=0.294
-Epoch 0 raw_loss=0.7520 acc=0.609 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Plateau 0 validation loss: 0.2119 K=0.187 C=0.185 S=0.332
-Final validation loss: 0.2188 K=0.187 C=0.176 S=0.330
-Collapsed model validation loss: 0.6897413730621338
-Diffusion sample: [1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1]
-Workflow results: [(0, 0.98046875, 0.09765625, 0.06689453125, 0.28478696942329407), (1, 1.0546875, 0.1337890625, 0.0908203125, 0.29406091570854187), (2, 0.2119140625, 0.1865234375, 0.1845703125, 0.33178743720054626), (3, 0.21875, 0.1865234375, 0.17578125, 0.32961323857307434)]
-Diffusion inference output bits: [1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1]
-Duration: 24.25s
-```
-
-## Rigorous Training Regime (2025-08-06)
-Ran `python tests/rigorous_training_regime.py`:
-
-```
-### Progressive Scale-Up (causal=True)
-
-Step 0 validation loss: 0.6921
-Scaled model to 1 layers and width 32
-Step 1 validation loss: 0.7171
-Scaled model to 1 layers and width 32
-Step 2 validation loss: 0.6914
-Scaled model to 1 layers and width 32
-Duration: 0.27s
-
-### Progressive Scale-Up (causal=False)
-
-Step 0 validation loss: 0.8465
-Scaled model to 1 layers and width 32
-Step 1 validation loss: 0.7123
-Scaled model to 1 layers and width 32
-Step 2 validation loss: 0.7009
-Scaled model to 1 layers and width 32
-Duration: 0.26s
-
-### Unified Workflow (causal=True)
-
-Epoch 0 raw_loss=1.1094 acc=0.593 | compressed_loss=1.1465 acc=0.599 direct_loss=0.0000 ratio=1.09
-Step 0 validation loss: 0.8945 K=0.301 C=0.092 S=0.339
-Epoch 0 raw_loss=0.9453 acc=0.601 | compressed_loss=0.9707 acc=0.617 direct_loss=0.0000 ratio=1.09
-Step 1 validation loss: 0.9180 K=0.301 C=0.088 S=0.338
-Epoch 0 raw_loss=0.8984 acc=0.593 | compressed_loss=0.9590 acc=0.599 direct_loss=0.0000 ratio=1.09
-Plateau 0 validation loss: 0.7969 K=0.243 C=0.095 S=0.324
-Final validation loss: 0.7930 K=0.244 C=0.094 S=0.324
-Safety gate triggered Safety gate triggered: C=0.484, S=0.314
-Collapsed model validation loss: 0.6552348732948303
-Workflow results: [(0, 0.89453125, 0.30078125, 0.09228515625, 0.33890560269355774), (1, 0.91796875, 0.30078125, 0.08837890625, 0.33844876289367676), (2, 0.796875, 0.2431640625, 0.0947265625, 0.32405367493629456), (3, 0.79296875, 0.244140625, 0.09423828125, 0.32419103384017944)]
-Inference on 'hi': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
-
-Duration: 5.26s
-
-### Unified Workflow (causal=False / Diffusion)
-
-Loaded model from weights/model.pt.gz
-Epoch 0 raw_loss=1.2266 acc=0.590 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 0 validation loss: 0.8359 K=0.165 C=0.032 S=0.296
-Epoch 0 raw_loss=0.7617 acc=0.603 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 1 validation loss: 0.7891 K=0.025 C=0.043 S=0.268
-Epoch 0 raw_loss=0.7158 acc=0.553 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Plateau 0 validation loss: 0.5391 K=0.113 C=0.056 S=0.287
-Final validation loss: 0.5391 K=0.116 C=0.060 S=0.287
-Collapsed model validation loss: 0.7268564701080322
-Diffusion sample: [1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1]
-Workflow results: [(0, 0.8359375, 0.1650390625, 0.0322265625, 0.29598498344421387), (1, 0.7890625, 0.0250244140625, 0.04345703125, 0.26766154170036316), (2, 0.5390625, 0.11328125, 0.05615234375, 0.2867652475833893), (3, 0.5390625, 0.1162109375, 0.06005859375, 0.28735819458961487)]
-Diffusion inference output bits: [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0]
-Duration: 3.70s
-```
-
-## Rigorous Training Regime (2025-08-06 - 10-step alt length/width)
-Ran `python tests/rigorous_training_regime.py`:
-
-```
-### Progressive Scale-Up (causal=True)
-
-Step 0 validation loss: 0.4615
-Step 1 validation loss: 0.4427
-Step 2 validation loss: 0.4282
-Step 3 validation loss: 0.4202
-Step 4 validation loss: 0.4175
-Scaled length; seq_len=128 width=32 params=8674
-Step 5 validation loss: 0.5383
-Scaled width; seq_len=128 width=64 params=33730
-Step 6 validation loss: 0.4334
-Step 7 validation loss: 0.4304
-Scaled length; seq_len=256 width=64 params=33730
-Step 8 validation loss: 0.5085
-Scaled width; seq_len=256 width=128 params=132994
-Step 9 validation loss: 0.4279
-Duration: 38.96s
-
-### Progressive Scale-Up (causal=False)
-
-Step 0 validation loss: 0.4292
-Step 1 validation loss: 0.4053
-Step 2 validation loss: 0.4003
-Step 3 validation loss: 0.3997
-Scaled length; seq_len=128 width=32 params=8674
-Step 4 validation loss: 0.4162
-Scaled width; seq_len=128 width=64 params=33730
-Step 5 validation loss: 0.4173
-Scaled length; seq_len=256 width=64 params=33730
-Step 6 validation loss: 0.4160
-Scaled width; seq_len=256 width=128 params=132994
-Step 7 validation loss: 0.4211
-Scaled length; seq_len=512 width=128 params=132994
-Step 8 validation loss: 0.4227
-Scaled width; seq_len=512 width=256 params=528130
-Step 9 validation loss: 0.4146
-Duration: 173.71s
-
-### Unified Workflow (causal=True)
-
-Epoch 0 raw_loss=3.1562 acc=0.540 | compressed_loss=3.4531 acc=0.529 direct_loss=0.0000 ratio=1.09
-Step 0 validation loss: 2.9688 K=0.559 C=0.220 S=0.475
-Epoch 0 raw_loss=2.7188 acc=0.540 | compressed_loss=2.9883 acc=0.529 direct_loss=0.0000 ratio=1.09
-Step 1 validation loss: 3.4531 K=0.566 C=0.222 S=0.481
-Epoch 0 raw_loss=3.0625 acc=0.540 | compressed_loss=3.4414 acc=0.529 direct_loss=0.0000 ratio=1.09
-Plateau 0 validation loss: 3.0781 K=0.559 C=0.219 S=0.474
-Final validation loss: 3.0938 K=0.559 C=0.220 S=0.475
-Safety gate triggered Safety gate triggered: C=0.484, S=0.466
-Collapsed model validation loss: 0.6677278280258179
-Workflow results: [(0, 2.96875, 0.55859375, 0.2197265625, 0.4746275246143341), (1, 3.453125, 0.56640625, 0.2216796875, 0.4808752238750458), (2, 3.078125, 0.55859375, 0.21875, 0.47436484694480896), (3, 3.09375, 0.55859375, 0.2197265625, 0.474519282579422)]
-Inference on 'hi': [1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1]
-
-Duration: 2.50s
-
-### Unified Workflow (causal=False / Diffusion)
-
-Loaded model from weights/model.pt.gz
-Epoch 0 raw_loss=4.3984 acc=0.271 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 0 validation loss: 4.9688 K=0.512 C=0.208 S=0.449
-Epoch 0 raw_loss=3.5859 acc=0.225 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Step 1 validation loss: 4.6562 K=0.477 C=0.200 S=0.428
-Epoch 0 raw_loss=3.3008 acc=0.225 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-Plateau 0 validation loss: 3.5469 K=0.439 C=0.158 S=0.396
-Final validation loss: 3.5625 K=0.436 C=0.156 S=0.396
-Collapsed model validation loss: 0.6747412085533142
-Diffusion sample: [1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1]
-Workflow results: [(0, 4.96875, 0.51171875, 0.2080078125, 0.44865939021110535), (1, 4.65625, 0.4765625, 0.2001953125, 0.4284386932849884), (2, 3.546875, 0.439453125, 0.158203125, 0.3957676589488983), (3, 3.5625, 0.435546875, 0.15625, 0.39555999636650085)]
-Diffusion inference output bits: [1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1]
-Duration: 3.42s
-```
-
-## WikiText Training Attempt (2025-09-??)
-Attempted minimal training on real WikiText-2 data using `train_loop` with dropout 0.1 and evaluation dropout 0.0. Training failed due to a telemetry shape mismatch:
-
-```
-RuntimeError: The size of tensor a (4) must match the size of tensor b (64) at non-singleton dimension 1
-```
-
-As a sanity check, ran `hil_safe_inference` on an untrained model in evaluation mode (dropout=0.0):
-
-```
-Inference output bits: [[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
-```
-
-## WikiText Training Debug (2025-09-??)
-Ran a minimal `train_loop` on parity-protected WikiText-2 samples with dropout 0.1:
-
-```
-Epoch 0 raw_loss=0.6278 acc=0.724 | compressed_loss=0.0000 acc=0.000 direct_loss=0.0000 ratio=0.00
-```

unified_workflow.py

@@ -1,165 +0,0 @@
-import argparse
-import os
-import subprocess
-import sys
-import time
-import torch
-from bit_transformer.utils import load_model
-from bit_transformer.hf_checkpoint import (
-    hf_login,
-    save_checkpoint,
-    download_checkpoint,
-)
-from bit_transformer import diffusion_inference
-from bit_transformer.cli_standards import create_workflow_parser, BitTransformerCLI
-
-from integration_schedule import integration_schedule
-
-
-def _launch_dashboard() -> list[subprocess.Popen]:
-    """Start MCP server and dashboard processes."""
-    server = subprocess.Popen([sys.executable, "mcp_server.py"]) 
-    time.sleep(2)
-    dash_env = dict(os.environ)
-    dash_env.setdefault("MCP_SERVER_ADDR", "http://127.0.0.1:7000")
-    dashboard = subprocess.Popen(
-        [sys.executable, "-m", "bit_transformer.dashboard_app"],
-        env=dash_env,
-    )
-    return [server, dashboard]
-
-
-def _terminate(procs: list[subprocess.Popen]) -> None:
-    for p in procs:
-        p.terminate()
-        try:
-            p.wait(timeout=5)
-        except Exception:
-            p.kill()
-
-
-def run_workflow(
-    steps: int = 10,
-    max_len: int = 64,
-    dataset_size: int = 128,
-    *,
-    launch_ui: bool = False,
-    weights_path: str = "weights/model.pt.gz",
-    collapsed_path: str = "weights/collapsed.pt.gz",
-    plateau_steps: int = 0,
-    epochs_per_step: int = 2,
-    extra_steps: int = 3,
-    collapse: bool = True,
-    hf_repo: str | None = None,
-    hf_token: str | None = None,
-    diffusion: bool = False,
-    noise_schedule: str = "linear",
-    diffusion_steps: int = 8,
-    diffusion_curriculum: bool = False,
-    use_checkpoint: bool = True,
-    reversible: bool = True,
-    qat: bool = False,
-) -> tuple:
-    """Run the full integration schedule with optional dashboard.
-
-    If ``qat`` is ``True`` the model undergoes 4-bit quantization-aware training
-    before being converted to quantized weights for safety checks.
-    """
-    procs: list[subprocess.Popen] = []
-    if launch_ui:
-        procs = _launch_dashboard()
-    if hf_repo:
-        hf_login(token=hf_token)
-        if not os.path.exists(weights_path):
-            download_checkpoint(weights_path, repo_id=hf_repo)
-    try:
-        results, collapsed = integration_schedule(
-            steps=steps,
-            max_len=max_len,
-            dataset_size=dataset_size,
-            weights_path=weights_path,
-            plateau_steps=plateau_steps,
-            collapsed_path=collapsed_path,
-            epochs_per_step=epochs_per_step,
-            extra_steps=extra_steps,
-            collapse=collapse,
-            diffusion=diffusion,
-            noise_schedule=noise_schedule,
-            diffusion_steps=diffusion_steps,
-            diffusion_curriculum=diffusion_curriculum,
-            use_checkpoint=use_checkpoint,
-            reversible=reversible,
-            qat=qat,
-        )
-        model = load_model(weights_path)
-        print("Workflow results:", results)
-        if diffusion:
-            sample = diffusion_inference(
-                model, length=max_len, steps=diffusion_steps, schedule=noise_schedule
-            )
-            print("Diffusion inference output bits:", sample[0].tolist())
-        if hf_repo:
-            save_checkpoint(model, repo_id=hf_repo)
-    finally:
-        if launch_ui:
-            _terminate(procs)
-    return model, collapsed
-
-
-if __name__ == "__main__":
-    # Use standardized CLI parser
-    parser = create_workflow_parser()
-    
-    # Add workflow-specific arguments
-    workflow_group = parser.add_argument_group('Workflow Configuration')
-    workflow_group.add_argument("--steps", type=int, default=10, 
-                               help="Number of progressive scale-up steps")
-    workflow_group.add_argument("--plateau-steps", type=int, default=0, 
-                               help="Extra training steps at final size")
-    workflow_group.add_argument("--epochs-per-step", type=int, default=2, 
-                               help="Epochs per training step")
-    workflow_group.add_argument("--extra-steps", type=int, default=3, 
-                               help="Optimizer updates after each epoch")
-    workflow_group.add_argument("--no-collapse", action="store_true", 
-                               help="Skip collapsed model generation")
-    workflow_group.add_argument("--dashboard", action="store_true", 
-                               help="Launch MCP server and dashboard UI")
-    
-    # Add advanced optimization arguments
-    opt_group = parser.add_argument_group('Advanced Optimization')
-    opt_group.add_argument("--no-checkpoint", action="store_true", 
-                          help="Disable gradient checkpointing (faster but more memory)")
-    opt_group.add_argument("--no-reversible", action="store_true", 
-                          help="Use standard transformer blocks instead of reversible layers")
-    opt_group.add_argument("--qat", action="store_true", 
-                          help="Enable 4-bit quantization-aware training")
-                          
-    # Override some defaults for workflow context
-    parser.set_defaults(
-        seq_length=64,  # Use seq-length instead of max-len
-        dataset_size=128,
-        weights_path="weights/model.pt.gz"
-    )
-    args = parser.parse_args()
-
-    run_workflow(
-        args.steps,
-        args.seq_length,  # Standardized name
-        args.dataset_size,
-        launch_ui=args.dashboard,
-        weights_path=args.weights_path,
-        collapsed_path=getattr(args, 'collapsed_path', 'weights/collapsed.pt.gz'),
-        plateau_steps=args.plateau_steps,
-        epochs_per_step=args.epochs_per_step,
-        extra_steps=args.extra_steps,
-        collapse=not args.no_collapse,
-        hf_repo=args.hf_repo,
-        hf_token=args.hf_token,
-        diffusion=args.diffusion_mode,  # Standardized name
-        noise_schedule=args.noise_schedule,
-        diffusion_steps=args.diffusion_steps,
-        diffusion_curriculum=args.diffusion_curriculum,
-        use_checkpoint=not args.no_checkpoint,
-        reversible=not args.no_reversible,
-        qat=args.qat,
-    )

watcher.py

@@ -1,80 +0,0 @@
-import argparse
-import subprocess
-import sys
-import time
-from pathlib import Path
-from watchdog.events import FileSystemEventHandler
-from watchdog.observers import Observer
-
-
-class RestartOnChange(FileSystemEventHandler):
-    """Restart a subprocess when watched files change."""
-
-    def __init__(self, command: list[str], watch_paths: list[str]) -> None:
-        self.command = command
-        self.watch_paths = [Path(p).resolve() for p in watch_paths]
-        self.process: subprocess.Popen | None = None
-        self.restart()
-
-    def restart(self) -> None:
-        if self.process and self.process.poll() is None:
-            self.process.terminate()
-            try:
-                self.process.wait(timeout=5)
-            except subprocess.TimeoutExpired:
-                self.process.kill()
-                self.process.wait()
-        self.process = subprocess.Popen(self.command)
-
-    def on_any_event(self, event) -> None:  # pragma: no cover - runtime utility
-        if event.is_directory:
-            return
-        path = Path(event.src_path)
-        if path.suffix != ".py":
-            return
-        if any(str(path).startswith(str(p)) for p in self.watch_paths):
-            print(f"[watcher] {path} changed, running tests...")
-            subprocess.run([sys.executable, "-m", "pytest", "-q"])
-            print("[watcher] restarting process...")
-            self.restart()
-
-
-def main() -> None:  # pragma: no cover - CLI entry
-    parser = argparse.ArgumentParser(
-        description="Watch files and restart a command on changes",
-    )
-    parser.add_argument(
-        "--command",
-        nargs="+",
-        default=[sys.executable, "mcp_server.py"],
-        help="Command to run",
-    )
-    parser.add_argument(
-        "--paths",
-        nargs="+",
-        default=["bit_transformer", "mcp_server.py"],
-        help="Paths to watch for changes",
-    )
-    args = parser.parse_args()
-
-    observer = Observer()
-    handler = RestartOnChange(args.command, args.paths)
-    for p in args.paths:
-        observer.schedule(handler, p, recursive=True)
-    observer.start()
-    try:
-        while True:
-            time.sleep(1)
-    except KeyboardInterrupt:
-        pass
-    finally:
-        observer.stop()
-        handler.restart()
-        if handler.process and handler.process.poll() is None:
-            handler.process.terminate()
-            handler.process.wait()
-        observer.join()
-
-
-if __name__ == "__main__":  # pragma: no cover - CLI entry
-    main()

wikitext_benchmark.py

@@ -1,47 +0,0 @@
-import torch
-import torch.nn.functional as F
-from datasets import load_dataset
-from bit_transformer import text_to_bits, collapse_submodel
-from progressive_scaleup import progressive_scale_up_text
-
-
-def lines_to_bits(lines, max_len=64):
-    data = []
-    for text in lines:
-        bits = text_to_bits(text)[:max_len]
-        if len(bits) < max_len:
-            bits.extend([0] * (max_len - len(bits)))
-        data.append(bits)
-    return data
-
-
-def main():
-    ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train[:1%]")
-    val_ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="validation[:1%]")
-    train_lines = [item["text"] for item in ds][:256]
-    valid_lines = [item["text"] for item in val_ds][:64]
-
-    train_bits = lines_to_bits(train_lines)
-    valid_bits = lines_to_bits(valid_lines)
-
-    progressive_scale_up_text(
-        eps=0.65,
-        steps=4,
-        width_mult=2.0,
-        max_len=64,
-        dataset_size=min(64, len(train_bits)),
-    )
-
-    target_params = dict(d_model=16, nhead=4, num_layers=1, dim_feedforward=64, max_seq_len=64)
-    model, _ = collapse_submodel(train_bits[:64], target_params, max_rounds=1)
-
-    val_tensor = torch.tensor(valid_bits, dtype=torch.long)
-    logits, _ = model(val_tensor)
-    pred = logits[:, :-1, :].reshape(-1, 2)
-    target = val_tensor[:, 1:].reshape(-1)
-    loss = F.cross_entropy(pred, target)
-    print("Collapsed model validation loss:", loss.item())
-
-
-if __name__ == "__main__":
-    main()

wikitext_schedule.py

@@ -1,130 +0,0 @@
-import numpy as np
-import torch
-import torch.nn.functional as F
-from torch.utils.data import Dataset
-from pathlib import Path
-from datasets import load_dataset
-
-from bit_transformer import (
-    BitTransformerLM,
-    configure_optimizer,
-    expand_model,
-    text_to_bits,
-)
-from bit_transformer.training import train_loop as basic_train
-
-
-def _build_memmap(lines, path: Path, max_len: int) -> None:
-    """Precompute bit tensors into a memory-mapped file."""
-    arr = np.memmap(path, mode="w+", shape=(len(lines), max_len), dtype="uint8")
-    for idx, text in enumerate(lines):
-        bits = text_to_bits(text)[:max_len]
-        if len(bits) < max_len:
-            bits.extend([0] * (max_len - len(bits)))
-        arr[idx] = np.array(bits, dtype="uint8")
-    arr.flush()
-
-
-class MemmapDataset(Dataset):
-    """Dataset backed by a memory-mapped array."""
-
-    def __init__(self, path: Path, length: int, max_len: int) -> None:
-        self.path = path
-        self.length = length
-        self.max_len = max_len
-        self._arr = np.memmap(path, mode="r", shape=(length, max_len), dtype="uint8")
-
-    def __len__(self) -> int:  # pragma: no cover - trivial
-        return self.length
-
-    def __getitem__(self, idx: int) -> torch.Tensor:
-        return torch.from_numpy(self._arr[idx].astype("int64"))
-
-
-def progressive_scale_schedule(steps=12, max_len=64, dataset_size=128):
-    """Run deterministic scale-up on WikiText data."""
-    ds = load_dataset("wikitext", "wikitext-2-raw-v1")
-    train_lines = [t for t in ds["train"]["text"] if t.strip()][:dataset_size]
-    valid_lines = [t for t in ds["validation"]["text"] if t.strip()][: dataset_size // 4]
-
-    train_path = Path("wikitext_train.memmap")
-    valid_path = Path("wikitext_valid.memmap")
-    _build_memmap(train_lines, train_path, max_len)
-    _build_memmap(valid_lines, valid_path, max_len)
-
-    train = MemmapDataset(train_path, len(train_lines), max_len)
-    valid = torch.from_numpy(
-        np.memmap(valid_path, mode="r", shape=(len(valid_lines), max_len), dtype="uint8")
-    ).long()
-
-    layers = 1
-    width = 32
-    params = dict(
-        d_model=width,
-        nhead=4,
-        num_layers=layers,
-        dim_feedforward=width * 2,
-        max_seq_len=max_len,
-        reversible=True,
-        chunk_size=max_len,
-        use_autocast=True,
-        use_act=True,
-        act_threshold=0.9,
-    )
-    model = BitTransformerLM(**params)
-    steps_per_epoch = max(1, (len(train) + 7) // 8)
-    optimizer, scheduler = configure_optimizer(model, lr=1e-3, total_steps=(steps + 1) * steps_per_epoch)
-
-    results = []
-    for step in range(steps + 1):
-        basic_train(
-            model,
-            train,
-            epochs=1,
-            compress_prob=0.5,
-            log=False,
-            forward_kwargs=None,
-            num_workers=2,
-        )
-
-        with torch.no_grad():
-            logits, _ = model(valid)
-            pred = logits[:, :-1, :].reshape(-1, 2)
-            target = valid[:, 1:].reshape(-1)
-            val_loss = F.cross_entropy(pred, target).item()
-        print(f"Step {step} validation loss: {val_loss:.4f}")
-        results.append((step, val_loss))
-
-        if step < steps:
-            if step % 2 == 0:
-                layers *= 2
-            else:
-                width *= 2
-            params = dict(
-                d_model=width,
-                nhead=4,
-                num_layers=layers,
-                dim_feedforward=width * 2,
-                max_seq_len=max_len,
-                reversible=True,
-                chunk_size=max_len,
-                use_autocast=True,
-                use_act=True,
-                act_threshold=0.9,
-            )
-            model = expand_model(model, params)
-            optimizer, scheduler = configure_optimizer(model, lr=1e-3, total_steps=(steps - step) * steps_per_epoch)
-            print(f"Scaled model to {layers} layers and width {width}")
-    return results
-
-
-if __name__ == "__main__":
-    import argparse
-
-    parser = argparse.ArgumentParser(description="Deterministic scale-up benchmark")
-    parser.add_argument("--steps", type=int, default=12, help="number of scale-up steps")
-    parser.add_argument("--max-len", type=int, default=64, help="sequence length")
-    parser.add_argument("--dataset-size", type=int, default=128, help="number of training lines")
-    args = parser.parse_args()
-
-    progressive_scale_schedule(steps=args.steps, max_len=args.max_len, dataset_size=args.dataset_size)