WCNegentropy
/

BitTransformerLM

@@ -1,177 +0,0 @@
-# BitTransformerLM
-**Project Status:** Pre-release (v1 candidate)
-BitTransformerLM is a bit-centric transformer language model built entirely in PyTorch. The project began as a small prototype but has matured into a near-production system capable of modeling raw binary streams with sophisticated safety telemetry and automated scale-up tooling. This repository now serves as the canonical implementation under WCNegentropy.
-## Historical Background
-- **Early Experiments** – Initial prototypes explored mapping text to parity-protected bits and training a minimal transformer on random data.
-- **Telemetry & Safety** – Added negentropy, LZ complexity and symbiosis scoring to measure information flow and gate unsafe outputs.
-- **Progressive Scaling** – Introduced reversible layers and automatic depth/width expansion for efficient curriculum training. The schedule now triggers expansions only when validation loss plateaus and decays the learning rate by √2 after each growth with a 100-step warm‑up.
-- **Compression Support** – Integrated run-length encoding and packed bit I/O with optional multi-task training on compressed sequences.
-- **Context Extension** – Implemented chunked attention and sliding-window inference for long sequences with optional overlapping windows.
-- **Attention Logging Toggle** – ``full_attn_logging=False`` skips reconstructing full ``T×T`` attention maps during chunked attention, cutting memory use for very long sequences.
-- **Diffusion LM Mode** – Enable bidirectional denoising by setting ``causal=False`` or toggling **Diffusion LM** in the dashboard. Chunked attention is automatically disabled in this mode and restored afterward.
-- **Dashboard & MCP Server** – Built a lightweight web UI backed by a management server for real‑time training, inference and model collapse. New `/metrics` and `/model_config` endpoints surface live telemetry and hyperparameters, and `/save_checkpoint` and `/download_checkpoint` enable Hugging Face weight sync. The insecure `/exec` route has been removed.
-- **Phase 1 Optimizations** – Configurable batch sizes with aligned OneCycle scheduling, gradient accumulation, mixed‑precision, memory‑mapped dataset streaming, scheduled compression ramps, selective ``torch.compile``, and an EMA‑smoothed safety gate with burn‑in to cut false positives.
-The codebase has undergone multiple stress tests and synthetic benchmarks (see `tests/TEST_RESULTS.md`) and now approaches a stable release.
-## Quick Start
-Install dependencies using the CPU wheel of PyTorch (default):
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
-```
-When GPU acceleration is toggled in the dashboard, the application automatically
-installs the CUDA-enabled wheel:
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.7.1+cu118
-```
-Run the example script:
-```bash
-python example.py
-```
-Adaptive scaling demo:
-The legacy `progressive_scaleup.py` script is retained for reference but has been
-superseded by `integration_schedule.py`, which offers a more flexible scaling
-workflow.
-Run the unified workflow:
-```bash
-python unified_workflow.py --dashboard
-# disable gradient checkpointing for faster but memory-hungry runs
-python unified_workflow.py --no-checkpoint
-# use standard (non-reversible) transformer blocks
-python unified_workflow.py --no-reversible
-# enable 4-bit quantization-aware training
-python unified_workflow.py --qat
-```
-For faster CPU execution, BitTransformerLM exposes a `cpu_autocast()` helper
-that enables bfloat16 mixed precision. Models created with
-`use_autocast=True` apply this automatically, or you can wrap individual
-forward passes:
-```python
-from bit_transformer.torch_utils import cpu_autocast
-with cpu_autocast():
-    logits, telemetry = model(bits)
-```
-Reduce memory use when chunked attention is active by disabling full
-attention logging:
-```python
-model = BitTransformerLM(chunk_size=128, full_attn_logging=False)
-```
-Enable Diffusion LM training and sampling:
-```bash
-python unified_workflow.py --diffusion --diffusion-steps 8 --dataset-size 32
-# choose noise schedule: linear, cosine, exp
-python unified_workflow.py --diffusion --noise-schedule cosine --diffusion-steps 16 --dataset-size 32
-# linearly decay noise over epochs
-python unified_workflow.py --diffusion --diffusion-curriculum --dataset-size 32
-```
-Higher `--diffusion-steps` (8–16) improves sample quality at the cost of compute. When using the dashboard, enable the **Diffusion LM** toggle to run the model without causal masking or chunked attention.
-Generated samples automatically fix parity bits so they can be decoded back to text.
-To resume training across machines using Hugging Face storage:
-```bash
-python unified_workflow.py --hf-repo your-username/bittransformerlm --hf-token $HF_TOKEN
-```
-The dashboard exposes matching controls under **Hugging Face Checkpoints**. Provide a repository ID and optional token (falling back to the `HF_TOKEN` environment variable) and click **Upload weights** or **Download weights** to sync the model.
-Run the unit tests:
-```bash
-pytest -q
-```
-### Mode management
-During training, ensure the model is in training mode with dropout enabled:
-```python
-from bit_transformer.utils import set_dropout
-model.train()
-set_dropout(model, 0.1)
-```
-Before running tests, performing inference, or committing weights to the repository, switch the model to evaluation mode and disable dropout:
-```python
-model.eval()
-set_dropout(model, 0.0)
-```
-This prevents CI failures from accidentally pushing weights that still have active dropout.
-## Telemetry Metrics Explained
-BitTransformerLM reports three bounded metrics in ``[0, 1]`` during training and inference:
-- **Negentropy (K)** – departure from random noise; ``1`` denotes perfectly ordered bits while ``0`` is uniform randomness.
-- **LZ Complexity (C)** – differentiable proxy for Lempel–Ziv compressibility; low values imply repetitive patterns and high values frequent transitions.
-- **Symbiosis (S)** – agreement between model predictions and a reference distribution via KL divergence; scores near ``1`` show strong alignment.
-An Adaptive Computation Time (ACT) mechanism lets layers halt early once confidence exceeds a threshold. Halt probabilities are exported as ``halt_probs`` in telemetry for inspection.
-These metrics are logged alongside losses and can trigger safety gates when thresholds are violated. The dashboard monitors drift and emits warnings when recent values deviate beyond a configurable threshold.
-## Core Features
-- **Bit-Native Modeling** – Works directly on 0/1 inputs with positional encodings and parity-protected text helpers.
-- **Telemetry Synthesizer** – Clusters activation summaries to surface coherent subspaces and detect drift.
-- **Submodel Distillation** – `TelemetrySynthesizer` selects representative sequences for `collapse_submodel`, which deepens
-  and widens once (`width_scale` = 1.5) if telemetry floors aren't met; `save_distilled_model` places a `metrics.json` summary
-  beside the distilled weights.
-- **Safety Gate** – `hil_safe_inference` enforces minimum complexity and symbiosis scores at runtime with EMA smoothing and a configurable burn‑in period.
-- **Quantization** – CPU inference can be quantized to int8 or trained with 4-bit QAT using the `--qat` flag.
-- **Distributed Training** – FSDP and pipeline helpers allow multi‑GPU scaling when hardware is available.
-- **Interactive Dashboard** – Live control of training, scaling and compression with optional GPU acceleration. The dashboard now exposes reversible layers, gradient checkpointing, ACT thresholds, λ floors, 4‑bit QAT and Diffusion LM toggles, real‑time telemetry charts powered by Chart.js, and Hugging Face checkpoint upload/download controls with `HF_TOKEN` fallback. Settings persist via `localStorage`.
-- **CI/CD Pipeline** – GitHub Actions install dependencies, run the tests and build distribution artifacts on every push.
-## Development Workflow
-1. Start the MCP server:
-   ```bash
-   python mcp_server.py
-   ```
-2. Launch the dashboard in another terminal:
-   ```bash
-   MCP_SERVER_ADDR=http://127.0.0.1:7000 python -m bit_transformer.dashboard_app
-   ```
-3. Submit training batches, scale the model and monitor telemetry from the web UI.
-   The dashboard's appearance is controlled by `bit_transformer/static/style.css`.
-A `watcher.py` script can automatically restart the server and run tests when files change during local development.
-## Container Deployment
-A `Dockerfile` and `start.sh` script build a minimal VM image that launches both the MCP server and dashboard.
-```bash
-docker build -t bittransformerlm .
-docker run -p 5000:5000 -p 7000:7000 bittransformerlm
-```
-By default the container installs the CPU-only PyTorch wheel. Set the build
-argument `TORCH_CUDA=cu118` to preinstall the GPU version. The container sets
-`MCP_SERVER_ADDR=http://127.0.0.1:7000` and exposes the dashboard on port 5000.
-## Roadmap
-- Finalize S attribution tools and metric drift detection.
-- Publish an initial release package and rename the repository to **BitTransformerLM**.
-- Continue benchmarking on real datasets and expanding context window capabilities.
-## Licensing
-This project is released under a combination of licenses and agreements to provide a clear framework for use, distribution, and contribution. All licensing documents can be found in the `LICENSE/` directory.
-The key documents are:
-* `LICENSE.txt`: The primary open-source license for the software, AGPLv3.
-* `COMMERCIAL_LICENSE.txt`: Terms for commercial use of the software.
-* `DISCLAIMER.txt`: Important legal disclaimers.
-* `ALIGNMENT_AND_TRANSPARENCY.txt`: Our commitment to alignment and transparency.
-* `TRADEMARK_POLICY.txt`: Guidelines for using the project's trademarks.
-* `CONTRIBUTOR_LICENSE_AGREEMENT.txt`: The agreement for all contributors to sign.
-Please review these documents carefully before using or contributing to the project.