Update README.md

README.md

@@ -1,25 +1,18 @@
 ---
-base_model: no_multiplication_model
+license: apache-2.0
+base_model: Qwen/Qwen2-0.5B
 tags:
-- experimental
-- inference
-- feature-selection
-- bayesian
-- Transformers
-- Safetensors
-- open-source
-- language-model
-- compression
-- quantisation
-
-
+  - experimental
+  - inference
+  - feature-selection
+  - bayesian
+  - compression
+  - lossless
 ---
 
-# **I made Qwen2-0.5B run 100× smaller with no matrix multiplications and almost no RAM usage — adaptive sparse selection at inference time, and it actually works.**
+# Adaptive Sparse Feature Selection at Inference Time (Qwen2-0.5B)
 
-# contact: Twitter: https://x.com/liberal17th  email: bogunusov@gmail.com 
-
-**Status: experimental / work in progress.** This is a research test, not a production release, not a compression method, and not a claim of a new architecture. Numbers below are placeholders — real plots and stats will be added once benchmark runs are complete.Currently, according to tests, it doesn't use any RAM at all, but there may be errors. With this compression, we compressed by about 30%, but the main matrices are not multiplied.
+**Status: experimental / work in progress.** This is a research test, not a production release, not a compression method, and not a claim of a new architecture. Numbers below are placeholders — real plots and stats will be added once benchmark runs are complete.
 
 ## What this actually is
 
@@ -27,87 +20,69 @@ During autoregressive generation, this repo tracks a small set of statistical fe
 
 The output metric is simple: **what fraction of tracked features get flagged as informative per step**, averaged over a generation. That's it. It's an exploration of whether activation statistics carry sparse, structured signal during inference — not a finished result and not a benchmark win yet.
 
+## What this is *not*
 
+To be upfront about scope, since it's easy to over-read either script:
 
-## Why this might be interesting anyway
+- **Not a compression method.** There is a second script in this repo (`storage_reconstruction_test.py`) that splits a weight tensor into a scalar mean and a residual tensor stored in two separate files, then reconstructs the original by adding them back together. This is a **storage/loading split test** — reconstruction is mathematically exact by construction (`mean + (original - mean) = original`), so the R²=1.0 you'll see is expected and is not a compression result. No compression ratio is claimed anywhere in this repo.
+- **Not a new computation method.** The Bayesian selector changes *what gets measured and tracked* during inference, not *how the forward pass computes logits*. The underlying Qwen2-0.5B forward pass is untouched.
+- **Not validated against a baseline yet.** There's no side-by-side comparison here (yet) showing that the selected feature subset actually predicts anything useful about output quality, speed, or attention patterns. Right now this is instrumentation, not a proven technique.
 
-Most work on transformer internals looks at weights (pruning, quantization, low-rank decomposition). This script instead asks: at inference time, does the *activation* stream flowing through each layer have a small, identifiable subset of statistics that matter more than the rest at any given step? If that subset is small and stable, it's a hint (not proof) that there's structure worth digging into — for interpretability, for adaptive compute, or just as a diagnostic tool for understanding what a layer is "paying attention to" numerically.
+If any of that changes as testing continues, this README will be updated to reflect it — the goal is to keep the claims here matched to what's actually been measured.
 
+## A third script: entropy-gated lossless compression
 
+Separate from the two experiments above, this repo also includes `entropy_gated_compression.py`, which applies **standard lossless compression to raw model weight bytes**, using a per-packet entropy check to decide whether compression is worth attempting.
 
-<div style="background: #ffffff; padding: 30px; border-radius: 12px; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; max-width: 650px; margin: 20px auto; border: 1px solid #e8e8e8;">
+### Measured result (single run, self-reported)
 
-<h3 style="margin: 0 0 20px 0; font-size: 18px; font-weight: 600; color: #1a1a1a; text-align: center;">📦 Model Size Comparison</h3>
+```
+Model:      Qwen2-0.5B, model.safetensors
+Original:   988,807,168 bytes  (943 MB)
+Compressed: 793,772,032 bytes  (757 MB)
+Space saved:      19.7%
+Compression ratio: 1.246x
+```
 
-<div style="margin-bottom: 16px;">
-  <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 4px;">
-    <span style="font-size: 14px; color: #333; font-weight: 500;">Qwen2-0.5B</span>
-    <span style="font-size: 14px; color: #555; font-weight: 600;">900 MB</span>
-  </div>
-  <div style="width: 100%; height: 24px; background: #f0f0f0; border-radius: 6px; overflow: hidden;">
-    <div style="width: 100%; height: 100%; background: #e74c3c; border-radius: 6px;"></div>
-  </div>
-</div>
+This is **lossless** — decompression was verified to reproduce every tensor bit-for-bit (SHA256 match on 100% of checked tensors, see "Integrity verification" below). There is no approximation, no quantization, and no information discarded. It's exact round-trip storage in less space, not a compressed *representation* of the model in the sense pruning or quantization would give you.
 
-<div style="margin-bottom: 20px;">
-  <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 4px;">
-    <span style="font-size: 14px; color: #333; font-weight: 500;">Adaptive Sparse (this)</span>
-    <span style="font-size: 14px; color: #555; font-weight: 600;">10 MB</span>
-  </div>
-  <div style="width: 100%; height: 24px; background: #f0f0f0; border-radius: 6px; overflow: hidden;">
-    <div style="width: 1.1%; height: 100%; background: #3498db; border-radius: 6px;"></div>
-  </div>
-</div>
+### How the entropy gate works
 
-<div style="display: flex; gap: 12px; justify-content: center; border-top: 1px solid #eee; padding-top: 16px;">
-  <span style="font-size: 13px; color: #666;"><span style="display: inline-block; width: 12px; height: 12px; background: #e74c3c; border-radius: 3px; vertical-align: middle; margin-right: 4px;"></span> Original</span>
-  <span style="font-size: 13px; color: #666;"><span style="display: inline-block; width: 12px; height: 12px; background: #3498db; border-radius: 3px; vertical-align: middle; margin-right: 4px;"></span> This model</span>
-  <span style="font-size: 13px; font-weight: 600; color: #2c3e50;">90× smaller</span>
-</div>
+For every 8 MB packet of raw tensor bytes, the packet is treated as a stream of 256 possible byte values (0–255) and a frequency histogram is built. From that histogram:
 
-</div>
+```
+p(v) = (count(v) + 1) / (total_bytes + 256)      # Laplace-smoothed frequency
+H    = - Sum p(v) * log2 p(v)                     # Shannon entropy, bits/byte
+```
 
----
+The `+1` in the numerator (and `+256` in the denominator to keep it a valid distribution) is **Laplace smoothing** — a standard fix so that byte values with zero observed occurrences don't produce `log2(0)`. This keeps every probability strictly positive. To be precise about terminology: this smoothing is a Bayesian-flavored technique (it's equivalent to placing a uniform Dirichlet prior over the byte distribution and taking a posterior mean), but the compression *decision* itself is a simple **entropy threshold**, not a Bayesian inference procedure over model parameters. If `H >= 7.90` bits/byte, the packet is treated as high-entropy (close to indistinguishable from random noise) and stored raw — attempting `zlib` on it would cost CPU time for little or no space savings. If `H < 7.90`, the packet goes through `zlib` at compression level 1, and the result is kept only if it's actually smaller than the raw packet.
 
-<div style="background: #ffffff; padding: 30px; border-radius: 12px; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; max-width: 650px; margin: 20px auto; border: 1px solid #e8e8e8;">
+### Why weight tensors compress at all
 
-<h3 style="margin: 0 0 20px 0; font-size: 18px; font-weight: 600; color: #1a1a1a; text-align: center;">🧠 RAM Usage During Inference(testing on google colab)</h3>
+Raw `float16`/`bfloat16` weights are not uniformly random at the byte level, even though they look noise-like in aggregate. Mantissa bits tend to be close to high-entropy, but:
 
-<div style="margin-bottom: 16px;">
-  <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 4px;">
-    <span style="font-size: 14px; color: #333; font-weight: 500;">Qwen2-0.5B (original)</span>
-    <span style="font-size: 14px; color: #555; font-weight: 600;">4 GB</span>
-  </div>
-  <div style="width: 100%; height: 24px; background: #f0f0f0; border-radius: 6px; overflow: hidden;">
-    <div style="width: 100%; height: 100%; background: #e74c3c; border-radius: 6px;"></div>
-  </div>
-</div>
+- Normalization layer weights (`input_layernorm`, `post_attention_layernorm`) and biases occupy a narrower value range than general weight matrices, which tends to concentrate their byte-value histograms and lower measured entropy.
+- Repeated or near-repeated byte patterns across a large tensor give `zlib`'s LZ77-style matching something to exploit, independent of the entropy gate.
 
-<div style="margin-bottom: 20px;">
-  <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 4px;">
-    <span style="font-size: 14px; color: #333; font-weight: 500;">Adaptive Sparse (this)</span>
-    <span style="font-size: 14px; color: #555; font-weight: 600;">~0 GB</span>
-  </div>
-  <div style="width: 100%; height: 24px; background: #f0f0f0; border-radius: 6px; overflow: hidden;">
-    <div style="width: 0%; height: 100%; background: #3498db; border-radius: 6px;"></div>
-  </div>
-</div>
+A 19.7% reduction is consistent with the range generally reported for applying generic lossless compressors (gzip/zlib-class) directly to raw transformer weight bytes — typically single-digit to ~25%, varying by layer type and model. This is **not a novel compression ratio** by research standards; it's roughly what's expected from entropy-aware lossless compression on this kind of data. The contribution here, if there is one, is the packet-level entropy gate deciding *where* to spend compression effort rather than applying `zlib` uniformly — this mainly saves CPU time on incompressible packets rather than improving the ratio itself.
 
-<div style="display: flex; gap: 12px; justify-content: center; border-top: 1px solid #eee; padding-top: 16px;">
-  <span style="font-size: 13px; color: #666;"><span style="display: inline-block; width: 12px; height: 12px; background: #e74c3c; border-radius: 3px; vertical-align: middle; margin-right: 4px;"></span> Original</span>
-  <span style="font-size: 13px; color: #666;"><span style="display: inline-block; width: 12px; height: 12px; background: #3498db; border-radius: 3px; vertical-align: middle; margin-right: 4px;"></span> This model</span>
-  <span style="font-size: 13px; font-weight: 600; color: #2c3e50;">4 GB → 0 GB</span>
-</div>
+### Integrity verification
 
-</div>
+Every packet is SHA256-hashed before compression and checked again after decompression. The proof-of-correctness run showed **100.0000% match across all checked tensors** — every layer's weight, bias, and normalization tensor was recovered bit-identical to the source file. This is the expected outcome for a correctly implemented lossless codec with a working verification step, not a measured quality metric — it confirms the round-trip is correct, the same way a working checksum is expected to match rather than being a research result in itself.
 
+## Why this might be interesting anyway
+
+Most work on transformer internals looks at weights (pruning, quantization, low-rank decomposition). This script instead asks: at inference time, does the *activation* stream flowing through each layer have a small, identifiable subset of statistics that matter more than the rest at any given step? If that subset is small and stable, it's a hint (not proof) that there's structure worth digging into — for interpretability, for adaptive compute, or just as a diagnostic tool for understanding what a layer is "paying attention to" numerically.
+
+That's the honest pitch. No claims beyond it yet.
 
 ## Files
 
 | File | What it does |
-|------|--------------|
+|---|---|
 | `terminal_chat_bayesian.py` | Main experiment. Loads Qwen2-0.5B, hooks every attention/FFN weight's input activations, runs the Bayesian feature selector during generation, prints the fraction of flagged features per response. Requires `bayes_analysis.safetensors` (see below). |
 | `storage_reconstruction_test.py` | Secondary test. Splits weight tensors into `(mean_scalar, residual_tensor)` across a JSON + safetensors file, reconstructs on load. Included for transparency — this is a loading mechanics test, not a result. |
+| `entropy_gated_compression.py` | Lossless compression of raw safetensors weight bytes using an entropy-gated packet scheme. Measured 19.7% size reduction on Qwen2-0.5B with verified bit-exact decompression. See "A third script" section above for details. |
 
 ## Requirements
 
@@ -144,16 +119,38 @@ python storage_reconstruction_test.py
 
 This will strip attention/FFN weights from the loaded model and reconstruct them from the two files, then start a basic chat loop. Reconstruction is exact by construction — see the "What this is not" section above for why.
 
-### 3. compression with Bayes
-
-Show on R2 test 100% score with compression.Compressed approximately 30%
+### 3. Entropy-gated lossless compression
 
 ```bash
-python bayes_compression.py
+python entropy_gated_compression.py
 ```
 
+By default this compresses `/content/Qwen2-0.5B` (edit `source_model_dir` at the bottom of the script for a different path) into a bundle directory, then immediately decompresses it back to verify correctness. Output:
+
+```
+Compression finished.
+Payload:  qwen2_0_5b_bayes_zlib_bundle/payload.bin
+Manifest: qwen2_0_5b_bayes_zlib_bundle/manifest.json
+Packet:   8.0 MB
+Restored to: qwen2_0_5b_restored
+```
 
+To use the compression/decompression functions directly instead of running the full round-trip:
 
+```python
+from entropy_gated_compression import compress_qwen2_safetensors_fast, decompress_qwen2_safetensors_fast
+
+compress_qwen2_safetensors_fast(
+    model_dir="/path/to/Qwen2-0.5B",
+    output_bundle_dir="my_bundle",
+    packet_mb=8,
+)
+
+decompress_qwen2_safetensors_fast(
+    bundle_dir="my_bundle",
+    restored_model_dir="my_restored_model",
+)
+```
 
 ## Code
 
@@ -661,9 +658,7 @@ if __name__ == "__main__":
     run_fast_terminal_chat()
 ```
 
-## Code
-
-### `bayes_compression.py`
+### `entropy_gated_compression.py`
 
 ```python
 from __future__ import annotations
@@ -734,6 +729,10 @@ def _packetize(raw: bytes, packet_size: int) -> List[bytes]:
 
 
 def _bayes_features(raw: bytes) -> Dict[str, float]:
+    # NOTE: named for the Laplace-smoothed histogram below, which is a
+    # Bayesian-flavored smoothing technique (equivalent to a uniform
+    # Dirichlet prior). The compression decision itself is a plain
+    # entropy threshold, not Bayesian inference over model parameters.
     if not raw:
         return {
             "n": 0,
@@ -751,7 +750,7 @@ def _bayes_features(raw: bytes) -> Dict[str, float]:
     n = int(u8.numel())
 
     counts = torch.bincount(u8.to(torch.int64), minlength=256).to(torch.float32)
-    posterior = counts + 1.0
+    posterior = counts + 1.0  # Laplace smoothing (uniform Dirichlet prior, alpha=1)
     total = float(posterior.sum().item())
     probs = posterior / total
 
@@ -1096,7 +1095,9 @@ if __name__ == "__main__":
 
 ## Results
 
-*Placeholder — to be filled in with real numbers from benchmark runs.*
+**Entropy-gated compression — measured, single run:** 19.7% size reduction (943 MB → 757 MB) on Qwen2-0.5B, with 100% bit-exact decompression verified via SHA256. See "A third script" section above for the full breakdown. This has been run once — repeating across other model sizes and dtypes (fp16 vs bf16 vs fp32) would show whether 19.7% generalizes or is specific to this checkpoint.
+
+**Bayesian feature selection during inference — still placeholder.** The activation-tracking experiment (main script) has not yet been benchmarked systematically:
 
 - [ ] Fraction of features flagged per layer, averaged across a test set of prompts
 - [ ] How the flagged fraction changes over the course of a generation (early tokens vs. late tokens)
@@ -1107,4 +1108,8 @@ if __name__ == "__main__":
 
 - Does the flagged feature subset stay stable across different prompts, or does it change drastically session to session?
 - Is there a relationship between which features get flagged and attention patterns in the same layer?
-- Right now `BAYES_EVERY_N = 8` and the deviation threshold (`> 1.0` std) are picked without tuning — sweeping these would show whether the flagged fraction is a real signal or just a threshold artifact.
+- Right now `BAYES_EVERY_N = 8` and the deviation threshold (`> 1.0` std) are picked without tuning — sweeping these would show whether the flagged fraction is a real signal or just a threshold artifact.
+
+## License
+
+Apache 2.0, matching the base model license. This repo builds on [Qwen/Qwen2-0.5B](https://huggingface.co/Qwen/Qwen2-0.5B).