v2: pin py3.12 + add inference tab on TinyLlama Q4_K_M

README.md

@@ -1,48 +1,33 @@
 ---
-title: TurboQuant Visualizer
+title: TurboCPP Demo
 emoji: 🌀
 colorFrom: indigo
 colorTo: blue
 sdk: gradio
-sdk_version: 4.44.0
+sdk_version: 4.44.1
 app_file: app.py
 pinned: false
 license: mit
-short_description: Visualize how Hadamard rotation Gaussianizes LLM weights
+python_version: "3.12"
+short_description: Live llama.cpp + Hadamard rotation visualizer (TurboQuant)
 ---
 
-# TurboQuant Visualizer
+# turbocpp — llama.cpp + TurboQuant
 
-Interactive demo of the offline weight-rotation step at the heart of
-[turbocpp](https://github.com/Ary5272/turbocpp). Drag the sliders to see
-how a Walsh-Hadamard transform reshapes a heavy-tailed LLM weight
-distribution into a near-Gaussian one — which is the exact distribution
-shape that Q4 / Q4_K / Q3 quantization handles best.
+Live demo of [github.com/Ary5272/turbocpp](https://github.com/Ary5272/turbocpp).
 
-## What you're looking at
+Two tabs:
 
-| panel | what |
-|---|---|
-| left   | raw synthetic weight (Gaussian bulk + ~5σ outliers — typical of LLaMA-style weights) |
-| middle | same weight after block-Hadamard rotation; bulk is preserved, tails collapse into the Gaussian |
-| right  | per-block max-abs distributions overlaid — the rotation makes each block's max-abs smaller and tighter, which is exactly what controls Q4 rounding error |
+1. **Run inference** — TinyLlama-1.1B-Chat (Q4_K_M) loaded via
+   `llama-cpp-python` and run on this Space's CPU. Type a prompt, get
+   tokens, see tok/s.
+2. **TurboQuant math viz** — interactive sliders showing how the
+   Hadamard rotation Gaussianizes per-block weight distributions and
+   reduces the per-block max-abs that drives Q4 / Q4_K rounding error.
 
-The text panel reports MSE at Q4 / Q3 / Q2 with and without rotation,
-plus the implied "drop a tier and run faster" speed estimate.
+## Notes
 
-## How to deploy this Space
-
-1. Create a new Space at https://huggingface.co/new-space (Gradio SDK).
-2. Copy `app.py`, `requirements.txt`, and this `README.md` into the
-   Space's repo.
-3. Also copy `turboquant/hadamard.py` and `turboquant/bench.py` (or run
-   `pip install git+https://github.com/Ary5272/turbocpp` from inside
-   the Space's `requirements.txt`).
-4. Push — HF builds the image automatically.
-
-## Local
-
-```bash
-pip install -e ".[demo]"
-python -m space.app
-```
+- Python pinned to 3.12 (3.13 dropped stdlib `audioop` which Gradio's
+  pydub dep needs).
+- First call cold-starts the model (~668 MB GGUF download). Subsequent
+  calls are fast.

app.py

@@ -1,49 +1,118 @@
-"""TurboQuant Visualizer — HuggingFace Space (Gradio).
+"""TurboCPP — llama.cpp + TurboQuant — HuggingFace Space.
 
-Interactive demo showing what the Hadamard rotation actually does to a
-weight tensor's quantization-error distribution. Three side-by-side
-plots:
-
-   1. raw weight histogram (heavy tail)
-   2. rotated weight histogram (Gaussianized)
-   3. per-block max-abs before vs after rotation
-
-Plus a numeric summary: MSE at Q4 / Q3 / Q2, with and without rotation,
-and the implied "drop a tier and run faster" speed-up estimate.
+Two tabs:
+  1. Run inference: live llama.cpp on TinyLlama-1.1B-Chat-Q4_K_M.
+  2. TurboQuant math viz: shows what the offline rotation does to the
+     weight distribution that quantization sees.
 """
+from __future__ import annotations
+
 import io
+import os
+import time
 
 import gradio as gr
 import matplotlib
-
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
 import numpy as np
 import torch
+from PIL import Image
 
-from bench import heavy_tailed_weight, measure
 from hadamard import block_hadamard_inplace
+from bench import heavy_tailed_weight, measure
 
 
-def _plot(W_raw: torch.Tensor, W_rot: torch.Tensor, block: int) -> "PIL.Image":
+# ---------------------------------------------------------------------------
+# Inference tab — lazy-load llama-cpp-python + a small GGUF.
+# ---------------------------------------------------------------------------
+_llm = None
+_load_error = None
+
+MODEL_REPO = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
+MODEL_FILE = "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
+
+
+def _ensure_llm():
+    global _llm, _load_error
+    if _llm is not None:
+        return _llm, None
+    if _load_error is not None:
+        return None, _load_error
+    try:
+        from huggingface_hub import hf_hub_download
+        from llama_cpp import Llama
+        path = hf_hub_download(
+            repo_id=MODEL_REPO,
+            filename=MODEL_FILE,
+            cache_dir=os.environ.get("HF_HOME", "/tmp/hf"),
+        )
+        _llm = Llama(
+            model_path=path,
+            n_ctx=2048,
+            n_threads=int(os.environ.get("LLAMA_THREADS", "2")),
+            n_batch=64,
+            verbose=False,
+        )
+        return _llm, None
+    except Exception as e:
+        _load_error = f"failed to load model: {e}"
+        return None, _load_error
+
+
+def chat(prompt: str, max_tokens: int, temperature: float):
+    llm, err = _ensure_llm()
+    if err:
+        return f"Loading error: {err}", ""
+    formatted = (
+        f"<|system|>\nYou are a concise assistant.</s>\n"
+        f"<|user|>\n{prompt}</s>\n"
+        f"<|assistant|>\n"
+    )
+    t0 = time.time()
+    out = llm(
+        formatted,
+        max_tokens=int(max_tokens),
+        temperature=float(temperature),
+        top_p=0.95,
+        stop=["</s>", "<|user|>"],
+        echo=False,
+    )
+    dt = time.time() - t0
+    text = out["choices"][0]["text"].strip()
+    n = out["usage"]["completion_tokens"]
+    tps = n / max(dt, 1e-3)
+    stats = (
+        f"**{n} tokens** in **{dt:.2f}s** -> **{tps:.1f} tok/s**\n\n"
+        f"This is baseline Q4_K_M. With TurboQuant rotation you can drop "
+        f"to Q3_K_M at similar quality and pick up ~25% more tok/s on the "
+        f"same hardware (math in the next tab)."
+    )
+    return text or "(empty)", stats
+
+
+# ---------------------------------------------------------------------------
+# Visualization tab
+# ---------------------------------------------------------------------------
+def _plot(W_raw, W_rot, block):
     fig, axes = plt.subplots(1, 3, figsize=(13, 3.6))
     raw = W_raw.flatten().numpy()
     rot = W_rot.flatten().numpy()
 
     bins = np.linspace(-0.5, 0.5, 121)
     axes[0].hist(raw, bins=bins, color="#888", alpha=0.85)
-    axes[0].set_title("Raw weights — heavy-tailed")
+    axes[0].set_title("raw weights - heavy-tailed")
     axes[0].set_xlim(-0.5, 0.5); axes[0].set_yscale("log")
 
     axes[1].hist(rot, bins=bins, color="#3B82F6", alpha=0.85)
-    axes[1].set_title("After block-Hadamard — Gaussianized")
+    axes[1].set_title("after block-Hadamard - Gaussianized")
     axes[1].set_xlim(-0.5, 0.5); axes[1].set_yscale("log")
 
     raw_blkmax = W_raw.reshape(-1, block).abs().amax(dim=-1).numpy()
     rot_blkmax = W_rot.reshape(-1, block).abs().amax(dim=-1).numpy()
     axes[2].hist(raw_blkmax, bins=40, alpha=0.6, label="raw",     color="#888")
     axes[2].hist(rot_blkmax, bins=40, alpha=0.6, label="rotated", color="#3B82F6")
-    axes[2].set_title(f"per-{block} block max|w|  (drives Q4 quant step)")
+    axes[2].set_title(f"per-{block} block max|w|")
     axes[2].legend()
 
     fig.tight_layout()
@@ -51,70 +120,90 @@ def _plot(W_raw: torch.Tensor, W_rot: torch.Tensor, block: int) -> "PIL.Image":
     fig.savefig(buf, format="png", dpi=110)
     plt.close(fig)
     buf.seek(0)
-    from PIL import Image
     return Image.open(buf)
 
 
-def run(rows: int, cols: int, block: int, seed: int):
-    W = heavy_tailed_weight(n_rows=int(rows), n_cols=int(cols), seed=int(seed))
+def visualize(rows, cols, block, seed):
+    W = heavy_tailed_weight(int(rows), int(cols), int(seed))
     W_rot = W.clone().double()
     block_hadamard_inplace(W_rot, axis=-1, block=int(block))
 
-    # Quantization MSE
-    bench_lines = []
+    lines = []
     for bits in (4, 3, 2):
         s_base = measure(W, bits=bits, rotated=False, block=int(block))
         s_rot  = measure(W, bits=bits, rotated=True,  block=int(block))
-        bench_lines.append(
-            f"  Q{bits}      raw MSE = {s_base.mse:.3e}    "
+        lines.append(
+            f"Q{bits}      raw MSE = {s_base.mse:.3e}    "
             f"TQ MSE = {s_rot.mse:.3e}    "
-            f"× {s_base.mse/max(s_rot.mse,1e-30):.1f} better"
+            f"x {s_base.mse/max(s_rot.mse,1e-30):.1f} better"
         )
 
-    # MSE-matched speed estimate.
-    base_q4 = measure(W, bits=4, rotated=False, block=int(block)).mse
-    speed_msg = "needs a deeper drop"
-    for bits in (3, 2):
-        s = measure(W, bits=bits, rotated=True, block=int(block))
-        if s.mse <= base_q4:
-            ratio = 4.625 / (bits + 1.0)
-            speed_msg = (f"TQ-Q{bits} matches baseline-Q4 quality at "
-                         f"~{ratio:.2f}× less memory bandwidth → faster decode")
-            break
-
     summary = (
-        f"weight shape = {rows}×{cols}, block_size = {block}\n"
-        f"per-block max|w|  raw mean  = {W.reshape(-1, int(block)).abs().amax(dim=-1).mean():.3f}\n"
-        f"per-block max|w|  rot mean  = {W_rot.reshape(-1, int(block)).abs().amax(dim=-1).mean():.3f}\n\n"
-        + "\n".join(bench_lines)
-        + "\n\nSpeed: " + speed_msg
+        f"weight = {rows} x {cols}, block = {block}\n"
+        f"per-block max|w|   raw mean = "
+        f"{W.reshape(-1, int(block)).abs().amax(dim=-1).mean():.3f}\n"
+        f"per-block max|w|   rot mean = "
+        f"{W_rot.reshape(-1, int(block)).abs().amax(dim=-1).mean():.3f}\n\n"
+        + "\n".join(lines)
     )
-
     return _plot(W, W_rot, int(block)), summary
 
 
-demo = gr.Interface(
-    fn=run,
-    title="TurboQuant — Hadamard Rotation Visualizer",
-    description=(
-        "Drag the sliders to see how Walsh-Hadamard rotation reshapes a "
-        "heavy-tailed LLM-style weight distribution. The rotation is "
-        "orthogonal so model fp32 output is unchanged — but quantization "
-        "error drops 3-5× because every block sees a near-Gaussian input. "
-        "[github.com/Ary5272/turbocpp](https://github.com/Ary5272/turbocpp)"
-    ),
-    inputs=[
-        gr.Slider(64,  4096, value=1024, step=64,  label="rows"),
-        gr.Slider(64,  4096, value=4096, step=64,  label="cols"),
-        gr.Slider(32,   256, value=128,  step=32,  label="Hadamard block size"),
-        gr.Slider(0,   1000, value=0,    step=1,   label="seed"),
-    ],
-    outputs=[
-        gr.Image(type="pil", label="distributions"),
-        gr.Textbox(label="quant-error report", lines=10),
-    ],
-    examples=[[1024, 4096, 128, 0], [4096, 4096, 64, 7]],
-)
+# ---------------------------------------------------------------------------
+# UI
+# ---------------------------------------------------------------------------
+with gr.Blocks(title="turbocpp - llama.cpp + TurboQuant",
+               theme=gr.themes.Soft()) as demo:
+    gr.Markdown("# turbocpp - llama.cpp + TurboQuant")
+    gr.Markdown(
+        "Live llama.cpp running TinyLlama-1.1B-Chat (Q4_K_M) plus an "
+        "interactive math visualizer for the Hadamard-rotation "
+        "preprocessor. "
+        "Code: [github.com/Ary5272/turbocpp](https://github.com/Ary5272/turbocpp)"
+    )
+
+    with gr.Tab("Run inference"):
+        gr.Markdown(
+            "Live llama.cpp inference on TinyLlama-1.1B-Chat at Q4_K_M, "
+            "loaded via `llama-cpp-python` on this Space's CPU."
+        )
+        prompt_in = gr.Textbox(
+            value="Explain quantization in one paragraph.",
+            label="prompt", lines=3,
+        )
+        with gr.Row():
+            max_t = gr.Slider(8, 256, value=96, step=8, label="max new tokens")
+            temp  = gr.Slider(0.0, 1.5, value=0.7, step=0.1, label="temperature")
+        run_btn = gr.Button("generate", variant="primary")
+        out_box = gr.Textbox(label="output", lines=10)
+        stats_box = gr.Markdown()
+        run_btn.click(chat, [prompt_in, max_t, temp], [out_box, stats_box])
+
+    with gr.Tab("TurboQuant math viz"):
+        gr.Markdown(
+            "Drag the sliders to see how a Walsh-Hadamard rotation "
+            "reshapes a synthetic LLM-style weight distribution. The "
+            "rotation is orthogonal - fp32 model output is unchanged - "
+            "but per-block max-abs drops 3-5x -> much smaller Q4 / Q4_K "
+            "rounding error."
+        )
+        with gr.Row():
+            rows  = gr.Slider(64, 4096, value=1024, step=64, label="rows")
+            cols  = gr.Slider(64, 4096, value=4096, step=64, label="cols")
+            block = gr.Slider(32,  256, value=128,  step=32, label="block size")
+            seed  = gr.Slider(0,  1000, value=0,    step=1,  label="seed")
+        viz_btn = gr.Button("visualize")
+        img_out = gr.Image(type="pil", label="distributions")
+        rep_out = gr.Textbox(label="quant-error report", lines=8)
+        viz_btn.click(visualize, [rows, cols, block, seed], [img_out, rep_out])
+        demo.load(visualize, [rows, cols, block, seed], [img_out, rep_out])
+
+    gr.Markdown(
+        "---\n"
+        "Want the actual A/B speed numbers? Clone the repo and run "
+        "`scripts/bench_e2e.sh /path/to/HF/Llama-3-8B`, or pull the Docker "
+        "image: `docker pull ghcr.io/ary5272/turbocpp:turboquant`."
+    )
 
 
 if __name__ == "__main__":

requirements.txt

@@ -1,5 +1,8 @@
-gradio>=4.40
+gradio==4.44.1
 matplotlib>=3.7
 numpy>=1.24
 torch>=2.0
 pillow>=10.0
+huggingface_hub>=0.24
+llama-cpp-python>=0.3.2
+audioop-lts; python_version >= "3.13"