| --- |
| license: other |
| license_name: polyform-noncommercial-license-1.0.0 |
| license_link: https://polyformproject.org/licenses/noncommercial/1.0.0/ |
| pipeline_tag: text-generation |
| tags: |
| - pytorch |
| - custom_code |
| - quantization |
| - leech-lattice |
| - leech-lattice-quantization |
| - sub-1-bit |
| - 0.76-bit |
| --- |
| |
| # Pollux-1152 10k — Native H24 Leech-Lattice Language Model |
|
|
| **Pollux-1152** is a **404M-parameter** decoder-only causal transformer trained from scratch at **native 0.76-bit quantization resolution** (V = 50,688, n_embd = 1152). By mapping the parameter manifold natively onto the H24 Leech lattice, the **287M-parameter backbone** compresses to just **27.3 MB of active SRAM**. |
| |
| This checkpoint represents the **structural convergence plateau** at **10,000 steps** (~2.6B tokens). All benchmark scores below are measured directly on the fully serialized **142 MB `.plx` deployment artifact**, confirming that the stated Iso-Memory footprints reflect true Edge AI deployment realities without statistical degradation. |
| |
| At this peak, Pollux-1152 achieves **69.9% BLiMP** (fluid intelligence), outperforming the continuous Pythia-160M baseline (69.7% BLiMP) at the 4.2B-token Iso-Data boundary. It achieves this syntactic parity despite an **83% reduction in active backbone SRAM** (27.3 MB vs. 162 MB). |
| |
| This Hugging Face repository is a **weight-hosting layer only**. Pollux is **not** compatible with the Hugging Face `transformers` library. All inference, evaluation, packing, and tokenization logic lives in the official [Pollux GitHub codebase](https://github.com/alavicka/pollux). |
| |
| --- |
| |
| ## A Stateless Reasoning Engine for Zero-Interference RAG |
| |
| Unlike conventional models that conflate fluid reasoning (syntax) with crystallised memory (factual trivia), Pollux acts as a purely structural engine. The **$C=\sqrt{2}$ Voronoi deep-hole barrier** acts as a geometric gradient coherence filter: |
| * **Fluid intelligence (structural):** Coherent, recurring gradient signals encoding invariant syntactic rules accumulate directed update momentum, cross the Voronoi barrier, and stabilize into $H_{24}$ kissing-point assignments. |
| * **Crystallised intelligence (factual):** High-entropy factual gradient signal lacks cross-batch directionality to cross the threshold and is absorbed by the zero-potential null attractor. |
|
|
| The resulting **near or modestly above random chance** performance on factual benchmarks (e.g., 50.3% SciQ vs. random-chance ≈ 25%) is bounded by high-frequency leakage for ubiquitous facts, and is **not a defect** but the defining feature for zero-interference Retrieval-Augmented Generation (RAG). By geometrically constraining parametric encoding, Pollux behaves as a **stateless reasoning engine**: it grounds its output in externally provided context, structurally reducing interference from internally stored parametric associations. |
|
|
| --- |
|
|
| ## Limitations & Hardware Constraints |
|
|
| The **0.76 bits/param** backbone footprint counts packed 18-bit indices plus one FP16 σ_rms per row. The **reference PyTorch runtime materialises these into dense FP16 weight matrices** at forward time for `cuBLAS` compatibility (~574 MB FP16 for the Pollux-1152 backbone alone, vs. ~27.3 MB packed). This is intentional for research reproducibility; **native LUT gather–accumulate kernels** are required to achieve SRAM-bound latency on edge devices. |
| |
| --- |
| |
| ## Files Included |
| |
| | File | Description | |
| |---|---| |
| | **`pollux_1152_10k.plx`** | **Recommended for inference.** Pollux-1152 packed artifact — **27.3 MB backbone SRAM**, **142 MB total** on disk including INT8 embeddings and LM head. Empirically verified lossless. Load with `generate.py` or `evaluate.py`. | |
| | **`pollux_1152_10k.pt`** | **Training checkpoint** with continuous pre-weights in optimiser state; observable weights are dynamic Castor H24 projections. Use for inspecting pre-weights or reproducing the packing step. | |
| |
| *(Note: Neither file can be consumed by `llama.cpp` or standard GGUF loaders without the custom runtime).* |
| |
| --- |
| |
| ## Evaluation Results |
| |
| Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). Pythia baseline: `EleutherAI/pythia-160m-deduped`. |
| |
| > *(Note: The Iso-Memory criterion isolates memory-bandwidth footprint under the targeted native LUT runtime. Under the current FP16 reference materialisation, FLOPs per token scale with backbone parameter count and are not matched between Pollux-1152 and Pythia baselines.)* |
| |
| | Task | Pollux-1152 @ 2.6B | Pythia-160M @ 4.2B (step 2k) | Pythia-160M @ 300B (step 143k) | |
| |---|---|---|---| |
| | **BLiMP mean (67 tasks)** | **69.9%** | 69.7% | 73.1% | |
| | **SciQ** | 50.3% | 58.7% | 72.3% | |
| | **HellaSwag** | 26.4% | 26.9% | 29.1% | |
| | **PIQA** | 57.7% | 58.4% | 61.9% | |
| | **Backbone SRAM** | **27 MB** | 162 MB | 162 MB | |
| | **Total on-disk footprint** | **142 MB** | 247 MB | 247 MB | |
| |
| --- |
| |
| ## Model Architecture Details |
| * **Architecture:** 18 layers · n_embd = 1152 · 48 heads · d_head = 24 |
| * **Training corpus:** FineWeb-Edu 10B subset |
| * **Token budget:** 10,000 optimizer steps (~2.6 billion tokens) |
| * **Optimizer:** Endogenous kinetic optimiser (`pollux_step`) with no architectural hyperparameters; γ = G24 ≈ 0.065771. Requires one corpus-specific environmental input: `H_floor` — the irreducible cross-entropy convergence floor of the training corpus, measured from a continuous FP16 baseline. |
| |
| ## Licensing & Citation |
| Released under the **PolyForm Noncommercial License 1.0.0** for academic research. |
| Commercial utilization requires a license (pending WIPO Application No. PCT/AT2026/060108 and Austrian Patent Application No. A65086/2026). |
| |
| ```bibtex |
| @misc{lavicka2026pollux, |
| title = {0.76 Bits Is All You Need: Vector Ternary Logic via Native H24 Leech-Lattice Quantization in LLMs}, |
| author = {Lavicka, Alexander}, |
| year = {2026}, |
| note = {Preprint. WIPO Patent Application No. PCT/AT2026/060108 and Austrian Patent Application No. A65086/2026}, |
| url = {https://papers.ssrn.com/abstract=6973978} |
| |
| --- |
| |
