---
license: other
license_name: polyform-noncommercial-license-1.0.0
license_link: https://polyformproject.org/licenses/noncommercial/1.0.0/
pipeline_tag: text-generation
tags:
  - pytorch
  - custom_code
  - quantization
  - leech-lattice
  - leech-lattice-quantization
  - sub-1-bit
  - 0.76-bit
---

# Pollux-1152 10k — Native H24 Leech-Lattice Language Model

**Pollux-1152** is a **404M-parameter** decoder-only causal transformer trained from scratch at **native 0.76-bit quantization resolution** (V = 50,688, n_embd = 1152). By mapping the parameter manifold natively onto the H24 Leech lattice, the **287M-parameter backbone** compresses to just **27.3 MB of active SRAM**.

This checkpoint represents the **structural convergence plateau** at **10,000 steps** (~2.6B tokens). All benchmark scores below are measured directly on the fully serialized **142 MB `.plx` deployment artifact**, confirming that the stated Iso-Memory footprints reflect true Edge AI deployment realities without statistical degradation.

At this peak, Pollux-1152 achieves **69.9% BLiMP** (fluid intelligence), outperforming the continuous Pythia-160M baseline (69.7% BLiMP) at the 4.2B-token Iso-Data boundary. It achieves this syntactic parity despite an **83% reduction in active backbone SRAM** (27.3 MB vs. 162 MB).

This Hugging Face repository is a **weight-hosting layer only**. Pollux is **not** compatible with the Hugging Face `transformers` library. All inference, evaluation, packing, and tokenization logic lives in the official [Pollux GitHub codebase](https://github.com/alavicka/pollux).

---

## A Stateless Reasoning Engine for Zero-Interference RAG

Unlike conventional models that conflate fluid reasoning (syntax) with crystallised memory (factual trivia), Pollux acts as a purely structural engine. The **$C=\sqrt{2}$ Voronoi deep-hole barrier** acts as a geometric gradient coherence filter:
* **Fluid intelligence (structural):** Coherent, recurring gradient signals encoding invariant syntactic rules accumulate directed update momentum, cross the Voronoi barrier, and stabilize into $H_{24}$ kissing-point assignments.
* **Crystallised intelligence (factual):** High-entropy factual gradient signal lacks cross-batch directionality to cross the threshold and is absorbed by the zero-potential null attractor.

The resulting **near or modestly above random chance** performance on factual benchmarks (e.g., 50.3% SciQ vs. random-chance ≈ 25%) is bounded by high-frequency leakage for ubiquitous facts, and is **not a defect** but the defining feature for zero-interference Retrieval-Augmented Generation (RAG). By geometrically constraining parametric encoding, Pollux behaves as a **stateless reasoning engine**: it grounds its output in externally provided context, structurally reducing interference from internally stored parametric associations.

---

## Limitations & Hardware Constraints

The **0.76 bits/param** backbone footprint counts packed 18-bit indices plus one FP16 σ_rms per row. The **reference PyTorch runtime materialises these into dense FP16 weight matrices** at forward time for `cuBLAS` compatibility (~574 MB FP16 for the Pollux-1152 backbone alone, vs. ~27.3 MB packed). This is intentional for research reproducibility; **native LUT gather–accumulate kernels** are required to achieve SRAM-bound latency on edge devices.

---

## Files Included

| File | Description |
|---|---|
| **`pollux_1152_10k.plx`** | **Recommended for inference.** Pollux-1152 packed artifact — **27.3 MB backbone SRAM**, **142 MB total** on disk including INT8 embeddings and LM head. Empirically verified lossless. Load with `generate.py` or `evaluate.py`. |
| **`pollux_1152_10k.pt`** | **Training checkpoint** with continuous pre-weights in optimiser state; observable weights are dynamic Castor H24 projections. Use for inspecting pre-weights or reproducing the packing step. |

*(Note: Neither file can be consumed by `llama.cpp` or standard GGUF loaders without the custom runtime).*

---

## Evaluation Results

Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). Pythia baseline: `EleutherAI/pythia-160m-deduped`.

> *(Note: The Iso-Memory criterion isolates memory-bandwidth footprint under the targeted native LUT runtime. Under the current FP16 reference materialisation, FLOPs per token scale with backbone parameter count and are not matched between Pollux-1152 and Pythia baselines.)*

| Task | Pollux-1152 @ 2.6B | Pythia-160M @ 4.2B (step 2k) | Pythia-160M @ 300B (step 143k) |
|---|---|---|---|
| **BLiMP mean (67 tasks)** | **69.9%** | 69.7% | 73.1% |
| **SciQ** | 50.3% | 58.7% | 72.3% |
| **HellaSwag** | 26.4% | 26.9% | 29.1% |
| **PIQA** | 57.7% | 58.4% | 61.9% |
| **Backbone SRAM** | **27 MB** | 162 MB | 162 MB |
| **Total on-disk footprint** | **142 MB** | 247 MB | 247 MB |

---

## Model Architecture Details
* **Architecture:** 18 layers · n_embd = 1152 · 48 heads · d_head = 24
* **Training corpus:** FineWeb-Edu 10B subset
* **Token budget:** 10,000 optimizer steps (~2.6 billion tokens)
* **Optimizer:** Endogenous kinetic optimiser (`pollux_step`) with no architectural hyperparameters; γ = G24 ≈ 0.065771. Requires one corpus-specific environmental input: `H_floor` — the irreducible cross-entropy convergence floor of the training corpus, measured from a continuous FP16 baseline.

## Licensing & Citation
Released under the **PolyForm Noncommercial License 1.0.0** for academic research. 
Commercial utilization requires a license (pending WIPO Application No. PCT/AT2026/060108 and Austrian Patent Application No. A65086/2026). 

```bibtex
@misc{lavicka2026pollux,
  title   = {0.76 Bits Is All You Need: Vector Ternary Logic via Native H24 Leech-Lattice Quantization in LLMs},
  author  = {Lavicka, Alexander},
  year    = {2026},
  note    = {Preprint. WIPO Patent Application No. PCT/AT2026/060108 and Austrian Patent Application No. A65086/2026},
  url     = {https://papers.ssrn.com/abstract=6973978}

---