docs: expand Z1T-0 model card from official Z1T publication

README.md

@@ -1,10 +1,390 @@
 ---
 library_name: z1t
 pipeline_tag: text-generation
+language: en
+license: apache-2.0
+tags:
+- z1t
+- sparse
+- probabilistic-computing
+- thermodynamic-computing
+- energy-efficient
+- gated-convolutional-attention
+- text-generation
+datasets:
+- openwebtext
 ---
 
 # Z1T-0
 
-Open weight for the first Z1T-0.
+## Summary
+
+Z1T-0 is the first open weight release in the Z1T family from Extropic Corp. Z1T is a family of sparse transformer like language models designed for probabilistic hardware, pairing Extropic Z1 probabilistic chips with FPGA or XPU companions for disaggregated inference.
+
+The design demonstrates substantial modeled energy efficiency gains over GPUs, reported as up to about 139x lower energy per token than an H100 at 10 percent model FLOPs utilization for the reference configuration, while preserving the familiar transformer outline with hardware compatible primitives.
+
+This release provides the Z1T-0 checkpoint for research on sparse scaling, hardware algorithm codesign, and energy efficient decoding. Training and architecture code is available in the official sparse transformers repository.
+
+Official resources:
+
+* Publication: https://extropic.ai/writing/z1t/
+* Training recipe: https://github.com/extropic-ai/sparse-transformers
+* Organization: https://huggingface.co/Extropic-AI
+* Z1 background: https://extropic.ai/writing/from-one-to-one-billion
+* Z1 launch video: https://www.youtube.com/watch?v=ITkoPau3k98
+
+## Model Details
+
+* Developer: Extropic Corp, San Francisco
+* Authors: Guillaume Verdon, Alexander Neagoe, Owen Lockwood, Seth Morton, and Extropic
+* Release date: September 4, 2026
+* Version: Z1T-0, first public Z1T checkpoint
+* Model type: Sparse causal language model with gated convolutional attention
+* Language: English
+* Tokenizer: GPT-2 byte pair encoding, vocabulary size 50257
+* Context length of released checkpoint: 256 tokens
+* Framework: JAX with Equinox, custom `z1t` library
+* Repository files: `config.json`, `load_model.py`, `model.eqx`
+* Checkpoint size: approximately 4.97 GB
+* License: Apache 2.0, matching the official sparse transformers repository. If Extropic publishes a distinct weight license, that notice will take precedence.
+* Status: Research release. Energy and latency figures in the publication are modeled estimates, not full system silicon measurements.
+
+### Released Checkpoint Configuration
+
+Values below are taken directly from the released `config.json`:
+
+* vocab: 50257
+* sequence: 256
+* n_layers: 4
+* n_embed: 12288
+* aft_kind: conv
+* aft_heads: 4
+* aft_ksize: 4
+* dyt_alpha: 0.5
+* linear_fan_in: 4
+* attn_fan_in: null
+* mlp_fan_in: null
+* tanh_linear: true
+* tanh_mlp: true
+* remat: true
+
+Note on reference versus released dimensions: the publication uses a compact operating point with L 4, D 512, T 1024, H 4, kernel 4, and sparse fan in 4 for concrete energy and latency modeling. The released Z1T-0 checkpoint is wider, with n_embed 12288 and sequence 256. Treat scaling plots and hardware estimates as family level evidence, and the configuration above as the exact artifact in this repository.
+
+## Intended Use
+
+### Intended research uses
+
+* Study of sparse transformer scaling under fixed hardware connectivity
+* Hardware algorithm codesign for probabilistic and thermodynamic substrates
+* Energy efficient autoregressive decoding research
+* Reproduction and extension of gated convolutional attention with tanh linear projections
+* Analysis of sparsity, quantization, and sampling precision tradeoffs
+
+### Out of scope uses
+
+* Production deployment without additional evaluation for safety, bias, and robustness
+* Long context prefill on Z1 style hardware, which the authors recommend running on GPUs for now
+* High precision computation without accounting for sampling noise and quantization
+* Use as a general instruction tuned assistant, as this is a base research checkpoint
+* Any use implying measured full chip system guarantees, since published system figures are modeled
+
+## Hardware Context: Z1 plus FPGA
+
+Z1 was designed as a probabilistic graphical model sampler, not originally as a neural network accelerator. Z1T refactors dense transformer layers so compatible sparse operations compile onto the fixed Z1 topology, while remaining dense or irregular operations run on a companion FPGA or XPU.
+
+Z1 characteristics reported by Extropic:
+
+* 269,568 probabilistic bits across 8 cores, with 33,696 pbits per core
+* 2,135,904 hardwired coupling edges
+* Fixed node degree 16
+* Chromatic Gibbs sampling in memory at 50 MHz internal update clock
+* Subthreshold CMOS implementation
+* Sampling energy estimate: 1.3e-14 J per sample
+* Die level target: sampling rate above 50 MHz at below 1 W
+* Data transfer assumption for modeling: 25.6 Gbit per s
+
+The central constraint is connectivity. A program maps efficiently only if its interaction graph is a subgraph of the fixed coupling graph. Dense matrix multiplication therefore is not ported unchanged. It is rewritten as many parallel sparse tanh linear units that respect the degree 16 pattern.
+
+Inference is disaggregated:
+
+* Z1: sampled sparse projections, Dynamic Tanh style normalization fused with following layers, sparse MLP segments
+* FPGA or XPU: embeddings, pooling, residuals, positional handling, orchestration, vocabulary readout and sampling
+
+The publication notes that the companion processor could in principle be a GPU or another accelerator, but all concrete estimates assume the FPGA companion.
+
+## Architecture
+
+Z1T preserves the transformer block outline but replaces dense primitives with Z1 compatible forms.
+
+### Dynamic Tanh normalization
+
+The reference block replaces RMSNorm:
+
+RMSNorm(x) = x / sqrt(mean(x^2) + eps) * gamma + beta
+
+with a scaled Dynamic Tanh form:
+
+DyT(x) = gamma * tanh(alpha * x) + beta
+
+This substitution follows recent work on normalization free transformers and fuses naturally with subsequent sparse linear operations on Z1.
+
+### Tanh linear unit
+
+The core hardware primitive is a probabilistic cell whose visible spin expectation is a tanh of its local field. For visible spin v and 16 neighbors h:
+
+E(v, h) = -(b_v * v + sum(J_j * v * h_j) + sum(b_j * h_j))
+
+Conditioned on h, the expectation is:
+
+E[v | h] = tanh(b_v + sum(J_j * h_j))
+
+Averaged over parallel samples, this implements a sparse matrix vector product fused with tanh. Each output pbit uses its 16 couplings for 4 input values times 4 dyadic pbits.
+
+### Continuous value encoding
+
+Activations and weights are continuous, while Z1 operates on binary stochastic units represented as spins in (-1, 1). Z1T uses a 4 bit dyadic encoding called dy4p:
+
+x approx sum(a_i * s_i), with s in (-1, 1)^4 and a_i = 2^(-i-1)
+
+A weighted activation is therefore encoded as:
+
+tanh(W x + b) approx tanh((W kron a) s + b)
+
+For an input in R^D with 4 spins per value, the weight matrix is encoded directly into interactions over the flattened spin vector. Parallel samples are averaged to estimate the scalar tanh linear output. More samples reduce empirical variance and increase effective precision at the same energy per sample but higher latency if serialized.
+
+### Gated convolutional attention
+
+Instead of dense softmax self attention with Q K transpose scoring of order T squared D, Z1T adapts gated convolutional attention. Sparse 4 fan in projections compute gates Q, K, and V. For token position t and head i:
+
+Y_t^i = tanh(Q_t^i) * (N_t^i / D_t^i)
+
+N_t^i = conv1d(exp(K^i) * V^i, exp(w^i) - 1) + sum(exp(K_j^i) * V_j^i) over j 1 to t
+
+D_t^i = conv1d(exp(K^i), exp(w^i) - 1) + sum(exp(K_j^i)) over j 1 to t
+
+The convolution kernel provides local context while the running sum provides longer range accumulation. Transcendental functions, pooling, and residual adds are assigned to the digital companion.
+
+### Feedforward block
+
+An MLP layer has the form:
+
+tanh(W x + b)
+
+For sparse W, the layer is compiled as dout parallel tanh linear units placed across Z1 cores, with samples streamed between cores. Depth comes from composition of these sampled programs.
+
+### Token path
+
+For one decoding step across a four block stack, the publication traces:
+
+FPGA embedding and input handling to Z1 sampled projections and normalization to FPGA pooling and residuals, repeated per block, followed by vocabulary readout and next token sampling. Z1 sampling and FPGA orchestration dominate modeled latency, while FPGA work dominates modeled energy.
+
+## Training
+
+### Data
+
+* OpenWebText
+* GPT-2 byte pair encoding for the Z1 matched sweep
+* Byte tokenized OpenWebText for the general connectivity sweep with a standard decoder
+* Final dense vocabulary to embedding matmul retained where applicable
+
+### Setup
+
+* JAX research implementation in `research/z1t` and related sparse transformer projects
+* 4 bit weights and 4 incoming edges per output node for the Z1 matched architecture
+* Fixed connectivity patterns rather than fixed percentage sparsity
+* Sparsity therefore grows toward 100 percent as width grows, unlike conventional fixed percentage pruning
+* Activation quantization is required for practical Z1 execution, but Figure 1 results in the publication do not include activation quantization
+
+### Scaling studies
+
+Two complementary directions are reported:
+
+1. Z1 matched gated convolutional architecture sweep from about 2e14 to about 2e18 training FLOPs, with a log log frontier fit extended to about 9.5e19 FLOPs to match GPT-2 small loss near 3.4.
+2. General sparsity study with a standard GPT-2 style decoder using RMS normalization and GELU feedforward blocks, varying connectivity c in 4, 16, 32, 64, 128 plus a dense baseline, sequence length 256, budgets from 3e14 to 1e18 FLOPs.
+
+Key observations:
+
+* Larger compute budgets reach lower validation loss at every connectivity.
+* Iso FLOP curves show U shaped minima that move toward larger models with higher compute.
+* Dense models appear more efficient per FLOP in these early results.
+* Sparse Z1 operations are modeled as vastly more energy efficient per operation, so the full Z1 plus FPGA system can still be orders of magnitude more power efficient at matched performance.
+
+## Evaluation
+
+This is a research checkpoint without a conventional assistant benchmark suite in the release. The publication focuses on scaling behavior rather than task scores.
+
+Reported reference points use the same body parameter convention, excluding the final embedding to vocabulary matmul:
+
+* GPT-2 small: 85 million body parameters
+* GPT-2 XL: 1476 million body parameters
+* Z1T reference modeled block: 11.55 million body parameters in fp16 for H100 timing, with D 512, L 4
+
+Users seeking task evaluation should report dataset, tokenizer handling, sampling settings, number of Z1 samples averaged, quantization, and whether the vocabulary readout is included, since readout dominates energy when included.
+
+## Energy Cost Model
+
+All figures below are modeled estimates based on theoretical Z1 chip energy anchored to experiments with similar pbits in X0. They exclude the dense final logit readout unless stated. They assume enough parallel Z1 chips to place all samples in a model parallel fashion. Required chip count is not modeled. Data movement between Z1 and FPGA is not included.
+
+Reference modeled configuration: L 4, D 512, T 1024, GCA H 4, kernel 4, sparse fan in 4.
+
+Assumptions:
+
+* Z1 sampling: 1.3e-14 J per sample
+* FPGA: 0.2 pJ per matrix multiply op, 3.0 pJ per scalar op, 1.5 W assumed static power
+* H100 reference: 0.177 pJ per floating point op at 32 bit peak, same next token step run densely with no sparsity exploited
+* H100 model FLOPs utilization varied because sparse models often achieve low utilization on GPUs
+
+Modeled per token energy for the Z1T block without final readout:
+
+* Total: 294.52 nJ per token
+* Z1 sampling: 8.74 nJ
+* Included FPGA work: 285.78 nJ
+* Final logit readout on FPGA if included: approximately 136.4 uJ per token
+
+Energy ratios, H100 relative to Z1T without final readout:
+
+<table>
+<thead>
+<tr><th>H100 utilization</th><th>H100 energy per token</th><th>H100 to Z1T system</th><th>H100 to Z1 layers only</th></tr>
+</thead>
+<tbody>
+<tr><td>10 percent</td><td>40.9 uJ</td><td>about 139x</td><td>about 4680x</td></tr>
+<tr><td>50 percent</td><td>8.17 uJ</td><td>about 28x</td><td>about 935x</td></tr>
+<tr><td>100 percent</td><td>4.09 uJ</td><td>about 14x</td><td>about 468x</td></tr>
+</tbody>
+</table>
+
+Interpretation:
+
+* At 10 percent utilization, representative of sparse workloads on GPUs, the modeled full system advantage is largest.
+* At 100 percent utilization, the modeled full system advantage is smaller but still above one order of magnitude.
+* Z1 only layers show much larger ratios because the FPGA accounts for more than 95 percent of modeled system energy.
+* Including the vocabulary readout changes the absolute picture substantially and should always be disclosed.
+
+## Latency Model
+
+Throughput figures assume a single serial stream with no advanced pipelining. Parallel sampling would keep energy similar while reducing time to target precision.
+
+Modeled serial path for one token, excluding final vocabulary logits:
+
+* Total: 58.8 us per token, about 17000 tokens per s
+* FPGA orchestration: 38.0 us, from 38 serial ops at 1 us
+* Z1 sampling: 16.0 us, from 25 layers times 32 samples at 50 MHz
+* Data reading: 3.2 us
+* Data writing: 1.3 us
+* FPGA sample averaging: 0.13 us
+* FPGA pooling, positional, and residual work: 0.16 us
+* Steps modeled: 25 sequential sampling layers, 8 FPGA to Z1 writes, 5 Z1 to FPGA reads
+
+H100 baselines measured by Extropic on 2026-08-12, NVIDIA H100 80GB HBM3, torch 2.7.0, dense equivalent model, D 512, L 4, 11.55M body parameters in fp16, batch 1 sequential decoding, excluding final vocabulary logits:
+
+<table>
+<thead>
+<tr><th>Operating point</th><th>Latency per token</th><th>Tokens per s</th></tr>
+</thead>
+<tbody>
+<tr><td>Z1T conservative serial estimate</td><td>58.8 us</td><td>about 17000</td></tr>
+<tr><td>H100 eager PyTorch</td><td>702 us</td><td>about 1425</td></tr>
+<tr><td>H100 torch.compile</td><td>102 us</td><td>about 9764</td></tr>
+</tbody>
+</table>
+
+Caveats:
+
+* The test model is small relative to H100 scale, so kernel launch overhead is material. Reported H100 utilization for this case is 0.006 percent.
+* Batching on GPU would improve GPU efficiency substantially.
+* The authors recommend this Z1 plus XPU setup for decode rather than prefill.
+
+## Usage
+
+Install the research package from its own project directory. See `research/z1t` in the official repository for exact commands. Typical requirements include Python 3.11 or later, JAX, Equinox, `huggingface_hub`, and the local `z1t` package.
+
+Minimal loading pattern, matching the released `load_model.py`:
+
+```python
+import json
+import equinox as eqx
+import jax
+from huggingface_hub import hf_hub_download
+from z1t.components import Config
+from z1t.model import create_model
+
+repo_id = "Extropic-AI/Z1T-0"
+
+config_path = hf_hub_download(repo_id, "config.json")
+weights_path = hf_hub_download(repo_id, "model.eqx")
+
+with open(config_path) as f:
+    config = Config(**json.load(f))
+
+model = create_model(config, jax.random.key(0))
+model = eqx.tree_deserialise_leaves(weights_path, model)
+```
+
+Practical notes:
+
+* The checkpoint is large. Ensure sufficient local storage and memory before download.
+* Generation behavior depends on sampling configuration, tokenizer postprocessing, number of probabilistic samples averaged, and quantization settings.
+* For fair comparison with the publication, state whether logits and sampling overhead are included.
+* For training or fine tuning, use the official repository configs, experiments, and scripts rather than this inference only loading path.
+
+## Limitations
+
+* Preliminary codesign study. Z1 predates Z1T style ideas, so the mapping is unoptimized by design.
+* System energy and latency are modeled, not end to end measured on deployed Z1 plus FPGA hardware for the full model.
+* Parallel Z1 capacity is assumed. Physical chip count, yield, control overhead, and interconnect energy are not modeled.
+* Activation quantization effects are not included in the main Z1 matched loss frontier.
+* Sparse models require substantially deeper and wider structures to match dense parameter counts at fixed connectivity.
+* OpenWebText has known quality, duplication, bias, and toxicity limitations. No additional safety tuning is documented for this checkpoint.
+* English only validation is reported. Multilingual behavior is unknown.
+* Future Z2 hardware may change the optimal sparsity, precision, and architecture choices.
+
+## Safety, Bias, and Responsibility
+
+No safety evaluation, red teaming, or bias mitigation is documented for Z1T-0. Language models trained on web text can reproduce harmful stereotypes, misinformation, sensitive content, and insecure code patterns.
+
+Do not deploy in high stakes settings without domain evaluation and appropriate guardrails. Document data filtering, sampling temperature, refusal behavior if added, and any post training applied downstream.
+
+## Citation
+
+If you use Z1T-0, cite the official publication page and the training repository. The following BibTeX entry is suggested for this model card:
+
+```bibtex
+@misc{extropic2026z1t,
+  title  = {Z1T: Sparse Transformer Like Models for Probabilistic Hardware},
+  author = {Verdon, Guillaume and Neagoe, Alexander and Lockwood, Owen and Morton, Seth and Extropic},
+  year   = {2026},
+  month  = {September},
+  publisher = {Extropic Corp},
+  howpublished = {Available at https://extropic.ai/writing/z1t/},
+  note   = {Model weights: https://huggingface.co/Extropic-AI/Z1T-0; Code: https://github.com/extropic-ai/sparse-transformers}
+}
+```
+
+## Selected References
+
+* Vaswani et al. Attention Is All You Need. NeurIPS 30, 2017.
+* Hoffmann et al. Training Compute Optimal Large Language Models. NeurIPS 35, 2022.
+* Zhai et al. An Attention Free Transformer. arXiv 2105.14103, 2021.
+* Zhang and Sennrich. Root Mean Square Layer Normalization. NeurIPS 32, 2019.
+* Hendrycks and Gimpel. Gaussian Error Linear Units. arXiv 1606.08415, 2016.
+* Zhu et al. Transformers Without Normalization. CVPR, 2025.
+* Ackley, Hinton, and Sejnowski. A Learning Algorithm for Boltzmann Machines. Cognitive Science 9:147 to 169, 1985.
+* Lockwood et al. A Blueprint for Equilibrium Based Differentiable Continuous Variable Thermodynamic Computing. arXiv 2607.16183, 2026.
+* Jelincic et al. An Efficient Probabilistic Hardware Architecture for Diffusion Like Models. npj Unconventional Computing 3:30, 2026.
+* Verdon et al. A Framework for Stochastic Differentiable Programming. arXiv 2608.01612, 2026.
+* Amico et al. Thermalizing Stochastic Programs. arXiv 2608.01615, 2026.
+* Extropic. From One to One Billion: Torx, Thermalizers, and Z1. 2026.
+* Grattafiori et al. The Llama 3 Herd of Models. arXiv 2407.21783, 2024.
+* Frantar et al. Scaling Laws for Sparsely Connected Foundation Models. arXiv 2309.08520, 2023.
+* Gale et al. Sparse GPU Kernels for Deep Learning. SC20, 2020.
+* Hooker. The Hardware Lottery. Communications of the ACM 64:58 to 65, 2021.
+
+Full bibliography with 23 entries is available in the official publication.
+
+## Contact
+
+* Extropic: contact@extropic.ai
+* Code issues: https://github.com/extropic-ai/sparse-transformers/issues
+* Model discussion: use the Community tab on the Hugging Face repository
 
-[GitHub](https://github.com/extropic-ai/sparse-transformers)