metadata
library_name: transformers
license: apache-2.0
language:
- en
tags:
- monoid
- causal-lm
- linear-attention
- state-space
- O(1)-inference
- reasoning
pipeline_tag: text-generation
model-index:
- name: Spartacus-1B-Instruct
results: []
Spartacus-1B-Instruct — Causal Monoid Language Model
A 1.3B parameter language model that replaces softmax attention with causal monoid state compression, achieving O(1) time per token and O(1) memory at inference — regardless of sequence length.
Fine-tuned for enhanced reasoning with structured chain-of-thought data.
Monoid Attention — Internal Structure
MonoidAttention (per layer, per head)
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ x_t ∈ R^{2048} │
│ │ │
│ ├──> q_proj ──> RMSNorm ──> q_t ∈ R^{d} (query) │
│ │ │
│ ├──> k_proj ──> RMSNorm ──> SiLU ──> k_t ∈ R^{d} (key, >= 0) │
│ │ │
│ ├──> v_proj ──> v_t ∈ R^{d} (value) │
│ │ │
│ └──> decay_proj ──> sigmoid ──> alpha_t ∈ (0,1) (decay gate) │
│ │
│ k_t (x) v_t │
│ │ ┌──────────────────────────────┐ │
│ │ │ State Matrix S_t ∈ R^{d x d} │ │
│ v │ │ │
│ S_t = alpha_t * S_{t-1} + k_t (x) v_t │ │
│ │ │ "Compressed causal history" │ │
│ │ └──────────────────────────────┘ │
│ v │
│ o_t = q_t . S_t ──> o_proj ──> output │
│ │
└─────────────────────────────────────────────────────────────────────┘
Monoid State Diagonal — O(1) Compression Contour
The state matrix S_t accumulates causal history along its diagonal. Each head maintains an independent d x d state that compresses ALL past tokens into a fixed footprint:
State Matrix S_t ∈ R^{64 x 64} (one per head, 32 heads per layer)
k-dim -->
0 8 16 24 32 40 48 56 63
┌───┬───┬───┬───┬───┬───┬───┬───┐ 0
│***│** │* │ │ │ │ │ │ v-dim
│***│** │* │. │ │ │ │ │ |
├───┼───┼───┼───┼───┼───┼───┼───┤ 8 |
│** │***│** │* │. │ │ │ │ v
│* │***│** │* │. │ │ │ │
├───┼───┼───┼───┼───┼───┼───┼───┤ 16
│* │** │***│** │* │. │ │ │
│. │* │***│** │* │. │ │ │
├───┼───┼───┼───┼───┼───┼───┼───┤ 24
│ │. │** │***│** │* │. │ │
│ │ │* │***│** │* │. │ │
├───┼───┼───┼───┼───┼───┼───┼───┤ 32
│ │ │. │** │***│** │* │. │
│ │ │ │* │***│** │* │. │
├───┼───┼───┼───┼───┼───┼───┼───┤ 40
│ │ │ │. │** │***│** │* │
│ │ │ │ │* │***│** │* │
├───┼───┼───┼───┼───┼───┼───┼───┤ 48
│ │ │ │ │. │** │***│** │
│ │ │ │ │ │* │***│** │
├───┼───┼───┼───┼───┼───┼───┼───┤ 56
│ │ │ │ │ │. │** │***│
│ │ │ │ │ │ │* │***│
└───┴───┴───┴───┴───┴───┴───┴───┘ 63
Legend: *** = high activation (recent tokens, alpha^0 ~ alpha^2)
** = medium (alpha^3 ~ alpha^5)
* = fading (alpha^6 ~ alpha^10)
. = near-zero (alpha^11+, effectively forgotten)
= zero (never reached or fully decayed)
The diagonal band emerges because S_t = SUM_{i<=t} alpha^{t-i} * k_i (x) v_i.
Recent outer products dominate near the diagonal; older ones decay
exponentially via alpha, creating this characteristic contour.
Key Properties
| Property | Transformer (Llama) | Spartacus (Monoid) |
|---|---|---|
| Inference time per token | O(T) -- scans full KV-cache | O(1) -- single state update |
| Inference memory per layer | O(T) -- stores all past K,V | O(1) -- fixed d x d state matrix |
| Sequence length extrapolation | Degrades beyond training length | Unlimited -- state size is constant |
| Causality | Imposed via attention mask | Built into the recurrence |
| Training complexity | O(T^2) | O(T) via parallel prefix scan |
The Monoid Recurrence
Standard attention computes:
o_t = sum_{i<=t} softmax(q_t . k_i) v_i -- requires O(T) KV-cache
Monoid attention compresses the entire causal history into a fixed-size state matrix S_t per head:
S_t = alpha_t * S_{t-1} + k_t (x) v_t -- explicit causal recurrence
o_t = q_t . S_t -- state readout
where alpha_t = sigmoid(decay_proj(x_t)) is a learned, content-dependent decay gate that controls how fast past information fades.
Explicit Causal Modeling
Unlike Transformers where causality is a constraint imposed by masking, Spartacus makes causality a first-class citizen:
- The decay gate
alpha_texplicitly controls per-head information retention at every timestep - The model learns when to forget rather than encoding where tokens are (no positional encoding needed)
- No attention mask required -- causality is structural, not enforced
Design Choices
- SiLU-activated keys:
k = SiLU(k_proj(x))ensures non-negative keys, making the state matrixSpositive semi-definite (PSD). This prevents "feature erasure" where one token's contribution cancels another's - Log-space decay: Working in log-space
log(alpha)avoids numerical underflow whenalpha^T -> 0for long sequences - Learnable h0: The initial state
S_0 = h0is a learnable parameter (zero-initialized), acting as a compressed "system prompt"
Model Details
| Parameter | Value |
|---|---|
| Model | NoesisLab/Spartacus-1B-Instruct |
| Architecture | MonoidForCausalLM |
| Parameters | ~1.34B (tied embeddings) |
| Hidden size | 2048 |
| Intermediate size (MLP) | 8192 |
| Layers | 16 |
| Attention heads | 32 |
| Head dimension | 64 |
| State matrix per head | 64 x 64 = 4096 floats |
| Vocabulary | 128,256 (Llama-3.2 tokenizer) |
| Precision | bfloat16 |
Benchmarks (0-shot)
| Task | Metric | Value | Stderr |
|---|---|---|---|
| ARC-Challenge | acc_norm | 0.3063 | ±0.0135 |
| ARC-Easy | acc | 0.5518 | ±0.0102 |
| HellaSwag | acc_norm | 0.4610 | ±0.0050 |
| PIQA | acc_norm | 0.6915 | ±0.0108 |
| WinoGrande | acc | 0.5225 | ±0.0140 |
Comparison with ~1B Baselines (acc_norm, 0-shot)
| Task | Spartacus-1B-Instruct | TinyLlama-1.1B | Llama 3.2-1B | Mamba-1.4B | RWKV-6-1.6B |
|---|---|---|---|---|---|
| ARC-C | 0.3063 | 0.3268 | ~0.359 | 0.284 | ~0.301 |
| ARC-E | 0.5518 | 0.5547 | ~0.752 | 0.512 | ~0.530 |
| HellaSwag | 0.4610 | 0.4670 | ~0.546 | 0.435 | ~0.450 |
| PIQA | 0.6915 | 0.7210 | ~0.740 | 0.655 | ~0.670 |
| WinoGrande | 0.5225 | 0.5040 | ~0.592 | 0.510 | ~0.515 |
Spartacus achieves competitive performance with sub-quadratic models (Mamba, RWKV) while maintaining O(1) inference time and memory per token. Scores marked with ~ are approximate community-reported values.
Training
Stage 1: General SFT
- Base weights: Transferred from Llama-3.2-1B-Instruct (embeddings, MLP, norms)
- Data: Capybara + smol-smoltalk (general conversation)
- Training: Full-parameter SFT
Stage 2: Reasoning Enhancement
- Data mix: 60% Qwen3-Short-Reasoning + 20% Capybara + 20% smol-smoltalk
- Steps: 2,000
- Learning rate: 2e-5 (cosine schedule, 50 warmup steps)
- Batch size: 8
- Sequence length: 2,048
- Precision: bfloat16
- Optimizer: AdamW (weight decay 0.01, max grad norm 1.0)
The reasoning data uses structured "Thought + Solution" format to strengthen chain-of-thought capabilities while the general data prevents catastrophic forgetting.
Parallel Scan Implementation
The monoid_scan_cuda.py module provides a Triton JIT-compiled parallel prefix scan:
- Forward: Sequential scan along T, parallelized across B x H x D on GPU via Triton kernels
- Backward: Reverse-order adjoint scan computes gradients for both values and log-decay gates
- Fallback: Pure PyTorch sequential scan for CPU/MPS
- Auto-dispatch: CUDA -> Triton kernel, otherwise -> PyTorch fallback
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"NoesisLab/Spartacus-1B-Instruct",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("NoesisLab/Spartacus-1B-Instruct")
messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
File Structure
MonoidForCausalLM.py # Model architecture (MonoidConfig, MonoidAttention, MonoidForCausalLM)
monoid_scan_cuda.py # Triton JIT parallel prefix scan + PyTorch fallback
model.safetensors # Model weights (bfloat16)
config.json # Model configuration
tokenizer.json # Llama-3.2 tokenizer
Citation
@software{spartacus2025,
title={Spartacus: Causal Monoid Language Model with O(1) Inference},
author={NoesisLab},
year={2025},
url={https://huggingface.co/NoesisLab/Spartacus-1B-Instruct},
description={Replaces softmax attention with monoid state compression for constant-time, constant-memory autoregressive generation}
}
License
Apache 2.0