---
license: other
license_name: polyform-noncommercial-1.0.0
license_link: https://github.com/Omibranch/Harmonic/blob/main/LICENSE
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
tags:
  - state-space-models
  - ssm
  - long-context
  - linear-attention-alternative
  - hierarchical
pipeline_tag: text-generation
---

# Hallamonic 1B

TinyLlama-1.1B with all 22 `LlamaAttention` layers replaced by `HarmonicBlock`, a
hierarchical state-space module. This removes TinyLlama's RoPE positional limit
(`max_position_embeddings=2048`): the unmodified model degrades catastrophically past
2K tokens, Hallamonic does not.

This is the final checkpoint (`Phase 2`, full fine-tune, step 5000) from the
**Harmonic** paper: [arxiv.org/abs/2606.24650](https://arxiv.org/abs/2606.24650).
See [Omibranch/Harmonic](https://github.com/Omibranch/Harmonic) for code and the
small-scale (7M-112M param) Harmonic results this architecture is based on.

## Result

Evaluated on three independent held-out benchmarks, none overlapping the
fineweb-edu training data:

| Dataset | Seq len | Hallamonic (bpt) | TinyLlama (bpt) | Δ |
|---|---|---|---|---|
| WikiText-103 | 1,024 | 0.43 | 2.84 | +2.41 |
| WikiText-103 | 8,192 | 0.48 | 10.56 | +10.08 |
| Lambada (clean) | 1,024 | 0.44 | 4.29 | +3.86 |
| Lambada (clean) | 8,192 | 0.45 | 9.89 | +9.44 |
| fineweb-edu held-out | 1,024 | 0.36 | 3.37 | +3.00 |
| fineweb-edu held-out | 8,192 | 0.36 | 10.82 | +10.45 |

TinyLlama's loss explodes past its 2K RoPE limit on every dataset. Hallamonic's loss
at 8K is within 0.02-0.04 bpt of its 1K value across all three.

## Training

- Base: `TinyLlama/TinyLlama-1.1B-Chat-v1.0` (892M params frozen: FFN + embeddings)
- New: 141M params (HarmonicBlock, `d_state=128`, compress ratio `K=4`)
- Phase 1 (SSM warmup): 10K steps, seq=512, batch=4, FFN frozen, lr 3e-4
- Phase 2 (full fine-tune): 5K steps, seq=1024, batch=8, grad-accum=4, lr 3e-5
- Data: fineweb-edu (sample-10BT)
- Cost: ~$15 on a single H100 (Modal), under 3 hours total

## Usage

This is a custom architecture, not a stock `transformers` model class. Load it with
the code in the [Harmonic repo](https://github.com/Omibranch/Harmonic/tree/main/hallamonic):

```bash
git clone https://github.com/Omibranch/Harmonic
cd Harmonic/hallamonic
```

```python
from huggingface_hub import snapshot_download
from model import load_hallamonic

ckpt = snapshot_download("Omibranch/harmonic-checkpoints-phase2-final")
model, tokenizer = load_hallamonic(ckpt, device="cuda")
model.eval()

input_ids = tokenizer.encode("The theory of relativity states that", return_tensors="pt").to("cuda")
out = model.generate(input_ids, max_new_tokens=150, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## Limitations

Single training run, no multi-seed replication. Evaluated on English text only.
Not benchmarked for instruction-following or chat quality — this is a base
language model demonstrating an architectural property (no positional limit at
long context), not a general-purpose assistant.

## License

Dual-licensed: free for noncommercial use (research, study, evaluation) under the
[PolyForm Noncommercial License 1.0.0](https://github.com/Omibranch/Harmonic/blob/main/LICENSE).
Commercial use requires a separate license — see
[LICENSE-COMMERCIAL.md](https://github.com/Omibranch/Harmonic/blob/main/LICENSE-COMMERCIAL.md).

## Citation

```bibtex
@software{harmonic2026,
  title     = {Harmonic: Hierarchical State Space Models},
  author    = {Omibranch and {Harmonic Labs}},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.20381713},
  url       = {https://github.com/Omibranch/Harmonic}
}
```