README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/dclm-edu
  - HuggingFaceTB/stack-edu
  - HuggingFaceTB/finemath
  - HuggingFaceTB/smollm-corpus
tags:
  - custom-code
  - causal-lm
  - quadorbit
  - complex-valued
  - recurrent-attention
---

# QuadOrbit-40M

QuadOrbit-40M is the public research checkpoint evaluated in the paper
"QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models."
It adds a small bounded complex recurrent memory branch to each Transformer
block while retaining causal attention and SwiGLU layers.

This is a base next-token language model. It is not instruction tuned and it
should not be presented as a production chatbot.

## Model details

| Property | Value |
|:--|:--|
| Parameters | 39,999,240 |
| Hidden width | 512 |
| Layers | 8 |
| Query heads | 8 |
| Key/value heads | 1 |
| Orbit width | 8 per layer |
| Context length | 512 tokens |
| Vocabulary | 32,768 byte-level BPE tokens |
| Training step | 2,999 |
| Seed | 2026 |

The uploaded weights are the renamed
`orbitoid_v2_stable_complex_orbit_attention_lm` checkpoint. The mathematical
model is unchanged. Only the public name is now QuadOrbit.

## Usage

Install the dependencies:

```bash
pip install "torch>=2.4" "transformers>=5.0" "tokenizers>=0.20" "safetensors>=0.4"
```

Load and generate:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Argo1-OOAS/QuadOrbit-40M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)

inputs = tokenizer("The future of language models", return_tensors="pt").to(device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        use_cache=False,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The repository contains custom model code, so loading requires
`trust_remote_code=True`. Review `modeling_quadorbit.py` before loading remote
code. This portable release uses the clear PyTorch recurrence and does not
require Triton. Generation is faster on a CUDA GPU.

## Training data

The model was trained from scratch on a 209,715,200-token mixture:

| Source | Share |
|:--|--:|
| FineWeb-Edu | 45% |
| DCLM-Edu | 25% |
| Stack-Edu | 15% |
| FineMath-4+ | 6% |
| InfiWebMath-4+ | 5% |
| Cosmopedia v2 | 4% |

Training used 3,000 updates with 262,144 sampled tokens per update, or about
786.4 million token presentations. Because the prepared corpus contains 209.7
million tokens, examples were sampled more than once during training.

## Evaluation

The full validation evaluation used 104,448 consecutive target tokens in FP32.

| Model | Validation loss | Perplexity |
|:--|--:|--:|
| Parameter-matched Transformer | 3.7383 | 42.02 |
| QuadOrbit-40M | 3.7398 | 42.09 |

The difference is small and comes from one training seed. It does not establish
an improvement over the Transformer at this scale. See `quadorbit.pdf` for the
full method, smaller-model result, proof, and limitations.

## Intended use

This release is intended for architecture research, reproducibility, analysis,
and small-scale experimentation. Users may study the recurrence, reproduce the
reported evaluation, or continue training under the license terms.

## Limitations

* It is a small base model and is not suitable for reliable factual assistance.
* It has not been safety tuned or instruction tuned.
* It may generate incorrect, biased, repetitive, or offensive text.
* The context length is limited to 512 tokens.
* The evaluation uses one corpus and one seed.
* The portable recurrence does not implement a generation KV cache.

Do not use this model for medical, legal, financial, safety-critical, or other
high-impact decisions.

## Licenses

The model weights and repository code are licensed under the Apache License 2.0.
The included research paper is licensed under CC BY 4.0. Dataset content is not
redistributed here and remains under the terms of its original providers.

## Citation

```bibtex
@misc{argo1ooas2026quadorbit,
  title  = {QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models},
  author = {Argo1-OOAS},
  year   = {2026},
  url    = {https://huggingface.co/Argo1-OOAS/QuadOrbit}
}
```