README.md

# TRM-text

TRM-text is an attention-free language model based on a Tiny Recursive Model (TRM) architecture.

Unlike Transformer-based language models, TRM-text removes self-attention entirely and replaces it with recursive computation built from causal dilated depthwise convolutions, hierarchical latent refinement, and parameter reuse.

The goal of TRM-text is to investigate whether recursive neural computation can provide competitive language modeling performance while dramatically reducing computational cost.

---

# Overview

TRM-text explores a different scaling path from Transformers.

Instead of increasing attention heads and context interactions, TRM-text repeatedly refines hidden representations using a hierarchy of recursive processing blocks.

Key properties:

* Attention-free
* Autoregressive language modeling
* Recursive computation
* Hierarchical latent refinement
* RoPE positional encoding
* Dilated depthwise convolution mixer
* Hugging Face compatible
* safetensors support

---

# Architecture

```text
Tokens
  │
  ▼
Embedding
  │
  ▼
Low-Level TRM
  │
  ▼
Mid-Level TRM
  │
  ▼
High-Level TRM
  │
  ▼
Recursive Feedback
  │
  ▼
LM Head
```

Each recursive block contains:

* RMSNorm
* RoPE
* Causal Dilated Depthwise Convolution
* SwiGLU Feed Forward Network
* Residual Recurrence

No self-attention layers are used.

---

# Model Configuration

Current release:

```text
Parameters: ~15M

dim = 256
hidden_dim = 512

low_steps = 6
mid_steps = 3
high_steps = 2

cycles = 3

kernel_size = 5

low_dilations  = [1,2,4,8]
mid_dilations  = [2,4,8,16]
high_dilations = [4,8,16,32]
```

---

# Compute Efficiency

Relative training cost:

| Architecture | Relative Cost |
| ------------ | ------------: |
| Transformer  |          1200 |
| HRM          |           100 |
| TRM-text     |             1 |

These values represent relative compute requirements under the experimental scaling assumptions used during development.

The objective of TRM-text is to maximize efficiency through:

* parameter reuse
* recursive computation
* hierarchical refinement
* elimination of attention operations

---

# Training

## Base Pretraining

Dataset:

```text
FineWeb Sample-10BT
```

Tokenizer:

```text
GPT-2 BPE
```

Objective:

```text
Causal Language Modeling
```

---

# Instruction Tuning

Dataset:

```text
tatsu-lab/alpaca
```

Format:

```text
### Instruction:
...

### Response:
...
```

---

# Loading

```python
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "summerMC/TRM-text",
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    "summerMC/TRM-text",
    trust_remote_code=True
)
```

---

# Inference

```python
prompt = """
### Instruction:
Explain artificial intelligence in simple terms.

### Response:
"""

inputs = tokenizer(
    prompt,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_k=40
)

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )
)
```

---

# Research Motivation

TRM-text investigates whether recursive neural systems can replace attention mechanisms in language modeling.

Research directions:

* recursive reasoning
* hierarchical computation
* efficient language models
* attention-free architectures
* low-cost scaling laws

---

# Limitations

Current checkpoint is experimental.

Known limitations:

* small parameter count
* limited instruction tuning
* lower capability than modern frontier models
* research-focused implementation
* benchmark coverage still limited

---

# Intended Use

TRM-text is intended for:

* language model research
* efficient architecture experimentation
* recursive computation studies
* attention-free modeling research

Not intended for:

* safety-critical systems
* medical decision making
* legal advice
* financial advice

---

# License

Apache-2.0

---

# Citation

```bibtex
@software{trm_text_2026,
  title={TRM-text: Attention-Free Recursive Language Modeling},
  author={summerMC},
  year={2026},
  url={https://huggingface.co/summerMC/TRM-text}
}
```