SurjoLabs

Surjo-100m

Older Sister. Higher Standards.

Surjo-100m is a 97.7M-parameter hybrid recurrent language model developed by SurjoLabs. Built on the Gen-3 Surjo architecture, it couples Gated DeltaNet-2 (GDN-2) linear attention with full attention (XSA).

By executing a weight-tied recurrent core across 2 passes, the model reaches an effective computational depth of 34 layers while retaining the memory footprint and latency profile of an 18-layer physical model.

Model Specifications

Attribute	Specification
Parameters	97.7M
Architecture	Hybrid GDN-2 + XSA (Subtractive Attention)
Physical Layers	18 (1 Prelude + 16 Recurrent Trunk + 1 Coda)
Effective Depth	34 Layers (2 Recurrent Passes)
Hidden Size	576
Intermediate Size	1536
Attention Configuration	GQA (9 Query heads, 3 KV heads; Head Dim: 64)
Linear Attention	GDN-2 (K: 64, V: 64, 9 V-Heads)
Context Length	1024 tokens pretraining (2048 max)
Vocabulary	32,768 (Custom BPE, tied embeddings)

Evaluation Results

Benchmark	Score
PIQA	63.87%
ARC-Easy	47.64%
ArithMark-3	38.90%
HellaSwag	35.05%
ARC-Challenge	25.85%
Int Index	18.86

Architecture Details

Surjo Gen-3 combines fast recurrent sequence processing with full-attention boundary layers:

• Prelude (Layer 0): Standard attention with subtractive value projection (XSA) to anchor context.
• Recurrent Core (Layers 1–16, tied 2× passes): Four interleaved groups consisting of 3× GDN-2 linear attention layers followed by 1× XSA full attention layer. GDN-2 uses chunked/fused recurrent kernels via Flash Linear Attention (FLA).
• Coda (Layer 17): Terminal full-attention XSA layer.
• Tied Head: The output projection shares weights with the 32,768-token embedding layer.

Pretraining

• Dataset (20B tokens):
  • 60% FinePhrase
  • 20% DCLM
  • 10% FineMath
  • 10% Cornstack Python
• Training Schedule: 20,000 steps (~1M tokens/step) at a sequence length of 1024 tokens.
• Optimizer: Hybrid Muon (peak lr = 0.02) and AdamW (peak lr = 0.0036).

Usage

Requirements

pip install torch transformers accelerate
pip install -U git+https://github.com/fla-org/flash-linear-attention

Generation Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Surjo-100m"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The secret to scaling small language models is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & License

This project is released under the Apache 2.0 License.

@software{surjo100m2026,
  author = {SurjoLabs},
  title = {Surjo-100m: Hybrid Recurrent Language Model},
  year = {2026},
  url = {https://huggingface.co/SurjoLabs/Surjo-100m}
}