BertBM-300m 🚀 fno+gnp+gnn+aegnn+riemann

Transformers only look at tokens. BertBM-300m looks at structure.

BertBM-300m bolts a whole geometry lab onto ModernBERT: Fourier transforms for the rhythm of language, graph networks for who-talks-to-whom, an event-driven graph that skips the parts that didn't change, and a Poincaré manifold where the latents actually live on a curved surface.

FNO | GeoGNP | GNN | AEGNN | Riemannian

---

⚡ Highlights

	BertBM-300m	ModernBERT-base
Architecture	ModernBERT + 5 operators + GateFuse	attention only
Parameters	305.0M	149M
Operator params	2.07M	0
Layers	28	22
Hidden size	768	768
Heads	12	12
Context	8192	8192
Tokenizer	answerdotai/ModernBERT-base (BPE)	same
Training step	1000	—

🧬 The operator stack

🌊 FNO — Fourier Neural Operator

Spectral mixing in the Fourier domain. Your text has rhythm, periodicity and repetition; FNO gives the model a Fourier lens instead of hoping attention figures it out on its own.

🕸️ GeoGNP — Geometric Neural Process

A variational graph operator over the token field. Think of it as a latent map of the sequence: who talks to whom, and how confident are we about that structure.

🕸️ GNN — Learnable Message Passing

Deterministic graph layers with learned gates and (optional) GAT-style edge attention. The graph learns how to propagate instead of trusting a fixed adjacency.

⚡ AEGNN — Asynchronous Event-Driven GNN

The speedrun build. Nodes that didn't change don't get recomputed. Fire an event, update that neighbourhood, leave the rest of the graph alone. Less compute, same brain.

🌀 Riemannian — Poincaré Tangent Mechanism

Latent states live on a hyperbolic manifold. Mixing happens in the tangent space (fast, stable Euclidean updates) and gets mapped back geodesically. Curvature regularization keeps the embeddings from collapsing.

🚦 GateFuse — the router

Every operator is a specialist; GateFuse is the editor. It learns, per sequence, how much of the language field, the spectral view, the geometric view, the learned graph and the event-driven graph to keep. It starts out almost fully on the plain language field (so the pretrained backbone is never stomped on) and the operators earn their way in during training.

🏋️ Training

Dataset	wikitext103:train
Steps	1000
Schedule	cosine + warmup, gradient clipping 1.0
Optimiser	AdamW
Init	answerdotai/ModernBERT-base backbone
Precision	bf16 / fp32

📈 Results

Metric	Value
Eval NLL	0.5575
Eval perplexity	1.75

Learning curve:

Step	Train loss	Eval NLL	Eval PPL
200	1.5356	0.7060	2.03
400	2.0420	0.7402	2.10
600	1.7560	0.7185	2.05
800	2.0292	0.4811	1.62
1000	2.2500	0.6598	1.93
1000	2.2500	0.5575	1.75

📌 Fair-play note: these numbers come from a limited-budget run (see the table above) — they're not directly comparable with production-scale numbers for ModernBERT, which see orders of magnitude more tokens. Use the curve above to compare operator configurations trained under the same protocol.

🔌 Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "scairesearch/bertbm-300m"
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

top = out.logits[0, inputs["input_ids"].shape[1] - 1].topk(5)
for idx, score in zip(top.indices, top.values):
    print(tokenizer.decode([idx]), f"{float(score):.2f}")

Just want the backbone?

The export also ships a plain ModernBERT-compatible checkpoint (model.safetensors.backbone) with the operators removed — drop it in as a modernbert model.

🧪 Try the other configs

The same repo family ships the isolated operator ablations, so you can see what each piece is actually worth:

Repo	Operators
scairesearch/bertbm-100m	fno + gnp + gnn + aegnn + riemann
scairesearch/bertbm-300m	fno + gnp + gnn + aegnn + riemann
scairesearch/bertbm-gnn-only	gnn
scairesearch/bertbm-aegnn-only	aegnn

📝 Citation

This model is part of the BertBM research line:

@misc{bertbm2026,
  title        = {BertBM: Spectral-Geometric Operator Fusion for Transformer Language Models},
  author       = {Niranjan Hasabnis and SCAI Research Labs},
  year         = {2026},
  eprint       = {arXiv:XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}

📜 License

Apache 2.0