scairesearch/bertbm-gnn_only

BertBM 🚀 gnn

Transformers only look at tokens. BertBM looks at structure.

BertBM bolts a whole geometry lab onto ModernBERT: Fourier transforms for the rhythm of language, graph networks for who-talks-to-whom, an event-driven graph that skips the parts that didn't change, and a Poincaré manifold where the latents actually live on a curved surface.

GNN

---

⚡ Highlights

	BertBM	ModernBERT-base
Architecture	ModernBERT + 1 operators + GateFuse	attention only
Parameters	191.3M	149M
Operator params	39.34M	0
Layers	12	22
Hidden size	768	768
Heads	12	12
Context	8192	8192
Tokenizer	answerdotai/ModernBERT-base (BPE)	same
Training step	1000	—

🧬 The operator stack

🕸️ GNN — Learnable Message Passing

Deterministic graph layers with learned gates and (optional) GAT-style edge attention. The graph learns how to propagate instead of trusting a fixed adjacency.

🚦 GateFuse — the router

Every operator is a specialist; GateFuse is the editor. It learns, per sequence, how much of the language field, the spectral view, the geometric view, the learned graph and the event-driven graph to keep. It starts out almost fully on the plain language field (so the pretrained backbone is never stomped on) and the operators earn their way in during training.

🏋️ Training

Dataset	wikitext2:train
Steps	1000
Schedule	cosine + warmup, gradient clipping 1.0
Optimiser	AdamW
Init	answerdotai/ModernBERT-base backbone
Precision	bf16 / fp32

📈 Results

Metric	Value
Eval NLL	0.3230
Eval perplexity	1.38

Learning curve:

Step	Train loss	Eval NLL	Eval PPL
200	2.1665	1.0074	2.74
400	1.8114	0.5682	1.77
600	1.7401	0.4155	1.52
800	1.6931	0.3241	1.38
1000	1.7209	0.3014	1.35
1000	1.7209	0.3230	1.38

📌 Fair-play note: these numbers come from a limited-budget run (see the table above) — they're not directly comparable with production-scale numbers for ModernBERT, which see orders of magnitude more tokens. Use the curve above to compare operator configurations trained under the same protocol.

🔌 Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "scairesearch/bertbm-None"
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

top = out.logits[0, inputs["input_ids"].shape[1] - 1].topk(5)
for idx, score in zip(top.indices, top.values):
    print(tokenizer.decode([idx]), f"{float(score):.2f}")

Just want the backbone?

The export also ships a plain ModernBERT-compatible checkpoint (model.safetensors.backbone) with the operators removed — drop it in as a modernbert model.

🧪 Try the other configs

The same repo family ships the isolated operator ablations, so you can see what each piece is actually worth:

Repo	Operators
scairesearch/bertbm-100m	fno + gnp + gnn + aegnn + riemann
scairesearch/bertbm-300m	fno + gnp + gnn + aegnn + riemann
scairesearch/bertbm-gnn-only	gnn
scairesearch/bertbm-aegnn-only	aegnn

📝 Citation

This model is part of the BertBM research line:

@misc{bertbm2026,
  title        = {BertBM: Spectral-Geometric Operator Fusion for Transformer Language Models},
  author       = {Niranjan Hasabnis and SCAI Research Labs},
  year         = {2026},
  eprint       = {arXiv:XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}

📜 License

Apache 2.0