Safetensors
rambley_v2
custom_code

4d696582-ab0b-44b9-b827-01ea7d71d55b

~50M model pretrained on an ~36B token mix of filtered general web data, math data, and in-context learning elicitation data. To the best of my knowledge, no LLM-generated synthetic data made it into the final corpus, excluding any possible contamination from the source datasets.

I'm not too happy with how this turned out, but...

Benchmarks

Tasks Version Filter n-shot Metric Value Stderr
arc_challenge 1 none 0 acc ↑ 0.2116 ± 0.0119
none 0 acc_norm ↑ 0.2423 ± 0.0125
arc_easy 1 none 0 acc ↑ 0.4949 ± 0.0103
none 0 acc_norm ↑ 0.4436 ± 0.0102
hellaswag 1 none 0 acc ↑ 0.3021 ± 0.0046
none 0 acc_norm ↑ 0.3359 ± 0.0047
piqa 1 none 0 acc ↑ 0.6496 ± 0.0111
none 0 acc_norm ↑ 0.6366 ± 0.0112

Architecture Details

  • Natively trained at BF16 (FP8 overhead caused too much slowdown)
  • SwiGLU MLPs
  • Virtual Head Attention with 32 fake query heads and 8 physical query heads (Ding et al., 2026)
  • Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families; Qiu et al., 2025)
  • Learned attention sinks on every attention head (various, but mainly OpenAI, 2026, LLM-Core Xiaomi, 2026)
  • SWA:SWA:GQA attention pattern, with the final layer forced to GQA
  • 128 sliding window size (Qiao et al., 2026)
  • 2048 native context length
  • HoPE positional embeddings (on both SWA and GQA layers; Chen et al., 2024)
  • Identity Hyperconnections (涮月亮的谪仙人, 2026)
  • Patch-level pretraining for 60% of the pretraining token budget (Shao et al., 2025; Peng et al., 2026)
  • Optimizers:
    • Muon at 0.02 LR for matrices with Hyperball weight norm constraints (Wen et al., 2026)
    • AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball
    • AIR LR scheduler (Yu et al., 2026)

Long-Context

0k

Tasks Version Filter n-shot Metric Value Stderr
babilong_longctx 0 none 2 acc ↑ 0.4864 ± 0.0061
- babilong_qa1 0 none 2 acc ↑ 0.4150 ± 0.0156
- babilong_qa2 0 none 2 acc ↑ 0.3153 ± 0.0147
- babilong_qa3 0 none 2 acc ↑ 0.1582 ± 0.0116
- babilong_qa4 0 none 2 acc ↑ 0.8929 ± 0.0098
- babilong_qa5 0 none 2 acc ↑ 0.6507 ± 0.0151

2k

Tasks Version Filter n-shot Metric Value Stderr
babilong_longctx 0 none 2 acc ↑ 0.1235 ± 0.0046
- babilong_qa1 0 none 2 acc ↑ 0.0620 ± 0.0076
- babilong_qa2 0 none 2 acc ↑ 0.0871 ± 0.0089
- babilong_qa3 0 none 2 acc ↑ 0.1854 ± 0.0123
- babilong_qa4 0 none 2 acc ↑ 0.0641 ± 0.0078
- babilong_qa5 0 none 2 acc ↑ 0.2192 ± 0.0131

8k

Tasks Version Filter n-shot Metric Value Stderr
babilong_longctx 0 none 2 acc ↑ 0.0258 ± 0.0022
- babilong_qa1 0 none 2 acc ↑ 0.0100 ± 0.0031
- babilong_qa2 0 none 2 acc ↑ 0.0320 ± 0.0056
- babilong_qa3 0 none 2 acc ↑ 0.0410 ± 0.0063
- babilong_qa4 0 none 2 acc ↑ 0.0150 ± 0.0038
- babilong_qa5 0 none 2 acc ↑ 0.0310 ± 0.0055

(greater than zero!)

Notes on Data

Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.

Code

Pretraining codebase: https://code.allura.moe/fizz/lobotomy Training data: https://code.allura.moe/fizz/lobotomy/src/commit/6353a73f483823eee6db1750e6a3b2fdb57e53f0/libs/data/lobotomy/data/mixes/rambley_16k/pretrain_v2.txt

Other Benchmarks

====================================================================
  ../exported_hf (49,718,600 params) RESULTS
====================================================================
  Raw continuation accuracy        36.50%
  Length-normalized accuracy       36.40%
  Primary (acc_norm)           36.40%
====================================================================
Downloads last month
30
Safetensors
Model size
49.7M params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train glossia/Rambley-V3-50M

Papers for glossia/Rambley-V3-50M