nbKPS/beam-reap-255b-bf16

Beam-1.0-REAP-255B-A23B

Unofficial 50% expert-pruned version of reflection/Beam-1.0-501B-A23B by Reflection, made with REAP (Router-weighted Expert Activation Pruning) from Cerebras:

• Paper: REAP the Experts: Why Pruning Prevails for One-Shot MoE compression (Lasby et al., 2025)
• Code: github.com/CerebrasResearch/reap

This is not an official Reflection release. All credit for the model goes to Reflection; see the original model card for capabilities, intended use, safety evaluations and the usage policy.

	Original	This model
Total parameters	501.27B	254.61B
Active parameters per token	~23B	~23B (unchanged)
Routed experts per MoE layer	256	128
Experts per token (top-k)	8	8
Expert groups / groups per token	8 / 4	8 / 4
Precision	BF16	BF16

Everything else (attention, the 3 dense layers, embeddings, LM head, tokenizer and chat template) is byte-identical to the original.

Method

For each of the 49 MoE layers, every routed expert j gets the REAP saliency

S_j = mean over tokens x routed to j of g_j(x) · ‖E_j(x)‖₂

where g_j(x) is the gate weight the model actually uses for that expert (Beam's normalized top-k sigmoid score) and E_j(x) is the expert's output. Saliency was accumulated with a layer-streaming forward pass of the original model (each layer's real forward pass, with hidden states carried from layer to layer), using about 925k tokens of a balanced calibration mix weighted toward coding and agent use, formatted with Beam's chat template.

Beam routes with grouped top-k (8 groups of 32 experts, 4 groups chosen per token). To keep the routing structure valid, the 16 highest-saliency experts within each group were kept, so num_experts becomes 128 with n_group still 8 (16 experts per group). The router weight and e_score_correction_bias are sliced to the kept experts, preserving their order. num_experts_per_tok, topk_group and routed_scaling_factor are unchanged.

A global top-128 by saliency (which would break the equal group sizes) would have picked nearly the same experts: on average 118.4 of the 128 experts are shared per layer (min 113), and the kept experts carry 58.5% of each layer's total REAP saliency on average, against 58.7% for the global choice. The global choice would also leave groups with anywhere from 9 to 25 experts, which the grouped router cannot represent.

Measured quality

Next-token NLL on 160 held-out samples (138,753 scored tokens, disjoint from the samples used to compute saliency), computed in the same streaming pass with the real final norm, LM head and logit softcap. The pruned model is the same stream with routing restricted to the kept experts:

	NLL	Perplexity
Original (501B)	1.201	3.32
REAP 50% (255B)	1.308	3.70

That is a +0.107 nat increase in NLL (+8.9% relative). The mean per-token KL divergence from the original is 0.27 nats, and the two models' top-1 next-token predictions agree on 84.7% of positions.

These are teacher-forced language-modelling numbers only. No downstream benchmarks or long-generation tests have been run on this checkpoint. Expect some loss of capability, most likely on domains that are under-represented in the saliency data (for example multilingual text or niche knowledge).

Usage

The architecture and tensor layout are the same as the original, with num_experts: 128, so it loads anywhere the original does (transformers>=5.20 BeamForCausalLM, vLLM's Beam implementation). At BF16 it needs about 510 GB of accelerator memory.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "mrfakename/Beam-1.0-REAP-255B-A23B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")

Use the same sampling settings as the original (temperature 0.7, top_p 0.99; these are in generation_config.json).

reap_saliency.json holds the per-layer REAP scores, routing counts and kept expert indices (original numbering).

License

Apache-2.0, the same as the original model. The original LICENSE file is included unchanged.

Citation

@misc{reflection2026introducingbeam,
  author = {{Reflection}},
  title  = {Introducing {Beam}: {Reflection}'s 501B open-weight model},
  year   = {2026},
  month  = oct,
  url    = {https://reflection.ai/blog/introducing-beam}
}

@misc{lasby2025reap,
  title         = {REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author        = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year          = {2025},
  eprint        = {2510.13999},
  archivePrefix = {arXiv}
}