Mamba-50M

A ~50M-parameter Mamba (selective state-space) causal language model, pretrained from scratch on English Wikipedia. Mamba replaces the attention mechanism of a Transformer with a selective state-space layer, giving linear-time sequence processing instead of the quadratic cost of self-attention.

This is a base model: pretrained only. It has not been fine-tuned, instruction-tuned, RLHF'd, or aligned in any way. It is a raw next-token predictor intended for research.

Model details

Architecture	Mamba (selective SSM)
Parameters	~50M
Context length	512 tokens
Tokenizer	GPT-NeoX-20B (BPE, ~50k vocab) — the same tokenizer used by the original state-spaces/mamba-* models
Language	English
License	Apache-2.0

Limitations

• Pretrained only. No fine-tuning, instruction tuning, or alignment. It does not follow instructions and has no safety filtering; it simply continues text.
• Small. At ~50M parameters it has limited fluency and reasoning; expect frequent hallucination and repetition.
• English only. Trained solely on English Wikipedia; other languages are out of distribution.
• Domain-narrow. Only Wikipedia was used as training data.

Training data

Pretrained on the English subset of Wikipedia: over 3 million articles.

Training procedure

Hyperparameter	Value
Learning rate	5e-4
Sequence length	512
Batch size	64
Tokenizer	GPT-NeoX-20B (EleutherAI/gpt-neox-20b)
Optimizer	AdamW
LR schedule / warmup	constant / 10000
Total tokens seen	~ 2.5-2.9B
Hardware	2x Nvidia Quadro RTX 6000 24GB

Evaluation

Evaluated on a held-out set of 10,000 Wikipedia articles that were not seen during training. The training and evaluation loss curves are shown below.

Citation

If you use this model, please cite the Mamba paper:

@article{gu2023mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}