BananaMind/BananaMind-2-Micro

BananaMind 2 Micro

BananaMind 2 Micro is a 2,933,193-parameter base causal language model trained from scratch on 74,998,349,824 tokens. It is a pretrained completion model, not an instruction-tuned chat model.

Highlights

• 2.93M parameters with tied input and output embeddings
• 4,096-token context window
• 2,048-token custom tokenizer
• GQA, pre-RoPE QK normalization, SwiGLU, and RMSNorm
• XSA refresh path with detached attention readout, causal depthwise gating, and token-embedding reinjection
• Trained with stock PyTorch Muon for matrix parameters and AdamW for embeddings and one-dimensional parameters

Benchmarks

All results below use the final checkpoint. BananaMind Base Bench 1.1 and ArithMark use raw continuation likelihood rather than generated answers.

Benchmark	Metric	Score
BananaMind Base Bench 1.1	Overall Elo	874
BananaMind Base Bench 1.1	Raw accuracy	34.57% (121/350)
BananaMind Base Bench 1.1	Weighted accuracy	32.39%
ArithMark 2.0	Accuracy	24.92% (623/2,500)
ArithMark 3.0	acc_norm	34.00% (340/1,000)
ArithMark 3.0	Raw accuracy	33.70% (337/1,000)

Base Bench category Elo

Category	Elo
Language completion	898
Commonsense	860
World knowledge	899
Context tracking	815
Quantitative	837
Logical reasoning	911
Code completion	927

Small-model comparison

ARC, PIQA, HellaSwag, and ArithMark 3 values are acc_norm. ArithMark 2 uses raw accuracy. CMA-1M-Mini was not run on either ArithMark version, so those cells are intentionally blank.

Model	Parameters	ARC Easy	ARC Challenge	PIQA	HellaSwag	ArithMark 2	ArithMark 3	Base Bench Elo	Base Bench accuracy
BananaMind 2 Micro	2,933,193	33.12%	21.93%	53.21%	28.27%	24.92%	34.00%	874	34.57%
GPT-S2-5M	5,384,258	33.42%	22.18%	56.42%	27.64%	27.20%	28.30%	883	35.71%
GPT-S-5M	5,158,464	32.83%	21.42%	57.07%	27.42%	27.04%	29.20%	889	37.14%
CMA-1M-Mini	958,692	29.25%	21.93%	54.84%	29.36%			812	27.43%

Base Bench values for GPT-S2-5M and CMA-1M-Mini come from the BananaMind Base Bench leaderboard. GPT-S-5M was not listed there when this card was prepared, so its complete official local Base Bench 1.1 result is shown.

Parameter efficiency

The comparison chart uses raw BananaMind Base Bench 1.1 accuracy and removes the four-choice random baseline before normalizing by model size:

efficiency = (accuracy - 25.00) / (parameters / 100,000)

The result is accuracy percentage points above random per 100K parameters. For example, 30% raw accuracy becomes 5 points above random before the size normalization. Higher is better.

Model	Parameters	Accuracy	Above random	Points / 100K params
BananaMind 2 Micro	2,933,193	34.57%	9.57	0.326
Syn-2.6M	2,604,210	32.57%	7.57	0.291
GPT-S-5M	5,158,464	37.14%	12.14	0.235
cma-8M	7,849,161	40.86%	15.86	0.202
GPT-S2-5M	5,384,258	35.71%	10.71	0.199
Supra-Mini-v5-8M	7,867,584	36.29%	11.29	0.144
Ant-5M	4,713,344	25.43%	0.43	0.009

This is a narrow parameter-efficiency measurement, not a complete model quality score. The chart can be regenerated with:

python make_parameter_efficiency_chart.py

Architecture

Field	Value
Parameters	2,933,193
Layers	9
Hidden size	128
Intermediate size	512
Query / KV heads	4 / 2
Head dimension	32
Context length	4,096
Vocabulary	2,048, tied
RoPE theta	100,000
Attention	GQA with pre-RoPE QK norm
MLP	SwiGLU
Normalization	RMSNorm
Refresh kernel	Strictly causal depthwise convolution, kernel 9

The refresh path reads the detached attention output as a signal, combines a learned projection with a strictly causal depthwise convolution, and gates a projection of the original token embedding. Its learned residual scalar was initialized to zero. During cached generation, convolution history is carried alongside each layer's key/value cache.

Training

Field	Value
Tokens seen	74,998,349,824
Optimizer for matrices	Stock torch.optim.Muon
Peak Muon learning rate	0.05
Optimizer for embeddings / 1D parameters	AdamW
Peak AdamW learning rate	0.004
Precision	bfloat16 autocast
Hardware	8 x NVIDIA RTX PRO 6000 Blackwell Server Edition
Training time	11,973 seconds (3h 19m 33s)

Curriculum

Token range	FineWeb-Edu	DCLM	Cosmopedia v2	FinePhrase	FineMath	NPSet2
0.00B-15.00B	55%	30%	6%	4%	4%	1%
15.00B-30.00B	48%	26%	9%	8%	7%	2%
30.00B-48.75B	42%	20%	12%	12%	11%	3%
48.75B-63.75B	36%	16%	14%	15%	15%	4%
63.75B-75.00B	30%	12%	15%	18%	21%	4%

The exact source revisions and observed source-token totals are preserved in checkpoint_metadata.json. The tokenizer corpus and revisions are preserved in tokenizer_training_manifest.json.

Usage

This repository contains custom Transformers architecture code. Review configuration_bananamind2micro.py and modeling_bananamind2micro.py before enabling trust_remote_code.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Micro"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
)

inputs = tokenizer("The future of small language models is", return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• This is a very small base model and can produce repetitive, incorrect, or incoherent text.
• It is not instruction aligned and does not reliably follow chat prompts.
• Training was primarily English, so other languages are not a target use.
• The model has no deployment safety guarantees and requires application-level evaluation and safeguards.
• Benchmark results describe fixed datasets and do not guarantee downstream performance.

Reproducibility

• Final preview revision: f45a0e949494c0e75892f65e0e5245f6060ae303
• Final training step: 35,762 / 35,762
• Weight SHA-256: f7a410921a02cd2617cd72c7cfe694da0501edcc6e4ca0747e2cdcdb5d2d14d9
• Weights are stored in float32 Safetensors format.

License

Released under the Apache License 2.0.