BananaMind-2-Pro

BananaMind-2-Pro is a 138,971,520-parameter decoder-only base language model trained from scratch by BananaMind. The completed pretraining run processed 99,999,449,088 tokens—the scheduled realization of a 100B-token curriculum—over 184,954 optimizer steps.

The model has a 3,072-token context window and a custom 32,768-token digit-aware byte-level BPE tokenizer. Its architecture uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with trust_remote_code=True.

Release Status

Field	Value
Release type	Final base checkpoint
Checkpoint step	184,953
Optimizer steps completed	184,954
Tokens seen	99,999,449,088
Training target	100B tokens
Final phase	Quality finish
Training status	Complete
Evaluation status	Complete

Model Details

Field	Value
Parameters	138,971,520
Architecture	BananaMind2Pro decoder-only Transformer
Layers	24
Hidden size	640
Intermediate size	1,920
Attention heads	8
KV heads	4
Head dimension	80
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocabulary size	32,768
Context length	3,072
Embeddings	Tied input/output embeddings
Generation cache	KV cache supported
Weight format	safetensors
HF architecture	BananaMind2ProForCausalLM
HF model type	bananamind2_pro

Architecture Overview

The model maps tokens through a tied input embedding, 24 pre-normalized Transformer blocks, a final RMSNorm, and the tied language-model head. Each block contains causal grouped-query self-attention followed by a SwiGLU feed-forward network, with residual connections around both sublayers.

Eight query heads share four key/value heads, reducing KV-cache size while retaining multi-head query capacity. QK normalization stabilizes attention logits, and RoPE supplies position information without learned absolute-position embeddings. During autoregressive generation, each layer can reuse cached keys and values instead of recomputing the full prefix.

Evaluation

These results describe the exact final checkpoint at step 184,953. ARC Easy, ARC Challenge, PIQA, and HellaSwag use zero-shot acc_norm,none. ArithMark 3 uses length-normalized continuation accuracy, while ArithMark 2 uses raw continuation accuracy. Code Elo is the Base Bench 1.1 code-completion category; Base Bench 1.1 Elo is the complete 350-item suite.

Benchmark	BananaMind-2-Pro	BananaMind-2-Pro-Preview	GPT-X2.5-135M	BananaMind-2-Medium	GPT-2
Trainable parameters	139M	139M	135M	49.6M	124M
ARC Easy	53.58%	51.01%	51.81%	43.81%	39.35%
ARC Challenge	27.82%	27.13%	29.18%	25.34%	22.35%
PIQA	67.52%	66.76%	69.42%	61.86%	62.08%
HellaSwag	42.78%	39.83%	40.57%	32.43%	31.26%
ArithMark 3	38.20%	38.90%	38.10%	36.20%	35.70%
ArithMark 2	32.08%	28.60%	N/A	28.20%	26.48%
INT Index	24.96	23.04	25.17	15.37	N/A
Code Elo	1407	1295	1253	1034	996
Base Bench 1.1 Elo	1124	1106	1106	1034	996

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark 3. ArithMark 2 is not included:

N(s, c) = 100 * (s - c) / (100 - c)
INT = [N(H,25) + N((ARC_E+ARC_C)/2,25) + N(P,50) + 0.65*N(A3,25)] / 3.65

INT Index vs Training Compute

Training compute is estimated consistently as 6 × parameters × training tokens. Using the exact 138,971,520-parameter count gives 83,382.91 PFLOPs for Pro Final at 100B tokens and 43,279.49 PFLOPs for Pro Preview at 51.904512B tokens. GPT-X2.5 uses 135M parameters, 75B tokens, and a 2,048-token context window; context length is not a separate term in this 6ND approximation.

Model	Estimated training compute	INT Index
BananaMind-2-Pro Final	83,382.91 PFLOPs	24.96
BananaMind-2-Pro Preview	43,279.49 PFLOPs	23.04
GPT-X2.5-135M	60,750.00 PFLOPs	25.17
GPT-X2-125M	56,286.75 PFLOPs	23.36
GPT-X-125M	11,210.56 PFLOPs	19.94
Supra2-100M	18,000.00 PFLOPs	19.41
SmolLM-135M	484,254.03 PFLOPs	25.74
BananaMind-2-Medium	14,867.33 PFLOPs	15.37
OPT-125M	135,000.00 PFLOPs	13.80

Base Bench Checkpoint Progression

This progression contains 41 complete 350-item evaluations from 2.70B tokens through the final 99.999B-token checkpoint. Every point uses CUDA, bfloat16, and batch size 1 for consistency. The final batch-1 endpoint is 1132 Elo, 236/350 correct (67.43%), and 64.69% weighted accuracy. The primary comparison table above retains the separately measured batch-32 result.

Base Bench 1.1 Category Results

The final model passed 233 of 350 cases for 66.57% raw accuracy and 63.73% weighted accuracy. Its overall fixed-item rating is 1124 Elo.

Category	Elo	Correct	Accuracy	Weighted accuracy
Language completion	1570	50/50	100.00%	100.00%
Commonsense	1185	40/50	80.00%	78.86%
World knowledge	1168	40/50	80.00%	77.12%
Context tracking	938	22/50	44.00%	41.72%
Quantitative	886	15/50	30.00%	28.71%
Logical reasoning	1059	24/50	48.00%	44.51%
Code completion	1407	42/50	84.00%	85.67%
Overall	1124	233/350	66.57%	63.73%

Base Bench was evaluated locally with CUDA, bfloat16, and batch size 32. ArithMark 3 used CUDA bfloat16 with batch size 64; ArithMark 2 used CUDA bfloat16 with batch size 1. Results can vary with harness versions, tokenizer handling, dtype, and scoring configuration. All values are self-reported evaluations.

Tokenizer

BananaMind-2-Pro uses a custom 32,768-token byte-level BPE tokenizer trained on 75 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing complete numbers from being merged into large number tokens.

Token	ID
0	19
1	20
2	21
3	22
4	23
5	24
6	25
7	26
8	27
9	28

Special token IDs:

Token	ID
`<	pad
`<	bos
`<	eos
`<	unk

Training Data

The 100B-token curriculum combines educational web text, broad web text, synthetic textbook material, mathematics, and Python educational code.

Dataset	Target tokens	Aggregate share
FineWeb-Edu	50.166B	50.17%
DCLM	26.125B	26.13%
Cosmopedia-v2	13.525B	13.53%
FineMath-4+	7.875B	7.88%
NPSet-2 Python Edu	2.309B	2.31%
Total	100.000B	100.00%

Training used a capacity-aware curriculum:

Phase	Token range	Purpose
Breadth foundation	0B to 25B	Web-heavy language and knowledge foundation
Knowledge ramp	25B to 40B	Gradual increase in synthetic, mathematics, and code data
Reasoning core	40B to 75B	Sustained reasoning-oriented mixture
Synthesis ramp	75B to 90B	Transition toward the finishing distribution
Quality finish	90B to 100B	Final quality-focused mixture

Training Setup

Field	Value
Sequence length	3,072
Micro batch	4
Gradient accumulation	44
Effective batch	176 sequences
Tokens per optimizer step	540,672
Optimizer steps	184,954
Scheduled training tokens	99,999,449,088
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	1.5e-3
Warmup steps	2,000
LR schedule	Warmup-stable-decay with cosine decay
Decay ratio	0.15
Weight decay	0.1, then 0.01 after 40B tokens
Gradient clipping	1.0
Z-loss coefficient	1e-4 until 40B tokens, then off
Compile	PyTorch compile enabled
Seed	1337

The schedule used a 2,000-step linear warmup, a long stable learning-rate region, and a final cosine decay over the last 15% of training. Weight decay dropped from 0.1 to 0.01 after 40B tokens, and the auxiliary z-loss was disabled at the same boundary. The final optimizer update completed the planned 99,999,449,088-token schedule without rounding up to an extra sequence.

Energy and Carbon Estimate

The following is an engineering estimate for the complete pretraining run, not a wall-meter measurement. Runtime is extrapolated from the observed 52,438-token/s run average. GPU and CPU package power were measured during training; motherboard, memory, storage, PSU loss, electricity use, and emissions are estimates.

Item	Basis	Value
Derived training time	99.999449B tokens / 52,438 tokens/s	529.72 hours (22.07 days)
GPU power	Measured 12-second nvidia-smi average at 99–100% utilization	262 W
CPU package power	Two Intel RAPL samples of 42 W and 38 W	40 W
Motherboard, memory, storage, and fans	Estimated combined power	50 W
DC system load	Estimated	352 W
PSU efficiency	Assumed	90%
Wall power	Estimated	391 W
Electricity use	Estimated	207.1 kWh
Austrian grid intensity used	Recent daily estimate	140 gCO2e/kWh
Complete-run emissions	Estimated	29.0 kg CO2e

Using the cited 2024 and 2025 Austrian flow-traced annual means of 125.5 and 169.4 gCO2e/kWh gives an estimated range of 26.0–35.1 kg CO2e. The grid figures come from Electricity Maps. This estimate excludes embodied hardware emissions, tokenizer training, evaluation, external networking, and remote storage.

Usage

Install the runtime dependencies:

pip install -U torch transformers safetensors

Load the custom architecture with remote code enabled:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Pro"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For deterministic continuation scoring, use do_sample=False. For free-form sampling, a temperature from 0.6 to 0.8, top_p=0.9, and repetition_penalty=1.1 are reasonable starting points.

Intended Use

BananaMind-2-Pro is intended for base-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, checkpoint analysis, and small-language-model comparisons.

Suitable uses include studying training dynamics, testing compact-model architectures, comparing tokenizers, measuring continuation likelihoods, and using the checkpoint as a starting point for task-specific fine-tuning.

Limitations

• This is a base model and does not reliably follow conversational instructions.
• The 3,072-token context window limits long-document use.
• At 139M parameters, factual recall and multi-step reasoning remain capacity-constrained.
• The training mixture is English-heavy, so performance in other languages is not characterized.
• The model has not received dedicated safety alignment and can produce inaccurate, biased, repetitive, or undesirable text.
• Loading requires repository-provided custom Transformers code with trust_remote_code=True.

Do not rely on the model for medical, legal, financial, safety-critical, or other high-stakes decisions.

License

This repository is released under the BananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.