BananaMind-2-Pro-Preview-Chat

BananaMind-2-Pro-Preview-Chat is the instruction-tuned version of BananaMind-2-Pro-Preview. It was fully fine-tuned for one epoch on HuggingFaceTB/smol-smoltalk, with loss applied only to assistant content and the assistant-ending EOS token.

The model has 138,971,520 parameters, a 3,072-token context window, and a custom 32,768-token digit-aware byte-level BPE tokenizer. It supports system prompts, multi-turn conversations, grouped-query attention, QK normalization, RoPE, tied embeddings, and KV-cached generation.

This chat model is based on the 96K Pro Preview checkpoint, which was pretrained on 51,904,512,000 tokens. It is a preview-derived instruction model rather than a fine-tune of the completed 100B-token BananaMind 2 Pro run.

Model Details

Field	Value
Parameters	138,971,520
Base model	BananaMind/BananaMind-2-Pro-Preview
Architecture	BananaMind2Pro decoder-only Transformer
Layers	24
Hidden size	640
Intermediate size	1,920
Attention heads	8
KV heads	4
Head dimension	80
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE, theta 100,000
Normalization	RMSNorm, epsilon 1e-6
Vocabulary size	32,768
Context length	3,072
Embeddings	Tied input/output embeddings
Generation cache	KV cache supported
Weight format	safetensors
Training type	Full-parameter supervised fine-tuning

Benchmarks

BananaMind Instruct Bench 1.1

Self-reported results from the official BananaMind Instruct Bench 1.1 script. The benchmark contains 300 deterministic, difficulty- and category-weighted instruction tasks. The BananaMind models used their native chat templates; Supra 1.5 used the benchmark's Alpaca-style fallback. Evaluation used greedy decoding, repetition_penalty=1.1, and seed 42.

Model	Overall Elo	General	Multi-turn	System Prompts	Context Recall	Code
BananaMind-2-Pro-Preview-Chat	888	720	911	793	1,194	1,337
BananaMind-2-Medium-Chat	787	534	872	666	1,085	1,291
BananaMind-2-Mini-Chat	654	389	812	743	924	667
Supra 1.5 50M Instruct	647	520	804	590	860	780
BananaMind-2-Nano-Chat	643	353	796	600	978	885

Detailed final result

Scope	Examples	Passed	Weighted Score	Elo
Overall	300	98	27.09%	888
General	120	33	21.84%	720
Multi-turn	75	26	26.32%	911
System Prompts	60	9	11.84%	793
Context Recall	30	19	50.53%	1,194
Code	15	11	65.26%	1,337

Difficulty	Examples	Passed	Weighted Score	Elo
Easy	100	59	60.58%	1,014
Medium	100	31	32.59%	900
Hard	100	8	8.53%	756

The complete final run used CUDA, bfloat16, the native chat template, and no sampling. Scores can vary with benchmark revision, Transformers version, dtype, hardware, and generation settings.

Fine-tuning Progression

Optimizer step	Packed tokens	Overall Elo	Passed	Weighted Score
500	49,152,000	811	77/300	19.98%
1,000	98,304,000	835	85/300	22.04%
1,500	147,456,000	858	91/300	24.19%
2,000	196,608,000	866	94/300	24.99%
2,500	245,760,000	874	94/300	25.69%
3,000	294,912,000	875	95/300	25.84%
3,500	344,064,000	855	88/300	23.88%
4,014 (final)	394,522,611	888	98/300	27.09%

The final export is the strongest measured checkpoint overall. Step 3,000 had a slightly higher code-category Elo of 1,345, but the final model passed more code tasks and scored higher overall.

Instruction Tuning

Field	Value
Dataset	HuggingFaceTB/smol-smoltalk
Dataset split	train
Dataset rows	460,341
Epochs	1
Final optimizer step	4,014
Packed tokens processed	394,522,611
Supervised assistant tokens	301,938,934
Sequence length	3,072
Micro batch	4
Gradient accumulation	8
Effective batch	32 sequences
Peak learning rate	1e-4
Warmup	100 steps
LR schedule	Constant after warmup
Optimizer	AdamW
Betas	0.9, 0.95
Weight decay	0.1
Gradient clipping	1.0
Compilation	PyTorch compile enabled
Seed	1337

System and user messages were retained as context but masked from the loss. Only assistant content and the assistant-ending EOS token contributed to the objective. All model parameters were trainable; no adapters or LoRA modules were used.

Chat Template

The tokenizer includes a Jinja template for system, user, and assistant messages:

<BOS><|system|>
{system message}
<|user|>
{user message}
<|assistant|>
{assistant response}<EOS>

The role markers are plain text encoded by the existing tokenizer. Fine-tuning did not add or resize any tokens, and the input and output embeddings remain tied.

Usage

This repository contains custom Transformers architecture code and must be loaded with trust_remote_code=True.

pip install -U torch transformers safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Pro-Preview-Chat"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

messages = [
    {
        "role": "system",
        "content": "You are a concise and helpful assistant.",
    },
    {
        "role": "user",
        "content": "Write a Python function that squares a number.",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
        use_cache=True,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

For multi-turn chat, append the generated assistant response and the next user message to messages, then render the full conversation with the chat template again.

Suggested Generation Settings

For stable responses:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=128 to 256
• use_cache=True

For more varied responses:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• repetition_penalty=1.1
• max_new_tokens=128 to 256

Keep a finite generation limit. The combined prompt and generated output must fit within the 3,072-token context window.

Repository Files

File	Description
config.json	Transformers model configuration
model.safetensors	Fine-tuned model weights
tokenizer.json	Custom 32,768-token tokenizer
tokenizer_config.json	Tokenizer metadata and context length
chat_template.jinja	System/user/assistant chat template
generation_config.json	Generation configuration
configuration_bananamind2pro.py	Custom Transformers config class
modeling_bananamind2pro.py	Custom Transformers causal LM class
sft_metadata.json	Fine-tuning provenance and token counts
LICENSE	BananaMind Community License 1.0
banner.png	Model-card banner
benchmarks.png	Instruct Bench model comparison
score_progression.png	Instruct Bench Elo progression across fine-tuning checkpoints

Intended Use and Limitations

BananaMind-2-Pro-Preview-Chat is intended for small-model research, local chat experiments, educational demonstrations, code-generation experiments, instruction-tuning studies, and compact-model comparisons.

The model has not received dedicated safety alignment. It can hallucinate facts, fail arithmetic or logical tasks, produce insecure or invalid code, misunderstand instructions, and generate biased, repetitive, or otherwise undesirable text. Do not rely on it for medical, legal, financial, safety-critical, or other high-stakes decisions.

License

This repository is released under the BananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.