🌶️ Training BananaMind2 with my GPU at 80°C

Banaxi PRO

Banaxi-Tech

24 20

Banaxi-Tech

AI & ML interests

SLMs, training from scratch, LoRA, TTS, Ternary models

Recent Activity

new activity about 5 hours ago

AxiomicLabs/Open_SLM_Leaderboard:Add Banaxi-Tech/cascade-alpha

published a model about 10 hours ago

Banaxi-Tech/cascade-alpha

updated a model about 10 hours ago

Banaxi-Tech/cascade-alpha

View all activity

Organizations

Posts 7

Post

BananaMind 2 Mini is now officially training!
Live Dashboard: BananaMind/BananaMind-2-Mini-Status (updates every few minutes)
We are training it on 30B Tokens and a mix of DCLM, FineWeb Edu, FineMath, and Cosmopedia!
We estimate the training to take around 2 days.
Follow us to know when it releases!

Post

4097

Today we are releasing BananaMind-KV1-8M-2Bit-Experimental, a KV-cache-aware trained model that stores its generation KV cache in 2-bit precision instead of the usual 16-bit precision.

Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.

Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental

The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.

KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.

During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.

WikiText-2 eval vs 16-bit KV cache reference:

Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675

If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental

Code: https://github.com/Banaxi-Tech/kv1