Scaling UP in Kai! 🌊
NoesisLab/Kai-3B-Instruct

Introducing NoesisLab/Kai-3B-Instruct What happens when you force a 3B model to reason entirely in its latent space ?
Meet Kai-3B, our latest industrial-grade reasoning model fine-tuned using the Adaptive Dual Search (ADS) algorithm.
GSM8K (0-shot, Direct Answer): 39.27% 🤯 (Llama-2-7B is ~14.6%)
HumanEval (Pass@1): 39.02% 💻 (Overtakes Gemma-2-2B's 30%)
MMLU (5-shot): 53.62% 📚 (Crushing the 50% barrier)
ARC-Challenge: 51.88%🎯
PIQA: 77.53%
HellaSwag: 69.53%
Kai-3B proves that reasoning density doesn't strictly require parameter bloat or verbose generation. It acts as a perfect, cold-blooded Agent action-engine—ideal for JSON routing, SWE-bench patch generation, and anywhere you need absolute structured certainty without token waste.

2 replies

updated a model 11 months ago

marcosstable/SmolLM2-FT-MyDataset

Text Generation • 0.1B • Updated Jun 7, 2025 • 2

published a model 11 months ago

marcosstable/SmolLM2-FT-MyDataset

Text Generation • 0.1B • Updated Jun 7, 2025 • 2

upvoted an article 11 months ago

Article

No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

toslali-ibm, mirinflim, qgallouedec, esnible, rganti, mudhakar

•

Jun 3, 2025

• 101

reacted to codys12's post with 🚀👀 12 months ago

Post

2493

Introducing bitnet-r1-llama-8b and bitnet-r1-qwen-32b preview! These models are the first successful sub 1-billion-token finetune to BitNet architecture. We discovered that by adding an aditional input RMSNorm to each linear, you can finetune directly to BitNet with fast convergence to original model performance!

We are working on a pull request to use this extra RMS for any model.

To test these models now, install this fork of transformers:

pip install git+https://github.com/Codys12/transformers.git

Then load the models and test:

from transformers import (AutoModelForCausalLM, AutoTokenizer)

model_id = "codys12/bitnet-r1-qwen-32b" 
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")

bitnet-r1-llama-8b and bitnet-r1-llama-32b were trained on ~ 300M and 200M tokens of the open-thoughts/OpenThoughts-114k dataset respectively, and were still significantly improving at the end of training. This preview simply demonstrates that the concept works, for future training runs we will leave the lm_head unquantized and align the last hidden state with the original model.

Huge thanks to the team that made this possible:
Gavin Childress, Aaron Herbst, Gavin Jones, Jasdeep Singh, Eli Vang, and Keagan Weinstock from the MSOE AI Club.