metadata
language:
- en
license: apache-2.0
tags:
- mamba
- ssm
- causal-lm
- neuralai
- base-model
- sft-queued
pipeline_tag: text-generation
base_model: state-spaces/mamba-2.8b
🧠 NeuralAI · Mamba K3
NeuralAI's 2.8B Mamba SSM — the third and largest model in the NeuralAI line (K1 130M → K2 793M → K3 2.8B), fine-tuned from state-spaces/mamba-2.8b for assistant-style chat with worked reasoning, code, math, writing, and safety behaviors.
Model Details
- Architecture: Mamba SSM (
state-spaces/mamba-2.8b) - Parameters: 2.8B
- Status: SFT queued — weights will land here when training completes
- Fine-tune: LoRA r=16, α=32 on Mamba
in_proj, bf16, cosine LR 2e-4 - Dataset: NeuralAI curated set — 3,068 examples (computed-answer math, reasoning puzzles, code, safety, writing, factual, multi-turn)
- Chat template: NeuralAI
[INST](same as Mamba K1) - Creator: De'Andrew Preston Harris, NeuralAI
Status
- This page is created ahead of the SFT run (per the model-line order in github.com/Subject-Emu-5259/NeuralAI).
- Training is paused pending GPU allocation on the Colab runtime; the dataset and training script are ready to launch the moment a GPU is available.
- When the SFT run completes, the merged weights + tokenizer + config are pushed here automatically.
Training Command (when GPU is available)
BASE_MODEL=state-spaces/mamba-2.8b HF_REPO=Subject-Emu-5259/NeuralAI-Mamba-K3 \
python training/finetune_run.py
Prompt Format
Same as Mamba K1 — the NeuralAI [INST] chat template:
<|endoftext|>[INST] {user message} [/INST]
License
Apache 2.0