nepetai/ivis-400m-gpu

Ivis-400M-Nepetai (GPU)

Hybrid Transformer + Mamba SSM + Mixture of Experts — 450M params.

Total params	~451M (0.45B)
Active params	~230M
Layers	10
Hidden	1,152
Heads	18 (GQA, 6 KV)
Experts	4 (2 active)
Mamba layers	2 (hybrid every 4)
Context	256 tokens
Vocab	65,536 (BPE)

Quick Start (any GPU with 8GB+ VRAM)

git clone https://huggingface.co/nepetai/ivis-400m-gpu
cd ivis-400m-gpu
pip install -r requirements.txt
python train.py --steps 100 --batch_size 2

On Colab / RunPod / Any GPU

from src.model import IvisModel
from configs.ivis_2b_tpu import get_ivis_200m_config

config = get_ivis_200m_config()
model = IvisModel(config).to('cuda')
print(f'Parameters: {model.num_params():,}')

Push to HuggingFace after training

HF_TOKEN=hf_xxx python train.py --steps 100 --hf_push --hf_repo nepetai/ivis-400m-gpu