abdallah400/qorva-model-fast

Qorva-Micro 32M

Hybrid Mamba-2 + GQA Attention language model trained on TinyStories.

Results

Metric	Value
Parameters	37.3M
Vocab size	50257
Tokens trained	380.1M
Final loss	2.053
Best loss	1.7397
Best PPL	5.7
Eval PPL	10.93
Dead layers	0/16

Architecture

• hidden_dim: 288 — num_layers: 16
• Hybrid ratio: 0.25 (Mamba + GQA Attention)
• Trained on: Kaggle T4 (free)

Load

import torch
from model.config import ModelConfig
from model.architecture import QorvaModel

ck    = torch.load('qorva_micro_final.pt', map_location='cpu')
cfg   = ModelConfig.from_dict(ck['config'])
model = QorvaModel(cfg)
model.load_state_dict(ck['model'])
model.eval()