Hugging Face's logo

kdirgul
/
smartcore-v1

Model card Files
xet
 Community
smartcore-v1 / code /kod /LAMBA_V1_MODEL_CARD.md
kdirgul's picture
kdirgul
model card: Related Mamba-3 models + hrsvrn acknowledgement
aa418ef verified
preview code
|
Raw
History Blame Contribute Delete
7.55 kB
metadata 
license: other
license_name: lamba-v1-license-tbd
language:
  - en
  - tr
tags:
  - mamba
  - mamba-3
  - state-space-model
  - gqa
  - hybrid
  - turkish
  - bilingual
  - rag
  - causal-lm
pipeline_tag: text-generation
library_name: mamba_ssm

LAMBA V1.0 — Mamba-3 + GQA Hybrid (EN + TR)

LAMBA V1.0 is a ~177M-parameter language model trained from scratch on a Mamba-3 (SISO) + Grouped-Query Attention hybrid architecture, for English (primary) and Turkish (secondary). It is a deliberate, honest first step: small enough to train on free/affordable compute (Google Colab A100), built end-to-end — tokenizer, data, pretraining, instruction tuning, and retrieval-augmented tuning — with no warm-start from any existing model.

TR: LAMBA V1.0, sıfırdan eğitilmiş ~177M parametreli, Mamba-3 + GQA hibrit mimarili, İngilizce (birincil) ve Türkçe (ikincil) bir dil modelidir. Hiçbir hazır modelden başlatılmadı; tokenizer'dan eğitime tüm hat baştan kuruldu.

⚠️ Read this first — what LAMBA is and isn't

LAMBA V1.0 is a 177M model. At this scale it has fluent local language ability and real commonsense signal, but limited factual knowledge and weak abstract reasoning. It will hallucinate facts if asked open-ended questions from memory. Use it with retrieval (RAG) for any factual task — when given the answer in context, it reliably extracts it.

This is not a GPT-4 competitor. It is an open, reproducible, honestly-scoped small model and a foundation to grow from. Treat its free-form factual claims as unreliable.

Architecture

Total params ~177M (tied embeddings)
d_model 768
Layers 20 — 17 Mamba-3 (SISO) + 3 GQA (attention at layers 5, 11, 17; 5:1 hybrid)
Mamba-3 d_state=128, head_dim=64, complex-SSM, rope_fraction=0.5, no conv1d
GQA 12 query heads / 3 KV heads, QK-norm + RoPE
MLP GatedMLP, inner 1500
Vocab 48,000 SentencePiece BPE (EN+TR, byte-fallback, digit-split)
Context 2048
Precision bf16

The hybrid puts attention layers in the depth-middle so induction heads can form, with Mamba-3 handling the bulk linear-time sequence mixing.

Training pipeline

  1. Pretraining — 12.0B tokens (EN 6.6B / TR 2.64B / code 1.56B / math 1.2B), WSD schedule, decontaminated against the eval suite. Final loss ~2.50 (PPL ~12).
  2. SFT — bilingual instruction tuning. Pivot-language design: reasoning in English (Magpie CoT), Turkish direct-answer (Quardo Alpaca-GPT-4o). Replay mixing prevents forgetting.
  3. RAG-aware SFT — extractive QA (SQuAD v2 EN + Turkish extractive QA), sentence-mode answers
    • abstain ("I don't know" when not in context). This is what makes RAG reliable.
  4. DPO — attempted; at 177M the preference signal was marginal (acc ~0.57), so the SFT+RAG checkpoint is shipped as final. Kept for transparency.

Evaluation (zero-shot, multiple-choice log-likelihood, N=300)

Task acc acc_norm random Note
XCOPA-tr (causal) 0.590 0.540 0.50 clear signal
Belebele-tr (reading) 0.330 0.340 0.25 above random
HellaSwag-en (commonsense) 0.333 0.337 0.25 above random
XNLI-tr (inference) 0.330 0.337 0.333 ≈ random
TurkishMMLU (academic) 0.177 0.207 0.20 ≈ random

Honest read: commonsense / reading / causal reasoning carry real signal; abstract inference (NLI) and knowledge-heavy academic QA (MMLU) sit at chance — expected for 177M. This is exactly why LAMBA is designed to be used with retrieval.

Intended use & limitations

  • Good for: retrieval-grounded QA, language-fluency tasks, on-context extraction, research on small SSM/hybrid models, Turkish+English experimentation.
  • Not good for: standalone factual Q&A, math, multi-step reasoning, anything safety-critical.
  • Bias/safety: trained on web/instruction data; may produce incorrect, biased, or unsafe content. Always keep a human in the loop. Do not use for medical/legal/financial decisions.
  • Compute: the Mamba-3 forward uses a Triton GPU kernel → currently requires a GPU (Google Colab works, free tier included). A pure-PyTorch CPU port is planned for v1.1.

How to run (Colab GPU) 

# 1) install the Mamba-3 fork wheel (provided in this repo under wheels/)
!pip -q install einops sentencepiece "huggingface_hub>=0.23"
!pip -q install --no-deps ./wheels/*.whl

# 2) run retrieval-augmented inference (recommended)
!python faz7_rag.py --demo --query "Türkiye'nin başkenti neresi?"
# or your own documents:
!python faz7_rag.py --docs /content/my_docs --query "..."

See LAMBA_Inference.ipynb for a one-click Colab notebook.

Files in this repo

  • checkpoints/ — model weights (LAMBA V1.0 = the final SFT+RAG checkpoint)
  • tokenizer/ — 48K SentencePiece model
  • wheels/ — prebuilt Mamba-3 fork wheels (GPU)
  • *.py — inference + RAG code
  • LAMBA_Inference.ipynb — Colab quickstart

License

TBD — to be finalized before release. Note: the model derives from the mamba-og (Mamba-3) codebase; its license affects redistribution of the weights/kernels. This will be settled and stated here explicitly.

💛 Support the project

LAMBA is built by an independent developer on self-funded compute (Colab Pro, storage). If you find it useful and want to help train bigger, better open LAMBA models, your support goes directly to GPU hours and data:

(More options, including crypto, coming later.) Thank you. 🙏

Citation 

@misc{lamba_v1_2026,
  title  = {LAMBA V1.0: A from-scratch Mamba-3 + GQA hybrid for English and Turkish},
  author = {Kadir Gül},
  year   = {2026},
  note   = {Hugging Face model card}
}

Related Mamba-3 models

LAMBA is part of a recent wave of from-scratch Mamba-3 models. For context:

Model Size Notes
hrsvrn/mamba3-180m-finewebedu-10B ~187M EN, norm-free Mamba-3 SISO baseline — LAMBA's recipe reference
ib-ssm/mamba3-370M-10BT 370M EN, Mamba-3 base
kikyo0114/nanochat-mamba3-mimo-r2 113M EN+JP, pure-PyTorch Mamba-3 MIMO chat
RtaForge/Mamba3-2.7B 2.7B Mamba-2 → Mamba-3 migrated (alpha)
RtaForge/Mistral-Mamba3-7B ~4B Mistral → Mamba-3 transfer (alpha)
batteryphil/mamba3-baremetal-rlf ~130M experimental bare-metal reasoning

What makes LAMBA different: to our knowledge the only bilingual English + Turkish Mamba-3 model; one of the few Mamba-3 + attention hybrids (most are pure SSM); and it ships instruction- and RAG-tuned, not as a base/alpha checkpoint.

Acknowledgements

Architecture builds on Mamba-3 (state-spaces) and the mamba-og fork. The training recipe and the feasibility of the ~180M / 768-d / 12-layer band were directly informed by hrsvrn/mamba3-180m (the norm-free Mamba-3 project) — thank you. Tokenizer, data, training recipe, and all weights are original to this project.