---
license: other
license_name: lamba-v1-license-tbd
language:
  - en
  - tr
tags:
  - mamba
  - mamba-3
  - state-space-model
  - gqa
  - hybrid
  - turkish
  - bilingual
  - rag
  - causal-lm
pipeline_tag: text-generation
library_name: mamba_ssm
---

# LAMBA V1.0 — Mamba-3 + GQA Hybrid (EN + TR)

**LAMBA V1.0** is a ~177M-parameter language model **trained from scratch** on a
**Mamba-3 (SISO) + Grouped-Query Attention** hybrid architecture, for **English (primary)
and Turkish (secondary)**. It is a deliberate, honest first step: small enough to train on
free/affordable compute (Google Colab A100), built end-to-end — tokenizer, data, pretraining,
instruction tuning, and retrieval-augmented tuning — with no warm-start from any existing model.

> **TR:** LAMBA V1.0, sıfırdan eğitilmiş ~177M parametreli, **Mamba-3 + GQA hibrit**
> mimarili, **İngilizce (birincil) ve Türkçe (ikincil)** bir dil modelidir. Hiçbir hazır
> modelden başlatılmadı; tokenizer'dan eğitime tüm hat baştan kuruldu.

---

## ⚠️ Read this first — what LAMBA is and isn't

LAMBA V1.0 is a **177M** model. At this scale it has **fluent local language ability and real
commonsense signal, but limited factual knowledge and weak abstract reasoning.** It **will
hallucinate** facts if asked open-ended questions from memory. **Use it with retrieval (RAG)
for any factual task** — when given the answer in context, it reliably extracts it.

This is not a GPT-4 competitor. It is an open, reproducible, honestly-scoped small model and a
foundation to grow from. Treat its free-form factual claims as unreliable.

---

## Architecture

| | |
|---|---|
| Total params | ~177M (tied embeddings) |
| `d_model` | 768 |
| Layers | 20 — **17 Mamba-3 (SISO)** + **3 GQA** (attention at layers 5, 11, 17; 5:1 hybrid) |
| Mamba-3 | `d_state=128`, `head_dim=64`, complex-SSM, `rope_fraction=0.5`, no conv1d |
| GQA | 12 query heads / 3 KV heads, QK-norm + RoPE |
| MLP | GatedMLP, inner 1500 |
| Vocab | 48,000 SentencePiece BPE (EN+TR, byte-fallback, digit-split) |
| Context | 2048 |
| Precision | bf16 |

The hybrid puts attention layers in the depth-middle so induction heads can form, with Mamba-3
handling the bulk linear-time sequence mixing.

## Training pipeline

1. **Pretraining** — 12.0B tokens (EN 6.6B / TR 2.64B / code 1.56B / math 1.2B), WSD schedule,
   decontaminated against the eval suite. Final loss ~2.50 (PPL ~12).
2. **SFT** — bilingual instruction tuning. Pivot-language design: reasoning in English
   (Magpie CoT), Turkish direct-answer (Quardo Alpaca-GPT-4o). Replay mixing prevents forgetting.
3. **RAG-aware SFT** — extractive QA (SQuAD v2 EN + Turkish extractive QA), sentence-mode answers
   + abstain ("I don't know" when not in context). **This is what makes RAG reliable.**
4. **DPO** — attempted; at 177M the preference signal was marginal (acc ~0.57), so the SFT+RAG
   checkpoint is shipped as final. Kept for transparency.

## Evaluation (zero-shot, multiple-choice log-likelihood, N=300)

| Task | acc | acc_norm | random | Note |
|---|---|---|---|---|
| XCOPA-tr (causal) | **0.590** | 0.540 | 0.50 | clear signal |
| Belebele-tr (reading) | 0.330 | **0.340** | 0.25 | above random |
| HellaSwag-en (commonsense) | 0.333 | 0.337 | 0.25 | above random |
| XNLI-tr (inference) | 0.330 | 0.337 | 0.333 | ≈ random |
| TurkishMMLU (academic) | 0.177 | 0.207 | 0.20 | ≈ random |

**Honest read:** commonsense / reading / causal reasoning carry real signal; abstract inference
(NLI) and knowledge-heavy academic QA (MMLU) sit at chance — expected for 177M. This is exactly
why LAMBA is designed to be used **with retrieval**.

## Intended use & limitations

- **Good for:** retrieval-grounded QA, language-fluency tasks, on-context extraction, research
  on small SSM/hybrid models, Turkish+English experimentation.
- **Not good for:** standalone factual Q&A, math, multi-step reasoning, anything safety-critical.
- **Bias/safety:** trained on web/instruction data; may produce incorrect, biased, or unsafe
  content. Always keep a human in the loop. Do not use for medical/legal/financial decisions.
- **Compute:** the Mamba-3 forward uses a **Triton GPU kernel** → currently **requires a GPU**
  (Google Colab works, free tier included). A pure-PyTorch CPU port is planned for **v1.1**.

## How to run (Colab GPU)

```python
# 1) install the Mamba-3 fork wheel (provided in this repo under wheels/)
!pip -q install einops sentencepiece "huggingface_hub>=0.23"
!pip -q install --no-deps ./wheels/*.whl

# 2) run retrieval-augmented inference (recommended)
!python faz7_rag.py --demo --query "Türkiye'nin başkenti neresi?"
# or your own documents:
!python faz7_rag.py --docs /content/my_docs --query "..."
```

See `LAMBA_Inference.ipynb` for a one-click Colab notebook.

## Files in this repo

- `checkpoints/` — model weights (LAMBA V1.0 = the final SFT+RAG checkpoint)
- `tokenizer/` — 48K SentencePiece model
- `wheels/` — prebuilt Mamba-3 fork wheels (GPU)
- `*.py` — inference + RAG code
- `LAMBA_Inference.ipynb` — Colab quickstart

## License

> **TBD** — to be finalized before release. Note: the model derives from the `mamba-og`
> (Mamba-3) codebase; its license affects redistribution of the weights/kernels. This will be
> settled and stated here explicitly.

## 💛 Support the project

LAMBA is built by an independent developer on self-funded compute (Colab Pro, storage).
If you find it useful and want to help train **bigger, better open LAMBA models**, your support
goes directly to GPU hours and data:

- **Patreon:** https://www.patreon.com/c/kdirgul/membership

(More options, including crypto, coming later.) Thank you. 🙏

## Citation

```bibtex
@misc{lamba_v1_2026,
  title  = {LAMBA V1.0: A from-scratch Mamba-3 + GQA hybrid for English and Turkish},
  author = {Kadir Gül},
  year   = {2026},
  note   = {Hugging Face model card}
}
```

## Related Mamba-3 models

LAMBA is part of a recent wave of from-scratch Mamba-3 models. For context:

| Model | Size | Notes |
|---|---|---|
| [hrsvrn/mamba3-180m-finewebedu-10B](https://huggingface.co/hrsvrn/mamba3-180m-finewebedu-10B) | ~187M | EN, norm-free Mamba-3 SISO baseline — **LAMBA's recipe reference** |
| [ib-ssm/mamba3-370M-10BT](https://huggingface.co/ib-ssm/mamba3-370M-10BT) | 370M | EN, Mamba-3 base |
| [kikyo0114/nanochat-mamba3-mimo-r2](https://huggingface.co/kikyo0114/nanochat-mamba3-mimo-r2) | 113M | EN+JP, pure-PyTorch Mamba-3 **MIMO** chat |
| [RtaForge/Mamba3-2.7B](https://huggingface.co/RtaForge/Mamba3-2.7B) | 2.7B | Mamba-2 → Mamba-3 migrated (alpha) |
| [RtaForge/Mistral-Mamba3-7B](https://huggingface.co/RtaForge/Mistral-Mamba3-7B) | ~4B | Mistral → Mamba-3 transfer (alpha) |
| [batteryphil/mamba3-baremetal-rlf](https://huggingface.co/batteryphil/mamba3-baremetal-rlf) | ~130M | experimental bare-metal reasoning |

**What makes LAMBA different:** to our knowledge the only **bilingual English + Turkish** Mamba-3
model; one of the few **Mamba-3 + attention hybrids** (most are pure SSM); and it ships
**instruction- and RAG-tuned**, not as a base/alpha checkpoint.

## Acknowledgements

Architecture builds on **Mamba-3** (state-spaces) and the `mamba-og` fork. The training recipe
and the feasibility of the ~180M / 768-d / 12-layer band were directly informed by
**[hrsvrn/mamba3-180m](https://huggingface.co/hrsvrn/mamba3-180m-finewebedu-10B)** (the norm-free
Mamba-3 project) — thank you. Tokenizer, data, training recipe, and all weights are original to
this project.