serda-dev/mamba-130m-hf-turkish

Turkish Continued Pretraining of mamba-130m-hf

This repository provides a Turkish continued-pretrained variant of state-spaces/mamba-130m-hf (Transformers-compatible Mamba 130M). The goal is to improve Turkish fluency and local domain robustness while keeping the original architecture and HF usage experience.

Developed by the LinguAI Team, affiliated with KTUN and the YAZGİT community.

What is Mamba?

Mamba is a selective State Space Model (SSM) architecture designed for efficient sequence modeling with linear-time scaling in sequence length. It was introduced by Gu & Dao in “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”.

---

Training summary (this checkpoint)

• Base model: state-spaces/mamba-130m-hf
• Training type: Continued pretraining (CPT) / domain-adaptation pretraining for Turkish
• Hardware: Single GPU NVIDIA GeForce RTX 4060 Laptop GPU
• Raw text used: ~400 MB Turkish text (after your preprocessing)
• Approx token estimate: ~80M–120M tokens (rule-of-thumb: ~3–5 bytes/characters per token depending on tokenizer + text composition; 400MB typically lands around this band)

Notes on the token estimate: without the exact tokenizer statistics (total input_ids count) and exact encoding (UTF-8 composition, whitespace density, punctuation rate), the most honest way is a range. If you want, you can compute the exact value by summing tokenized lengths across your dataset shards and update this section.

---

Intended use

• Turkish text generation (base LM behavior; not instruction-tuned)
• Turkish domain adaptation for downstream fine-tuning (LoRA / full fine-tune)
• Experimentation with SSM-based backbones in transformers

Not intended for:

• Safety-critical decisions
• Legal/medical advice
• “Chat assistant” behavior out of the box (this is a base causal LM; you’ll need instruction tuning + safety alignment for assistant-like use)

---

Quickstart

Install requirements (recommended)

The original publisher recommends installing transformers from main (historically required until a given release), plus the optimized CUDA-kernel dependencies for best performance: causal-conv1d and mamba-ssm.

pip install git+https://github.com/huggingface/transformers@main
pip install "causal-conv1d>=1.2.0"
pip install mamba-ssm

If causal-conv1d and/or mamba-ssm are not installed, Transformers will fall back to an “eager” implementation; with them installed, it can use optimized CUDA kernels when available. ([Hugging Face][1])

---

Usage (generation)

Below is the standard transformers generate workflow used by the upstream model card.

import torch
from transformers import AutoTokenizer, MambaForCausalLM

MODEL_ID = "serda-dev/mamba-130m-hf-turkish"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = MambaForCausalLM.from_pretrained(MODEL_ID)

prompt = "Türkiye'de yazılım mühendisi olmak hakkında kısa bir paragraf yaz:"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.9,
        top_p=0.95,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Tips

• For fastest inference on NVIDIA GPUs, ensure CUDA kernels are enabled by installing mamba-ssm + causal-conv1d.

• If you run into build issues for these packages, double-check:

  • Your PyTorch CUDA build matches your driver/runtime
  • You have a compiler toolchain (e.g., build-essential) on Linux
  • You’re using a compatible Python version

---

Fine-tuning (PEFT / LoRA)

The upstream model card includes a PEFT fine-tuning example and recommends keeping the model in float32 during finetuning in that example context.

High-level LoRA recipe:

• Keep LR conservative for CPT-adapted models if your dataset is small
• Target Mamba projection modules similarly to upstream suggestions (e.g., x_proj, in_proj, out_proj, embeddings)
• Validate perplexity on a held-out Turkish set

(If you want, you can paste your exact training script + config and I’ll write a “Reproducibility” section with command lines and hyperparameters.)

---

Evaluation (what to check)

For a CPT’d base LM, common quick checks:

• Perplexity on a held-out Turkish slice
• Qualitative prompts: news style, conversational Turkish, formal writing, domain slang
• Degeneration: repetition, short loops, weird token fragments
• Catastrophic forgetting: basic English capability (if you care)

---

Limitations & safety

• This is a base language model (not instruction-tuned). It may:

  • Hallucinate facts
  • Produce biased or unsafe text
  • Reflect issues present in training data

• Use standard filtering and safety layers for deployments.

---

Acknowledgements (upstream credit)

This model is a continued-pretrained derivative of state-spaces/mamba-130m-hf. The installation and usage instructions above are based on the upstream Hugging Face model card for Transformers-compatible Mamba.

Mamba architecture reference:

• Albert Gu, Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces.

---

Citation

If you use this model in academic work, please cite the Mamba paper:

@article{gu2023mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}

Also consider citing the upstream HF checkpoint:

• state-spaces/mamba-130m-hf

---

Team & Affiliations

This model was developed by the LinguAI Team, an independent research-oriented AI team affiliated with Konya Technical University (KTUN) and operating under the YAZGİT community.

LinguAI Team Members (Core Contributors):

• Ahmet Furkan Kalle
• Alican Tanyeri
• Baris Icoz
• Behlul Anik
• Murat Serda Çelik