README.md

---
library_name: transformers
license: apache-2.0
language:
  - eu
  - en
  - es
tags:
  - mamba-2
  - basque
  - autocomplete
  - on-device
  - low-resource
pipeline_tag: text-generation
---

# Morpheus v2 (Mamba-2) — Basque Autocomplete

A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion.

## Model Details

- **Architecture:** Mamba-2 (State Space Model)
- **Parameters:** 91M
- **Embedding vocab:** 4,000 (Unigram SentencePiece)
- **Hidden dimension:** 768
- **Layers:** 24
- **State dimension:** 64
- **Head dimension:** 64
- **Inner dimension:** 1,536
- **Sequence length:** 1,024
- **Training tokens:** ~10 billion
- **Training steps:** 76,000 (best checkpoint at 74,000)
- **Held-out PPL:** 7.13
- **Trained without BOS token**

## Tokenizer

A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on evidence that lower vocab sizes achieve lower downstream perplexity for agglutinative low-resource languages (cf. QuechuaTok).

- `add_bos_token: false` (the model was trained without a BOS token)
- EOS token: `</s>` (id=2)
- UNK token: `<unk>` (id=0)

## Intended Use

On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via llama.cpp (see the GGUF quantized versions at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf)).

## Training Data

Trained on a ~22 GB cleaned Basque text corpus comprising Wikipedia, news (Berria), literature, and other web-crawled sources. The corpus underwent a multi-stage cleaning pipeline (deduplication, language filtering, quality auditing).

## Quantized Versions

GGUF quantized models (Q4_K_M, Q5_K_M) for llama.cpp inference are available at:
[itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf)

## Citation

```bibtex
@misc{morpheus_v2_mamba,
  author = {Xabier Ezpeleta},
  title = {Morpheus v2: On-Device Basque Autocompletion with Mamba-2},
  year = {2026},
}
```