Lernex/Metis-1.3-base

Metis-1.3 Base

Metis-1.3 Base is a 201M-parameter English-first decoder model from the Metis family. It uses a hybrid Mamba2 plus attention backbone and was trained as the base model for later chat and reasoning post-training.

Model summary

• Family: Metis
• Stage: Base pretraining checkpoint exported for Hugging Face
• Parameters: 201,490,560
• Architecture: Mamba2-attention hybrid decoder
• Context length: 4096
• Vocabulary size: 8192
• Dtype: bfloat16 weights

Architecture

Metis-1.3 uses 28 decoder blocks:

• 21 Mamba2 blocks
• 7 attention blocks at layers 3, 7, 11, 15, 19, 23, and 27
• Grouped-query attention with 18 query heads and 6 KV heads
• Tied input and output embeddings
• RMSNorm

This release is intended as the raw pretrained base for further instruction tuning and reasoning tuning.

Training

• Pretraining target: 12B train tokens
• Tokenizer: custom 8k tokenizer trained on an 8M-document sample
• Intended data mix: English-first web, educational, math, and code-heavy mixture

Intended use

This checkpoint is mainly useful for:

• research and experimentation on small hybrid Mamba models
• base-model comparison against the chat and think variants
• continued fine-tuning or alignment work

It is not the most user-friendly conversational variant. For direct assistant use, the chat or think releases are likely better starting points.

Limitations

• This is a small model and will still make factual, reasoning, and instruction-following mistakes.
• The eval setup used in this project is lightweight and should be treated as a smoke test rather than a comprehensive benchmark.
• English is the intended primary language.

Files

• model.safetensors
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• special_tokens_map.json

License

This release inherits the licensing and attribution obligations of the upstream training data sources used in the Metis pipeline. Review dataset licenses and usage constraints before production use.