---
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.3
library_name: pytorch
pipeline_tag: text-generation
tags:
- safetensors
- mamba3
- ssm
- subsuminator
- heist
- alpha
---

# Mistral-Mamba3-7B (Alpha)

Cross-architecture **Subsuminator heist**: [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) → **Mamba3-7B** SSM body.

## CE gate (measured)

| Metric | Value |
|--------|-------|
| Protocol | subsuminator/subsume.py CE gate (random tokens, shifted CE) |
| Source CE | **13.3000** |
| Heisted CE | **10.4125** |
| log(vocab) baseline | 10.3972 |
| **CE ratio vs source** | **0.7829×** |

Random-token next-step CE on `mistralai/Mistral-7B-Instruct-v0.3` vs this checkpoint (seed=42, n=5, seq_len=32).

**Interpretation:** Heisted CE ≈ log(vocab) (10.41 vs 10.40) — the fresh SSM body behaves like a **random baseline** on uniform tokens. Ratio **0.7829× vs source** (10.41 / 13.30) is *below* 1.0 because Mistral's trained transformer *raises* CE on garbage random inputs; this is **not** capability preservation. For comparison, Mamba2→Mamba3 structural port (trained→trained) achieved **1.0016×** with CE near source.

## What transferred

- Token embeddings, final norm, lm_head, per-layer input norms from Mistral-7B

## What did not (fresh orthogonal init)

- All Mamba3 SSM mixer weights (`in_proj`, `out_proj`, `dt_bias`, state dynamics)

**Alpha checkpoint — fine-tune before production use.**

## Run it (Avocado)

Sovereign local inference for **mamba2** + **mamba3** only:

- **[rideitlikeyoustoleit](https://github.com/Rta-Forge/heists-galore/tree/main/rideitlikeyoustoleit)** — static Avocado binaries (`--yeehaw`, `--arnie`, `--giddyup`)
- Download a release build, point at this checkpoint, `trust_remote_code` for HF load or Avocado native splat path

```bash
./avocado run --model RtaForge/Mistral-Mamba3-7B --prompt "Come with me if you want to live."
```

## Usage (trust_remote_code required)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RtaForge/Mistral-Mamba3-7B", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("RtaForge/Mistral-Mamba3-7B")
```