README.md

---
base_model: mistralai/Mistral-7B-Instruct-v0.3
tags:
  - cuda
  - custom-kernels
  - inference-optimization
  - mistral
license: apache-2.0
---

# Optimized Transformers — mistralai/Mistral-7B-Instruct-v0.3

This package contains an auto-generated optimized build of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) produced by the NeuralNova Auto-Optimization pipeline. The forward and backward passes of the model's bottleneck operations have been replaced with custom CUDA kernels, improving inference throughput over stock Transformers.

**This repo does not host model weights.** It ships the optimization code only; weights are still pulled from `mistralai/Mistral-7B-Instruct-v0.3` at load time.

**Optimized ops**: MistralRMSNorm (25.1x standalone speedup), MistralMLP (4.0x standalone speedup)
**Throughput improvement**: 1.31x inference throughput (51.51 → 67.38 tok/s), 1.73x finetune throughput (4787.9 → 8263.8 tok/s)
**Output quality**: WARN — 16/21 prompts identical to baseline, 5/21 show phrasing variation in long-context generation; zero hallucinations detected

---

## ⚠️ Kernel binaries — read before using

`kernels/MistralRMSNorm` and `kernels/MistralMLP` ship as **precompiled `.so` binaries only** — the CUDA source (`kernel.cu`) is not included in this release. They will only load on a matching stack:

- Python 3.12 (`cp312`)
- CUDA 13.0, torch 2.11.0
- GPU compute capability sm_80 / sm_86 / sm_89 / sm_90 (A100, H100, RTX 3080–4090)

On any other stack, `pip install` will succeed but importing the extension will fail or crash. If you need a different environment, you'll need to rebuild from source — source is not currently published here.

---

## Installation

Install in order:

**Step 1 — Install Python dependencies**
```bash
pip install -r requirements.txt
```

**Step 2 — Install CUDA kernels**

Pre-built binaries are included — no compiler or CUDA toolkit required (see compatibility warning above):

```bash
pip install kernels/MistralRMSNorm
pip install kernels/MistralMLP
```

**Step 3 — Apply the patched Transformers file**

This build modifies exactly one file in [huggingface/transformers](https://github.com/huggingface/transformers) v5.8.1: `modeling_mistral.py` (`MistralRMSNorm.forward` and `MistralMLP.forward` only, verified by diff against the upstream release). Install upstream transformers at that version, then drop in the patched file from `patched_transformers/`:

```bash
pip install transformers==5.8.1
python -c "import transformers, os, shutil; d = os.path.dirname(transformers.__file__) + '/models/mistral'; shutil.copy('patched_transformers/modeling_mistral.py', d)"
```

**Step 4 — Install flash-attn**

The patched Transformers uses FlashAttention-2 for the attention op. Install from a prebuilt wheel — no compiler or CUDA toolkit required:

```bash
# Install wheel support
pip install wheel

# Install flash-attn from prebuilt wheel
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.4/flash_attn-2.8.3+cu130torch2.11-cp312-cp312-linux_x86_64.whl

# Verify
python -c "import flash_attn; print('flash-attn OK, version:', flash_attn.__version__)"
```

---

## Usage

Use patched Transformers as you would the standard `transformers` library — the CUDA kernels are injected transparently. Mistral-7B-Instruct is a chat-tuned model, so use `apply_chat_template` rather than passing raw text:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")
model = model.cuda()
outputs = model.generate(inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Serving

To serve the model with `transformers serve`:

```bash
transformers serve --model mistralai/Mistral-7B-Instruct-v0.3 --port 8000
```

---

## Benchmark Results

| Metric | Baseline | Optimized | Delta |
|---|---|---|---|
| Inference throughput (tok/s) | 51.51 | 67.38 | **+30.8%** |
| GSM8K accuracy (50-sample) | 0.46 | 0.38 | -0.08 (within statistical variance) |
| Training throughput (tok/s) | 4,787.9 | 8,263.8 | **+72.6% (1.73x)** |

**Hallucination check**: WARN — 16/21 prompts identical to baseline, 5/21 show phrasing variation. Zero hallucinations. All divergences occur in long-context generation (400–1000 token outputs), where minor RMSNorm numerical differences shift the greedy sampling trajectory.

---

## Notes

- This package was generated for **mistralai/Mistral-7B-Instruct-v0.3** — kernels are tuned for this model's specific layer shapes and dtypes.
- **System requirements**: Python 3.12, CUDA 13.0, GPU with sm_80 / sm_86 / sm_89 / sm_90 architecture (A100, H100, RTX 3080+, RTX 4090).
- **Injected ops**: MistralRMSNorm and MistralMLP only. A MistralAttention kernel was built by the pipeline but **not injected** — it's incompatible with the KV-cache / `position_embeddings` API needed for autoregressive generation, so standard FlashAttention-2 is used instead.
- **Training note**: The MLP kernel's `backward()` does not return weight gradients (it's an inference-optimized kernel). During full finetuning, MLP projection weights stay frozen while attention weights train normally — disable the MLP kernel if you need to finetune MLP weights.
- The patched file in `patched_transformers/` contains targeted modifications only to `MistralRMSNorm.forward` and `MistralMLP.forward`, based on transformers v5.8.1. `modular_mistral.py` is unmodified from upstream and is not included here.