---
language:
  - en
  - fr
  - de
  - es
  - it
  - pt
  - ru
  - zh
  - ja
license: apache-2.0
base_model: mistralai/Mistral-Nemo-Instruct-2407
library_name: vllm
tags:
  - safetensors
  - compressed-tensors
  - awq
  - w4a16
  - quantization
  - 4bit
  - mistral
  - mistral-nemo
  - instruct
  - vllm
  - agentic
  - tool-calling
  - structured-output
  - pbh-applied-systems
  - quant-eval
---

# Mistral-Nemo-Instruct-2407 · AWQ 4-bit (compressed-tensors W4A16, async vLLM)

**Quantized and evaluated by [PBH Applied Systems, LLC](https://pbhappliedsystems.com)**
— Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

> 🔬 **This repository is part of a production-oriented evaluation series.** Every model published under [`pbhappliedsystems`](https://huggingface.co/pbhappliedsystems) has been independently evaluated using **quant_eval v7.21** — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.

> ⚠️ **Single-runner evaluation.** No FP16 baseline was evaluated for this model; all behavioral data comes from the 4-bit `quantized_vllm` runner only. Aggregate dimension scores are reported as per-family pass rates rather than a quantization-degradation delta.

> 🧩 **Backend note.** This is the **AWQ INT4 / compressed-tensors W4A16-asymmetric** checkpoint, served through **vLLM's asynchronous OpenAI-compatible server** (`vllm serve --quantization compressed-tensors`). The asymmetric W4A16 scheme requires **vLLM 0.8.5.post1**, whose scheme matcher accepts asymmetric weights.

---

## Try This Model in the Live AI Agent Demo

[**Launch the PBH Applied Systems AI Agent Demo →**](https://pbhappliedsystems.com/assistant.html)

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses `quant_eval` results to show how model selection changes by task. This AWQ build is the **fast, small-footprint** option in the series: a 12B model in ~8 GB, served on vLLM at sub-second latency per evaluated case, with clean tool-call output. The evaluation results below explain where it preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.

---

## Model Description

This repository contains the **AWQ 4-bit (compressed-tensors W4A16-asymmetric)** checkpoint of [`mistralai/Mistral-Nemo-Instruct-2407`](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407), a 12-billion parameter instruction-tuned model jointly developed by Mistral AI and NVIDIA. The weights were compressed with **llmcompressor** (AWQ algorithm, compressed-tensors format) and are served through vLLM.

### Key Characteristics

- **Parameters:** 12B (`MistralForCausalLM`)
- **Format:** compressed-tensors · W4A16-asymmetric (4-bit weights, 16-bit activations) · AWQ
- **Quantization tool:** llmcompressor (recipe in `recipe.yaml`, scheme in `quantization_manifest.json`)
- **File size:** ~8.35 GB (2 × `safetensors` shards, 8,350,915,416 bytes)
- **Serving runtime:** vLLM 0.8.5.post1 (async OpenAI server), `--quantization compressed-tensors`
- **Minimum VRAM (GPU inference):** ~10–12 GB (weights + KV cache at modest context)
- **Recommended GPU tier:** RTX 3060 12 GB · RTX 4090 · A10G · A100
- **Context window:** 128K tokens (native); evaluated at `max_model_len=4096`
- **Inference speed (eval hardware):** avg **0.936 sec/case** on RTX 4090
- **License:** Apache 2.0

### Quantization Scheme

- **Weights:** 4-bit, group-wise, **asymmetric** (zero-point), compressed-tensors packed.
- **Activations:** 16-bit (`float16`) — W4A16.
- **Algorithm:** AWQ (activation-aware weight quantization) via llmcompressor.

Exact group size and per-layer scheme are recorded in `recipe.yaml` and `quantization_manifest.json`, published in this repository. The asymmetric weight scheme is the reason vLLM **0.8.5.post1** is required — earlier vLLM scheme matchers reject asymmetric W4A16.

---

## PBH Applied Systems Evaluation — quant_eval v7.21

> **Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21**
> Run ID: `20260629_184555` · Fixtures: `golden_oracle_fixtures_v7_21` (SHA256: `6d71a0b9147c...`) · Seed: 42
> Hardware: NVIDIA RTX 4090 · Runner: `quantized_vllm` (compressed-tensors W4A16) · Total rows: 42

### Per-Family Pass Rates (AWQ W4A16)

| Family | N | Pass Rate | Avg Secs | Bucket Score | Notes |
|---|---:|---:|---:|---:|---|
| json\_multistep | 5 | **0.400** | 2.603 | 1.800 | Inconsistent self-checks — see Finding 1 |
| stateful\_followup | 2 | **1.000** | 0.290 | 2.000 | Both turns exact match |
| toolcall\_only | 2 | **1.000** | 0.310 | 2.000 | Correct values; `action` key normalized — see Finding 4 |
| mixed\_brief\_json | 2 | **1.000** | 0.370 | 2.000 | Clean ANSWER + JSON |
| toolcall | 2 | **1.000** | 0.485 | **11.000** | Stage-1 **and** final correct — no EOS contamination |
| json | 4 | n/a | 1.203 | 10.000 | All pass |
| fuzz | 20 | n/a | 0.924 | 10.000 | All 20 pass |
| mcq | 5 | n/a | 0.020 | **0.600** | 3/5 — see Finding 2 |

---

## Key Findings

### Finding 1: Multi-Step Planning — Inconsistent Self-Checks (0.400)

`json_multistep` is the weakest family for this checkpoint. The output format is always valid (`schema_ok` 1.000), but the model's self-check reasoning is frequently internally inconsistent (`checks_consistent_ok` **0.400**) and it tends to STOP early.

**Case-level breakdown:**

| Case | Difficulty | Result | Got Plan | Expected | Failure |
|---|---|---|---|---|---|
| ms\_easy\_01 | Easy | ✅ | `["A"]` | `["A"]` | — |
| ms\_easy\_02 | Easy | ❌ | `["A","STOP"]` | `["A","A"]` | premature STOP; cc=0 |
| ms\_med\_01 | Medium | ✅ | `["A","B","C"]` | `["A","B","C"]` | — |
| ms\_med\_02 | Medium | ❌ | `["A","B","C"]` | `["A","B","C"]` | correct plan, but cc=0 / stop=0 |
| ms\_hard\_01 | Hard | ❌ | `["A","A","STOP"]` | `["A","B","C"]` | wrong plan; cc=0 |

`ms_med_02` is the telling case: the model reaches the **correct** final plan (`oracle_equiv_ok=1`) but with an inconsistent reasoning trace and incorrect stop semantics, so it does not pass. The model lands the right plan only about half the time, and its intermediate reasoning is unreliable.

**What this means for production:** do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle-validation loop.

### Finding 2: MCQ — 3/5, Mild A-Bias

| Case | Result | Raw |
|---|---|---|
| mcq\_01 | ✅ | `B` |
| mcq\_02 | ❌ | `A` |
| mcq\_03 | ✅ | `C` |
| mcq\_04 | ✅ | `B` |
| mcq\_05 | ❌ | `A` |

Extraction is clean (single letters); both misses defaulted to `A`. This is a genuine capability result, not a parsing artifact — a mild A-bias on harder items.

### Finding 3: toolcall — Correct **and** Clean (No EOS Contamination)

Both `toolcall` cases pass stage-1 **and** stage-2 for a perfect bucket score of 11.000:

| Case | Raw | Final | Result |
|---|---|---|---|
| tool\_01 | `{"tool_name": "add", "args": {"a": 2, "b": 3}}` ⏎ `5` | `5` | ✅ |
| tool\_02 | `{"tool_name": "add", "args": {"a": 10, "b": -4}}` ⏎ `6` | `6` | ✅ |

When the tool schema is scaffolded, the model emits the **canonical** `tool_name`/`args` structure. vLLM decodes without special-token leakage, so the final answer is clean — no EOS stripping required downstream.

### Finding 4: toolcall_only — Correct Values, ReAct `action` Key (1.000)

In the **bare** tool-call setting the model free-forms to the ReAct convention:

| Case | Raw |
|---|---|
| toolonly\_01 | `{"action": "add", "args": {"x": 5, "y": 10}}` |
| toolonly\_02 | `{"action": "add", "args": {"x": 25, "y": 75}}` |

The tool and argument **values are correct**; the model names the tool under `"action"` (ReAct) instead of `"tool_name"`, and uses `x`/`y` for operands. **quant_eval v7.21 normalizes tool-name aliases** (`tool_name` / `tool` / `name` / `action` / `function`) and arg keys (`x/y → a/b`), so `tool_name_ok=1.000` and `args_ok=1.000`. The outer wrapper still differs from the strict schema (`schema_ok=0`, hence bucket 2.000). If your downstream stack does not normalize, enforce the exact `tool_name` key via system prompt — note the model emits canonical `tool_name` when the schema is scaffolded (Finding 3).

### Finding 5: Stateful and Hybrid — Clean and Fast

`stateful_followup` (1.000) and `mixed_brief_json` (1.000) pass cleanly at sub-0.4-second latency, no EOS contamination:

| Family | Case | Raw |
|---|---|---|
| stateful | state\_01 | `{"counter": 2}` → `{"counter": 5}` |
| stateful | state\_02 | `{"items": ["a","b"]}` → `{"items": ["a","b","c"]}` |
| mixed | mixed\_01 | `ANSWER: 13` ⏎ `{"a": 4, "b": 9, "sum": 13}` |
| mixed | mixed\_02 | `ANSWER: 6` ⏎ `{"a": -2, "b": 8, "sum": 6}` |

---

## Signal-Level Diagnostics (AWQ W4A16)

### json_multistep

| Signal | Rate | Notes |
|---|---:|---|
| schema\_ok | 1.000 | Perfect |
| checks\_consistent\_ok | 0.400 | Inconsistent self-checks |
| stop\_semantics\_ok | 0.800 | One stop-semantics miss |
| oracle\_equiv\_ok | 0.600 | Right final plan 3/5 |

### toolcall_only

| Signal | Rate | Notes |
|---|---:|---|
| tool\_name\_ok | 1.000 | `add` recognized (`action` aliased) |
| args\_ok | 1.000 | Values correct (`x/y → a/b`) |

### stateful_followup / mixed_brief_json

All gating signals (`turn1/2_parse_ok`, `turn1/2_exact_match`; `answer_line_ok`, `json_parse_ok`, `schema_ok`) = **1.000**.

---

## Recommended Use Cases

### ✅ Deploy with Confidence (AWQ W4A16)

- **Tool-calling agents** — `toolcall` 11.000 (clean), `toolcall_only` 1.000. Canonical tool output when scaffolded; no EOS stripping required.
- **Stateful multi-turn agents** — 1.000 at both turns, ~0.29 sec/case.
- **Hybrid brief + JSON responses** — `mixed_brief_json` 1.000.
- **Structured JSON outputs (single-step)** — `json` and `fuzz` both bucket=10.000.
- **Latency-sensitive / cost-sensitive deployment** — 12B in ~8 GB, sub-second per evaluated case on vLLM; fits modest GPUs.

### ⚠️ Use with Guardrails (AWQ W4A16)

- **Multi-step planning** — 0.400 pass rate, inconsistent self-checks. Add an external planner/oracle-validation loop.
- **MCQ / single-choice** — 3/5 with a mild A-bias; verify on your option distributions.
- **Bare tool-call dispatch in non-normalizing stacks** — the model defaults to the ReAct `action` key. Enforce `tool_name` via system prompt; values are already correct.

---

## Hardware Requirements

| Configuration | VRAM Required | Notes |
|---|---|---|
| W4A16 (this repo) · modest context | ~10–12 GB | ~8 GB weights + KV cache |
| W4A16 · larger context (32K+) | ~16–24 GB | Raise `--max-model-len`; KV cache grows |
| FP16 *(not in this repo)* | ~24 GB+ | Base Mistral-Nemo-Instruct-2407 |

---

## Usage

> Requires the vLLM-compatible stack: **vLLM 0.8.5.post1, transformers 4.51.3, tokenizers 0.21.1, huggingface_hub 0.30.2** in a dedicated environment. Newer `transformers` removes `all_special_tokens_extended`, which vLLM 0.8.5.post1 needs at startup.

### Serving — vLLM async server (OpenAI-compatible)

```bash
pip install "vllm==0.8.5.post1" "transformers==4.51.3" "tokenizers==0.21.1" "huggingface_hub==0.30.2"

vllm serve pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async \
  --quantization compressed-tensors \
  --dtype float16 \
  --max-model-len 4096
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
resp = client.chat.completions.create(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    messages=[
        {"role": "system", "content": "You are a precise assistant. Return structured outputs when requested."},
        {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
    ],
    temperature=0.3,
)
print(resp.choices[0].message.content)
```

### Offline — vLLM `LLM` (synchronous, batched)

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    quantization="compressed-tensors",
    dtype="float16",
    max_model_len=2560,
    gpu_memory_utilization=0.90,
)
# Mistral chat format: <s>[INST] ... [/INST]
prompts = ["<s>[INST] Briefly describe synthetic data generation. [/INST]"]
out = llm.generate(prompts, SamplingParams(max_tokens=128, temperature=0.3, top_p=0.92))
print(out[0].outputs[0].text)
```

### Bare tool-call dispatch — enforce `tool_name` if your stack does not normalize

```python
SYS = (
    'Respond ONLY with a JSON object using EXACTLY these keys:\n'
    '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
    'Do not use "action" or "x"/"y". No other text, no markdown.'
)
# Send SYS as the system message; the model emits canonical tool_name/args when constrained.
```

---

## Evaluation Artifacts

The full per-case evaluation CSV (`comparison_results_v7_21_Mistral_Nemo_Instruct_2407_AWQ_20260529_152601_20260629_184555.csv`) and `rollup.json` are published in this repository for independent verification.

---

## Artifact Provenance

| Artifact | Format | Size (bytes) | SHA256 |
|---|---|---:|---|
| `model-00001-of-00002.safetensors` | compressed-tensors W4A16 | 4,987,280,248 | `85b1041123245e5876912c4ad4d2fef0b7f740b116f91631f44be8b059b7505a` |
| `model-00002-of-00002.safetensors` | compressed-tensors W4A16 | 3,363,635,168 | `075a19ac17db4a32b5adfde7bb6727e7996c6196aff22cdc65038d17dbb6e5f0` |
| `model.safetensors.index.json` | JSON | 106,519 | `b678bf572b88b130e3a3a51369ef699aaad9a8fe084bf6e044ab2aa7ed6cd215` |
| `quantization_manifest.json` | JSON | 2,831 | `d5dabea10e791f0c921c2c678e9f172cadac00bc548b0f3d674a79a76ddbcc3c` |
| `recipe.yaml` | YAML | 865 | `2ab77dc9f98273b7d01a83b3805f41c73e2f47cfa2d65d3aa34f6ffd4a1bd376` |

Total quantized weights: **8,350,915,416 bytes (~8.35 GB)**. Per-file SHA256 for every file in the repository is published in `file_hashes.json` for independent verification. Quantized from `mistralai/Mistral-Nemo-Instruct-2407` with llmcompressor (AWQ, compressed-tensors W4A16-asymmetric). The FP16 base was not evaluated.

---

## Evaluation Methodology

**quant_eval v7.21** — proprietary behavioral evaluation harness, PBH Applied Systems.

**Fixture set:** `golden_oracle_fixtures_v7_21` (SHA256: `6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0`)

| Family | Description | Pass Signals |
|---|---|---|
| `fuzz` | Property-based regression; structured placement correctness | schema\_ok, constraints\_ok |
| `json` | Single-step structured JSON with constraint rules | schema\_ok, constraints\_ok |
| `json_multistep` | Multi-step planning with self-check and oracle verification | schema\_ok, checks\_consistent\_ok, stop\_semantics\_ok, oracle\_equiv\_ok |
| `mcq` | Multiple-choice extraction | choice\_ok |
| `stateful_followup` | Two-turn state tracking; turn-2 correct given turn-1 | turn1/2\_parse\_ok, turn1/2\_exact\_match |
| `mixed_brief_json` | Hybrid: natural language answer + valid JSON block | answer\_line\_ok, json\_parse\_ok, schema\_ok |
| `toolcall` | Tool call embedded in response; parse + schema + final value | stage1\_tool\_parse\_ok, stage1\_tool\_schema\_ok |
| `toolcall_only` | Bare schema-only tool call; tool name + args (alias-normalized) | tool\_name\_ok, args\_ok |

**Evaluation hardware:** NVIDIA RTX 4090 (24 GB) · **Runner:** `quantized_vllm` (compressed-tensors W4A16) · **Evaluation date:** June 29, 2026 · **Seed:** 42

---

## About PBH Applied Systems

[**PBH Applied Systems, LLC**](https://pbhappliedsystems.com) is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

**Patrick Hill, M.S.** — Founder · Data Scientist · AI/ML Engineer · Author of **[Applied Machine Learning: Concepts, Tools, and Case Studies](https://a.co/d/05qat7Xz)** (required reading, UAT CSC 373)

**Core Service Areas:** LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

---

## 🔬 About quant_eval & This Evaluation Series

[**quant_eval**](https://pbhappliedsystems.com) is a proprietary behavioral evaluation harness developed by [PBH Applied Systems, LLC](https://pbhappliedsystems.com). It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under [`pbhappliedsystems`](https://huggingface.co/pbhappliedsystems) has been independently evaluated using quant_eval before being recommended for any production role.

**See it in action:** [**Live AI Agent Demo →**](https://pbhappliedsystems.com/assistant.html)

> **Need a deployment recommendation?**
> Not sure which quantization — AWQ, NF4, GGUF — is right for your hardware, latency target, or agent type?
> [**→ pbhappliedsystems.com**](https://pbhappliedsystems.com)

---

*Evaluated and published by [PBH Applied Systems, LLC](https://pbhappliedsystems.com) · [patrick@pbhappliedsystems.com](mailto:patrick@pbhappliedsystems.com)*

---

## 📞 Work With PBH Applied Systems

This AWQ build is the series' fast, small-footprint agent backend: a 12B model in ~8 GB serving sub-second on vLLM with clean tool-call output. The evaluation tells you exactly where to lean on it (tool dispatch, stateful, structured JSON) and where to add guardrails (multi-step planning, MCQ).

👉 **[Book a Scoping Call](https://pbhappliedsystems.com)** · 👉 **[Request an Evaluation Report](https://pbhappliedsystems.com)** — from $2,500

### Connect

| | |
|---|---|
| 🌐 | [pbhappliedsystems.com](https://pbhappliedsystems.com) |
| 📧 | [patrick@pbhappliedsystems.com](mailto:patrick@pbhappliedsystems.com) |
| 💼 | [LinkedIn](https://www.linkedin.com/company/pbh-applied-systems-llc) |
| ▶️ | [YouTube](https://www.youtube.com/@pbhappliedsystems) |
| 📸 | [Instagram](https://www.instagram.com/pbhappliedsystems) |
| 👍 | [Facebook](https://www.facebook.com/pbhappliedsystems) |

---

## License

This repository inherits the license of the base model:
**Apache 2.0** — [`mistralai/Mistral-Nemo-Instruct-2407`](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

---

*AWQ / compressed-tensors quantization and behavioral evaluation performed by [PBH Applied Systems, LLC](https://pbhappliedsystems.com) · quant_eval v7.21 · Run ID: `20260629_184555`*