pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async

Mistral-Nemo-Instruct-2407 · AWQ 4-bit (compressed-tensors W4A16, async vLLM)

Quantized and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.

⚠️ Single-runner evaluation. No FP16 baseline was evaluated for this model; all behavioral data comes from the 4-bit quantized_vllm runner only. Aggregate dimension scores are reported as per-family pass rates rather than a quantization-degradation delta.

🧩 Backend note. This is the AWQ INT4 / compressed-tensors W4A16-asymmetric checkpoint, served through vLLM's asynchronous OpenAI-compatible server (vllm serve --quantization compressed-tensors). The asymmetric W4A16 scheme requires vLLM 0.8.5.post1, whose scheme matcher accepts asymmetric weights.

---

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. This AWQ build is the fast, small-footprint option in the series: a 12B model in ~8 GB, served on vLLM at sub-second latency per evaluated case, with clean tool-call output. The evaluation results below explain where it preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.

---

Model Description

This repository contains the AWQ 4-bit (compressed-tensors W4A16-asymmetric) checkpoint of mistralai/Mistral-Nemo-Instruct-2407, a 12-billion parameter instruction-tuned model jointly developed by Mistral AI and NVIDIA. The weights were compressed with llmcompressor (AWQ algorithm, compressed-tensors format) and are served through vLLM.

Key Characteristics

• Parameters: 12B (MistralForCausalLM)
• Format: compressed-tensors · W4A16-asymmetric (4-bit weights, 16-bit activations) · AWQ
• Quantization tool: llmcompressor (recipe in recipe.yaml, scheme in quantization_manifest.json)
• File size: ~8.35 GB (2 × safetensors shards, 8,350,915,416 bytes)
• Serving runtime: vLLM 0.8.5.post1 (async OpenAI server), --quantization compressed-tensors
• Minimum VRAM (GPU inference): ~10–12 GB (weights + KV cache at modest context)
• Recommended GPU tier: RTX 3060 12 GB · RTX 4090 · A10G · A100
• Context window: 128K tokens (native); evaluated at max_model_len=4096
• Inference speed (eval hardware): avg 0.936 sec/case on RTX 4090
• License: Apache 2.0

Quantization Scheme

• Weights: 4-bit, group-wise, asymmetric (zero-point), compressed-tensors packed.
• Activations: 16-bit (float16) — W4A16.
• Algorithm: AWQ (activation-aware weight quantization) via llmcompressor.

Exact group size and per-layer scheme are recorded in recipe.yaml and quantization_manifest.json, published in this repository. The asymmetric weight scheme is the reason vLLM 0.8.5.post1 is required — earlier vLLM scheme matchers reject asymmetric W4A16.

---

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260629_184555 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner: quantized_vllm (compressed-tensors W4A16) · Total rows: 42

Per-Family Pass Rates (AWQ W4A16)

Family	N	Pass Rate	Avg Secs	Bucket Score	Notes
json_multistep	5	0.400	2.603	1.800	Inconsistent self-checks — see Finding 1
stateful_followup	2	1.000	0.290	2.000	Both turns exact match
toolcall_only	2	1.000	0.310	2.000	Correct values; action key normalized — see Finding 4
mixed_brief_json	2	1.000	0.370	2.000	Clean ANSWER + JSON
toolcall	2	1.000	0.485	11.000	Stage-1 and final correct — no EOS contamination
json	4	n/a	1.203	10.000	All pass
fuzz	20	n/a	0.924	10.000	All 20 pass
mcq	5	n/a	0.020	0.600	3/5 — see Finding 2

---

Key Findings

Finding 1: Multi-Step Planning — Inconsistent Self-Checks (0.400)

json_multistep is the weakest family for this checkpoint. The output format is always valid (schema_ok 1.000), but the model's self-check reasoning is frequently internally inconsistent (checks_consistent_ok 0.400) and it tends to STOP early.

Case-level breakdown:

Case	Difficulty	Result	Got Plan	Expected	Failure
ms_easy_01	Easy	✅	["A"]	["A"]	—
ms_easy_02	Easy	❌	["A","STOP"]	["A","A"]	premature STOP; cc=0
ms_med_01	Medium	✅	["A","B","C"]	["A","B","C"]	—
ms_med_02	Medium	❌	["A","B","C"]	["A","B","C"]	correct plan, but cc=0 / stop=0
ms_hard_01	Hard	❌	["A","A","STOP"]	["A","B","C"]	wrong plan; cc=0

ms_med_02 is the telling case: the model reaches the correct final plan (oracle_equiv_ok=1) but with an inconsistent reasoning trace and incorrect stop semantics, so it does not pass. The model lands the right plan only about half the time, and its intermediate reasoning is unreliable.

What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle-validation loop.

Finding 2: MCQ — 3/5, Mild A-Bias

Case	Result	Raw
mcq_01	✅	B
mcq_02	❌	A
mcq_03	✅	C
mcq_04	✅	B
mcq_05	❌	A

Extraction is clean (single letters); both misses defaulted to A. This is a genuine capability result, not a parsing artifact — a mild A-bias on harder items.

Finding 3: toolcall — Correct and Clean (No EOS Contamination)

Both toolcall cases pass stage-1 and stage-2 for a perfect bucket score of 11.000:

Case	Raw	Final	Result
tool_01	{"tool_name": "add", "args": {"a": 2, "b": 3}} ⏎ 5	5	✅
tool_02	{"tool_name": "add", "args": {"a": 10, "b": -4}} ⏎ 6	6	✅

When the tool schema is scaffolded, the model emits the canonical tool_name/args structure. vLLM decodes without special-token leakage, so the final answer is clean — no EOS stripping required downstream.

Finding 4: toolcall_only — Correct Values, ReAct action Key (1.000)

In the bare tool-call setting the model free-forms to the ReAct convention:

Case	Raw
toolonly_01	{"action": "add", "args": {"x": 5, "y": 10}}
toolonly_02	{"action": "add", "args": {"x": 25, "y": 75}}

The tool and argument values are correct; the model names the tool under "action" (ReAct) instead of "tool_name", and uses x/y for operands. quant_eval v7.21 normalizes tool-name aliases (tool_name / tool / name / action / function) and arg keys (x/y → a/b), so tool_name_ok=1.000 and args_ok=1.000. The outer wrapper still differs from the strict schema (schema_ok=0, hence bucket 2.000). If your downstream stack does not normalize, enforce the exact tool_name key via system prompt — note the model emits canonical tool_name when the schema is scaffolded (Finding 3).

Finding 5: Stateful and Hybrid — Clean and Fast

stateful_followup (1.000) and mixed_brief_json (1.000) pass cleanly at sub-0.4-second latency, no EOS contamination:

Family	Case	Raw
stateful	state_01	{"counter": 2} → {"counter": 5}
stateful	state_02	{"items": ["a","b"]} → {"items": ["a","b","c"]}
mixed	mixed_01	ANSWER: 13 ⏎ {"a": 4, "b": 9, "sum": 13}
mixed	mixed_02	ANSWER: 6 ⏎ {"a": -2, "b": 8, "sum": 6}

---

Signal-Level Diagnostics (AWQ W4A16)

json_multistep

Signal	Rate	Notes
schema_ok	1.000	Perfect
checks_consistent_ok	0.400	Inconsistent self-checks
stop_semantics_ok	0.800	One stop-semantics miss
oracle_equiv_ok	0.600	Right final plan 3/5

toolcall_only

Signal	Rate	Notes
tool_name_ok	1.000	add recognized (action aliased)
args_ok	1.000	Values correct (x/y → a/b)

stateful_followup / mixed_brief_json

All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.

---

Recommended Use Cases

✅ Deploy with Confidence (AWQ W4A16)

• Tool-calling agents — toolcall 11.000 (clean), toolcall_only 1.000. Canonical tool output when scaffolded; no EOS stripping required.
• Stateful multi-turn agents — 1.000 at both turns, ~0.29 sec/case.
• Hybrid brief + JSON responses — mixed_brief_json 1.000.
• Structured JSON outputs (single-step) — json and fuzz both bucket=10.000.
• Latency-sensitive / cost-sensitive deployment — 12B in ~8 GB, sub-second per evaluated case on vLLM; fits modest GPUs.

⚠️ Use with Guardrails (AWQ W4A16)

• Multi-step planning — 0.400 pass rate, inconsistent self-checks. Add an external planner/oracle-validation loop.
• MCQ / single-choice — 3/5 with a mild A-bias; verify on your option distributions.
• Bare tool-call dispatch in non-normalizing stacks — the model defaults to the ReAct action key. Enforce tool_name via system prompt; values are already correct.

---

Hardware Requirements

Configuration	VRAM Required	Notes
W4A16 (this repo) · modest context	~10–12 GB	~8 GB weights + KV cache
W4A16 · larger context (32K+)	~16–24 GB	Raise --max-model-len; KV cache grows
FP16 (not in this repo)	~24 GB+	Base Mistral-Nemo-Instruct-2407

---

Usage

Requires the vLLM-compatible stack: vLLM 0.8.5.post1, transformers 4.51.3, tokenizers 0.21.1, huggingface_hub 0.30.2 in a dedicated environment. Newer transformers removes all_special_tokens_extended, which vLLM 0.8.5.post1 needs at startup.

Serving — vLLM async server (OpenAI-compatible)

pip install "vllm==0.8.5.post1" "transformers==4.51.3" "tokenizers==0.21.1" "huggingface_hub==0.30.2"

vllm serve pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async \
  --quantization compressed-tensors \
  --dtype float16 \
  --max-model-len 4096

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
resp = client.chat.completions.create(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    messages=[
        {"role": "system", "content": "You are a precise assistant. Return structured outputs when requested."},
        {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
    ],
    temperature=0.3,
)
print(resp.choices[0].message.content)

Offline — vLLM LLM (synchronous, batched)

from vllm import LLM, SamplingParams

llm = LLM(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    quantization="compressed-tensors",
    dtype="float16",
    max_model_len=2560,
    gpu_memory_utilization=0.90,
)
# Mistral chat format: <s>[INST] ... [/INST]
prompts = ["<s>[INST] Briefly describe synthetic data generation. [/INST]"]
out = llm.generate(prompts, SamplingParams(max_tokens=128, temperature=0.3, top_p=0.92))
print(out[0].outputs[0].text)

Bare tool-call dispatch — enforce tool_name if your stack does not normalize

SYS = (
    'Respond ONLY with a JSON object using EXACTLY these keys:\n'
    '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
    'Do not use "action" or "x"/"y". No other text, no markdown.'
)
# Send SYS as the system message; the model emits canonical tool_name/args when constrained.

---

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Mistral_Nemo_Instruct_2407_AWQ_20260529_152601_20260629_184555.csv) and rollup.json are published in this repository for independent verification.

---

Artifact Provenance

Artifact	Format	Size (bytes)	SHA256
model-00001-of-00002.safetensors	compressed-tensors W4A16	4,987,280,248	85b1041123245e5876912c4ad4d2fef0b7f740b116f91631f44be8b059b7505a
model-00002-of-00002.safetensors	compressed-tensors W4A16	3,363,635,168	075a19ac17db4a32b5adfde7bb6727e7996c6196aff22cdc65038d17dbb6e5f0
model.safetensors.index.json	JSON	106,519	b678bf572b88b130e3a3a51369ef699aaad9a8fe084bf6e044ab2aa7ed6cd215
quantization_manifest.json	JSON	2,831	d5dabea10e791f0c921c2c678e9f172cadac00bc548b0f3d674a79a76ddbcc3c
recipe.yaml	YAML	865	2ab77dc9f98273b7d01a83b3805f41c73e2f47cfa2d65d3aa34f6ffd4a1bd376

Total quantized weights: 8,350,915,416 bytes (~8.35 GB). Per-file SHA256 for every file in the repository is published in file_hashes.json for independent verification. Quantized from mistralai/Mistral-Nemo-Instruct-2407 with llmcompressor (AWQ, compressed-tensors W4A16-asymmetric). The FP16 base was not evaluated.

---

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Family	Description	Pass Signals
fuzz	Property-based regression; structured placement correctness	schema_ok, constraints_ok
json	Single-step structured JSON with constraint rules	schema_ok, constraints_ok
json_multistep	Multi-step planning with self-check and oracle verification	schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
mcq	Multiple-choice extraction	choice_ok
stateful_followup	Two-turn state tracking; turn-2 correct given turn-1	turn1/2_parse_ok, turn1/2_exact_match
mixed_brief_json	Hybrid: natural language answer + valid JSON block	answer_line_ok, json_parse_ok, schema_ok
toolcall	Tool call embedded in response; parse + schema + final value	stage1_tool_parse_ok, stage1_tool_schema_ok
toolcall_only	Bare schema-only tool call; tool name + args (alias-normalized)	tool_name_ok, args_ok

Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_vllm (compressed-tensors W4A16) · Evaluation date: June 29, 2026 · Seed: 42

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

Need a deployment recommendation? Not sure which quantization — AWQ, NF4, GGUF — is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

---

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

---

📞 Work With PBH Applied Systems

This AWQ build is the series' fast, small-footprint agent backend: a 12B model in ~8 GB serving sub-second on vLLM with clean tool-call output. The evaluation tells you exactly where to lean on it (tool dispatch, stateful, structured JSON) and where to add guardrails (multi-step planning, MCQ).

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect

🌐	pbhappliedsystems.com
📧	patrick@pbhappliedsystems.com
💼	LinkedIn
▶️	YouTube
📸	Instagram
👍	Facebook

---

License

This repository inherits the license of the base model: Apache 2.0 — mistralai/Mistral-Nemo-Instruct-2407

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

---

AWQ / compressed-tensors quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_184555