MedPsy-4B - ONNX 4-bit (GPTQ)
An ONNX conversion of qvac/MedPsy-4B for ONNX Runtime, quantized to a 4-bit body with GPTQ and 8-bit sensitive layers, for on-device use on Android, desktop and the browser (Transformers.js). The model is QVAC's MedPsy medical fine-tune of Qwen3-4B; OpenMed made and published this conversion and is not affiliated with or endorsed by the model's authors.
Revision of 2026-09-25. The weights are now GPTQ-calibrated. They replace the round-to-nearest build first published on 2026-09-23, at the same size and in the same file layout; mean next-token divergence from the source fell by 36 to 53% on three held-out texts (table below).
Size and fidelity
| Source (BF16) | This repo | |
|---|---|---|
| Parameters | 4.02 B | 4.02 B (unchanged) |
| Weights, measured from the tensors | - | 2.94 GiB |
| Bits per weight, measured from the tensors | 16 | 6.28 |
Measured against the source model (Transformers, FP32) with ONNX Runtime 1.30.0 on CPU, over 4,092 scored tokens per text; none of these texts is in the calibration set:
| Text | Mean KL divergence | Top-1 agreement | Perplexity change | KL before calibration |
|---|---|---|---|---|
| Public-domain prose (Project Gutenberg) | 0.037 | 88.8% | +1.9% | 0.077 |
| Medical prose (MedlinePlus summaries) | 0.020 | 93.3% | +0.8% | 0.031 |
| Held-out encyclopaedic text (wikitext-2 test) | 0.029 | 92.8% | +2.2% | 0.062 |
The width is higher than the "4-bit" label suggests because the tensors that lose most at 4 bits, and the shared embedding table, are kept at 8 bits. Weights exclude the rotary-position tables stored in the graph.
Quantization
| Field | Value |
|---|---|
| Body | int4 asymmetric GPTQ, group 32 (MatMulNBits block 32), fp32 scales; accuracy_level 4 |
| Kept at 8 bits | GPTQ int8, group 32, on the fused q/k/v and down-projection tensors of the layers where llama.cpp's Q4_K_M rule upgrades the attention value and down projections |
| Embedding and output head | one int8 block-32 table stored once, read by GatherBlockQuantized and by MatMulNBits (bits 8); round-to-nearest, unchanged from the first build |
| Calibration | GPTQ (512 sequences x 512 tokens of generic English text: wikitext-2-raw-v1 train, revision b08601e04326, set sha256 4c59b35612487c49…); no medical text and no evaluation data |
| Tools | GPTQModel 7.5.0; onnxruntime-genai 0.16.0 model builder for the graph; onnxruntime 1.30.0 |
The tokenizer, chat template and generation defaults are the upstream files, unchanged. Exact settings, tool versions, per-corpus results and every file's SHA-256 are in openmed_build.json; the graph's metadata_props record the source revision and the modification notice.
Architecture
| Field | Value |
|---|---|
| Source model type | qwen3 (Qwen3ForCausalLM), a fine-tune of Qwen3-4B |
| Hidden size | 2560 |
| Layers | 36, grouped-query attention |
| Vocabulary | 151,936, tied input/output embeddings |
Quick start
Transformers.js (browser or Node)
import { pipeline } from "@huggingface/transformers";
const generator = await pipeline("text-generation", "OpenMed/MedPsy-4B-ONNX", { dtype: "q4" });
const messages = [
{ role: "user", content: "Explain the difference between a panic attack and a heart attack in plain language." },
];
const output = await generator(messages, { max_new_tokens: 256, do_sample: false });
console.log(output[0].generated_text.at(-1).content);
ONNX Runtime (Python, Android and elsewhere)
The graph is a standard ONNX Runtime decoder: feed input_ids, attention_mask and the past_key_values.* cache inputs, then feed each present* output back as the matching past* input on the next step. It uses ONNX Runtime's com.microsoft operators (MatMulNBits, GatherBlockQuantized, GroupQueryAttention), so run it with ONNX Runtime 1.30 or later (onnxruntime-android on Android).
Tested with onnxruntime 1.30.0 (CPU execution provider) on macOS and with Transformers.js 4.3.0, which produced identical greedy tokens. It has not yet been measured on an Android device.
File set
| File | Size | SHA-256 |
|---|---|---|
ATTRIBUTIONS.md |
0.1 MiB | 6344cf5bde340ab3… |
LICENSE |
0.0 MiB | 809fa1ed21450f59… |
added_tokens.json |
0.0 MiB | c0284b582e14987f… |
chat_template.jinja |
0.0 MiB | a8b0dcfcc923bd26… |
config.json |
0.0 MiB | 6137e74668e6860a… |
generation_config.json |
0.0 MiB | ac1d241e9617d6ca… |
merges.txt |
1.6 MiB | 8831e4f1a0444713… |
onnx/model_q4.onnx |
0.4 MiB | aadae2c876bf9b86… |
onnx/model_q4.onnx_data |
2,038.4 MiB | 900050b08e562988… |
onnx/model_q4.onnx_data_1 |
1,098.7 MiB | 57b05faa9e5905a5… |
special_tokens_map.json |
0.0 MiB | 76862e765266b85a… |
tokenizer.json |
10.9 MiB | aeb13307a71acd8f… |
tokenizer_config.json |
0.0 MiB | 9253c1f4ca0a328b… |
vocab.json |
2.6 MiB | ca10d7e9fb3ed185… |
Intended use
For research and education. QVAC describes MedPsy as a model for research and educational purposes, trained on data generated from CC-BY-NC datasets; see the upstream card. It is not a medical device and not a substitute for professional medical advice, diagnosis or treatment. Outputs can be wrong; verify them.
Licence
Distributed under the Apache License 2.0, the source model's licence (with the upstream attributions in ATTRIBUTIONS.md). The ONNX graphs and weight files in onnx/ were converted and quantized from the source model's PyTorch weights by OpenMed; all other files are unchanged upstream copies.
Source: qvac/MedPsy-4B at revision 77aec3ab8c6eae92e18951e0a86c68c84a067d28. Please cite the original model when you use this conversion.
- Downloads last month
- 359