Qwen3.8-27B-text2sql-DynQuant-4bit

Qwen/Qwen3.8-27B, fine-tuned for text-to-SQL with QLoRA and then quantized with DynQuant to 3.999 bits per weight (12.54 GiB on disk).

DynQuant gives every module its own width, driven by two signals measured during the fine-tune: how much activation mass each weight sees, and how unstable its gradient is across optimizer steps. Modules the training dynamics say are load-bearing keep their bits; the rest pay for them.

At 3.999 bits this arm sits essentially at the architecture's floor budget of 4.0196 bits -- the narrowest average width at which every module can still hold the minimum its role requires. It still breaks 9 of them, listed below.

Results

Execution accuracy on the held-out validation split of spider, gretel, wikisql, 400 problems, greedy decode.

Model	Bits	Size	Accuracy	vs bf16	p
bf16 (unquantized)	16	--	85.50%	--	--
this checkpoint	3.999	12.54 GiB	84.25%	-1.25	0.2266

The comparison is paired: both arms answered the same problems in the same order, so the difference is a McNemar test on the per-problem outcomes rather than two independent accuracies subtracted.

Decode budget was 1024 new tokens, and 0 generations reached it without finishing. A short decode budget scores a model that deliberates as though it were wrong, so this is reported rather than assumed.

What the allocator did

498 modules were quantized at group size 128, scored by the rank_product allocator.

Width	Modules	Parameters	Share
2-bit	1	89,128,960	0.33%
3-bit	135	11,759,779,840	43.73%
4-bit	265	13,749,452,800	51.13%
8-bit	97	1,294,991,360	4.82%

Floors

Each role carries a minimum width below which that role is known to break -- an embedding, an LM head, an attention projection and an MLP gate do not tolerate the same compression. When a budget cannot pay for every floor, DynQuant breaks the cheapest ones and reports every one it broke, rather than quietly lowering a floor until the arithmetic works.

9 modules were allocated below the floor their role requires:

Role	Modules	Parameters	Floor	Given
mlp.gate	2	178,257,920	4b	3b
lin_attn.qkv	2	104,857,600	4b	3b
mlp.up	1	89,128,960	3b	2b
attn.q_gate	1	62,914,560	4b	3b
attn.o	1	31,457,280	4b	3b
lin_attn.z	1	31,457,280	4b	3b
lin_attn.out	1	31,457,280	4b	3b

Use it

pip install "dynquant[hf]"

from transformers import AutoModelForCausalLM, AutoTokenizer
import dynquant

# Both lines are needed. transformers has no entry-point discovery for
# quantization methods, so without the registration call it skips the
# quantization it does not recognise and hands back a randomly initialised
# model -- fluent-looking output, no exception, no non-zero exit.
dynquant.register_hf_quantizer()

model = AutoModelForCausalLM.from_pretrained("VikramPal/Qwen3.8-27B-text2sql-DynQuant-4bit", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("VikramPal/Qwen3.8-27B-text2sql-DynQuant-4bit")

That path is measured rather than assumed: this repo was pulled from the Hub and loaded with the released wheel on torch 2.13, and the graph that came back was checked to hold 498 DynQuant modules at the widths in the table above.

Serving

vllm serve VikramPal/Qwen3.8-27B-text2sql-DynQuant-4bit does not work, and the failure belongs here rather than in your terminal:

ValueError: There is no module or parameter named 'embed_tokens.offsets' in Qwen3_5Model

vLLM builds this architecture's input embedding without offering it to a quantization plugin at all, so a packed model.embed_tokens has nowhere to land. It is not a flag you are missing and it is not the plugin failing to register: on vLLM 0.27.1 the DynQuant plugin loads, the engine reports quantization=dynquant, and then this raises.

That one tensor is the whole gap. Re-exported with the input embedding in the compute dtype and the other 497 quantized modules left exactly as they are, this arm serves on vLLM 0.27.1 and answers correctly -- so nothing else in the allocation is in the way. That export is not what is published here. Until it is, use the transformers path above.

How it was made

• Base: Qwen/Qwen3.8-27B, text tower only

• Regime: qlora, LoRA rank 32

• Data: 9,999 conversations from spider, gretel, wikisql, create-context, 350,799 supervised tokens

• Steps: 625 at effective batch 16, lr 0.0001, final train loss 0.0963

• Signals: collected from 546 modules during the fine-tune itself, with no extra forward or backward pass

• DynQuant: 0.5.2

• Contamination: the training split was matched against every evaluation item and 601 examples were dropped for colliding with one. No source overlaps an evaluation task after that.

Citation

The method has no published paper yet, so these cite the code and these weights.

@software{dynquant,
  author  = {Kamboj, Vikrampal},
  title   = {{DynQuant}: dynamic-signal quantization for large language models},
  version = {0.5.2},
  url     = {https://github.com/kambojvikram/dynquant},
  year    = {2026}
}

@misc{qwen3_8_27b_text2sql_dynquant_4bit,
  author       = {Kamboj, Vikrampal},
  title        = {Qwen3.8-27B-text2sql-DynQuant-4bit},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/VikramPal/Qwen3.8-27B-text2sql-DynQuant-4bit}}
}

Limitations

• Fine-tuned and evaluated on text-to-SQL. General-purpose ability was not measured, and quantization is not free elsewhere.

• Only the base model's text tower was trained and quantized. This checkpoint does not carry the vision path.

• The score is execution accuracy on the datasets named above, against their own schemas. Accuracy on your schemas is a different measurement.

• 9 modules sit below their role's floor; the table above says which.