Instructions to use mayankbhaskarr/qwen3-4b-coder-ft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mayankbhaskarr/qwen3-4b-coder-ft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mayankbhaskarr/qwen3-4b-coder-ft")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mayankbhaskarr/qwen3-4b-coder-ft")
model = AutoModelForCausalLM.from_pretrained("mayankbhaskarr/qwen3-4b-coder-ft", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mayankbhaskarr/qwen3-4b-coder-ft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mayankbhaskarr/qwen3-4b-coder-ft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mayankbhaskarr/qwen3-4b-coder-ft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mayankbhaskarr/qwen3-4b-coder-ft

SGLang

How to use mayankbhaskarr/qwen3-4b-coder-ft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mayankbhaskarr/qwen3-4b-coder-ft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mayankbhaskarr/qwen3-4b-coder-ft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mayankbhaskarr/qwen3-4b-coder-ft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mayankbhaskarr/qwen3-4b-coder-ft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use mayankbhaskarr/qwen3-4b-coder-ft with Docker Model Runner:
```
docker model run hf.co/mayankbhaskarr/qwen3-4b-coder-ft
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Uploaded finetuned model

Developed by: mayankbhaskarr
License: apache-2.0
Finetuned from model : unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit

This qwen3 model was trained 2x faster with Unsloth and Huggingface's TRL library.

Model summary

qwen3-4b-coder-ft is a QLoRA fine-tune of the Qwen3-4B-Instruct-2507 family, adapted for coding and code-instruction tasks. The repo contains the full BF16 weights (two-shard safetensors, ~8.0 GB) plus the Qwen3 chat template (chat_template.jinja). A quantized Q8_0 GGUF build is available in the companion repo mayankbhaskarr/qwen3-4b-coder-ft-gguf, and the model was validated with a live runtime benchmark (38 executed tasks + throughput profiling) — full results below.

Benchmark evaluation (October 22, 2026)

The model was evaluated with a live runtime benchmark: a 38-item executed suite (12 easy + 12 medium + 6 hard Python coding tasks scored by running generated code against held-out tests, plus 8 instruction-following items with objective rubric checks) and standardized llama-bench throughput profiling. Inference ran on CPU-only hardware (2 vCPU, ~3.7 GB usable RAM) with llama.cpp v0.7.0, greedy decoding (temperature 0.0), context 1280, 2 threads.

Because the BF16 (~8.0 GB) and Q8_0 (4.28 GB) artifacts exceed the host's usable RAM, the benchmark quantized the uploaded Q8_0 GGUF (SHA-256 verified across downloads) into Q5_K_M and Q4_K_M derivatives with llama-quantize --allow-requantize, then ran the full suite on both. Q8_0 is a documented near-lossless format (+0.0004% relative perplexity, k-quants reference data), so these results represent the fine-tuned weights faithfully, with a slight conservative bias.

Results

Tier	Q5_K_M	Q4_K_M
Easy coding (12)	12/12 (100%)	12/12 (100%)
Medium coding (12)	12/12 (100%)	12/12 (100%)
Hard coding (6)	3/6 (50.0%)	2/6 (33.3%)
Instruction following (8)	6/8 (75.0%)	7/8 (87.5%)
Overall (38)	33/38 (86.8%)	33/38 (86.8%)

Quantization agreement: 36/38 items identical (94.7%), with the two disagreements split in both directions (H04 JSON-flattener: Q5 pass / Q4 fail; I07 fib-refactor: Q5 empty-EOS / Q4 pass) — i.e., within greedy-decoding noise, no systematic quality gap between Q5_K_M and Q4_K_M.
All failures in the hard tier (LRU cache, spec-precise tokenizer, sliding-window rate limiter) are boundary-condition assertion errors in otherwise correct scaffolding. The only instruction failure (I04, code-review rubric) yields the correct fixed loop but only partially names the bug class.

Throughput (llama-bench, pp512 / tg128, 3 reps)

Config	Q5_K_M	Q4_K_M
Prompt eval, t=1	28.88 tok/s	28.30 tok/s
Prompt eval, t=2	40.54 tok/s	41.93 tok/s
Generation, t=1	2.58 tok/s	3.32 tok/s
Generation, t=2	3.33 tok/s	4.58 tok/s

Server cold-load: 8.55 s (Q5_K_M) / 6.3 s (Q4_K_M) to first healthy response; peak RSS ~3.8 GB including the 1,280-token KV cache.

Interpretation

The fine-tune preserves base-model coding competence: perfect 24/24 on easy+medium executed tasks and reliable strict-format compliance (JSON schema, exact bullets, exact answer format, executed refactors) under greedy decoding.
Q4_K_M is the recommended deployment quant for constrained hardware (37.5% faster generation, faster loading, 0.39 GB smaller, identical overall accuracy on this suite) — see the GGUF repo.
This suite measures the fine-tuned weights only; it does not compare against the base checkpoint. A base-model control run is the recommended next experiment.

Quantization footprint

Usage (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mayankbhaskarr/qwen3-4b-coder-ft"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Implement binary search in Python."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

For CPU / low-memory deployment, use the Q8_0 GGUF from the companion repo with llama.cpp (llama-server -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf --jinja).

Reproduce

# engines
git clone --depth 1 --branch v0.7.0 https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build -j --target llama-server llama-bench

# speed
./build/bin/llama-bench -m <model>.gguf -p 512 -n 128 -r 3 -t 1,2

# serve (chat template via --jinja)
./build/bin/llama-server -m <model>.gguf -c 1280 -t 2 --jinja --temp 0.0 --top-p 1.0

Raw per-item outputs: benchmark/results/ · suite definition: benchmark/harness/bench_suite.py · runner: benchmark/harness/run_bench.py

Citation

@misc{bhaskar2026qwen3coderft,
  title  = {Comparative Benchmark Evaluation of a QLoRA Fine-Tuned Qwen3-4B Coder Model and its GGUF Quantization},
  author = {Bhaskar, Mayank},
  year   = {2026},
  month  = {October},
  url    = {https://huggingface.co/mayankbhaskarr}
}

References

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs (arXiv:2305.14314)
Yang, A., et al. (2025). Qwen3 Technical Report (arXiv:2505.09388)
Kawrakow, I. (2022). K-quantization methods (llama.cpp PR #1684)
Kurt, U. (2026). Which quantization should I use? A unified evaluation of llama.cpp quantization methods (arXiv:2601.14277)