Privoid-ai/Privoid-Commerce-Qwen3-4B

Privoid Commerce

A LoRA adapter for Qwen/Qwen3-4B, revision 1cfa9a7208912126459214e8b04321603b3df60c, continued from Privoid Tool. It specializes in e-commerce tool calling: orders, shipping, returns, refunds, exchanges, cart and product search.

Test verdict: IMPROVED over Qwen3-4B: test success rose and the 95% range stays above zero.

Benchmark

On 956 held-out examples, task success rose by +0.240 against Qwen3-4B (95% range +0.189 to +0.296) and by +0.199 against Privoid Tool (95% range +0.163 to +0.241).

This improvement does not extend to multi-step retail actions. On the τ-bench retail replay, Privoid Commerce scored 0.197 against 0.223 for Qwen3-4B (change -0.025, 95% range -0.060 to +0.006). That is no demonstrated gain, and about four in five of these writes still fail. Do not use it for order changes without validation and human confirmation.

Metric	Qwen3-4B	Privoid Tool	Privoid Commerce
Task score	0.698	0.852	0.928
Tool selection	0.884	0.882	0.902
Complete call	0.518	0.525	0.758
Schema validity	0.973	0.967	0.999
Mock execution	0.821	0.861	0.966
Unnecessary calls (lower is better)	0.101	0.106	0.048

Test set	Examples	Qwen3-4B	Privoid Tool	Privoid Commerce	Change vs Qwen3-4B (95% range)
tau_retail_write	157	0.223	0.178	0.197	-0.025 (-0.060 to +0.006)
qrz_heldout_schema	180	0.272	0.311	0.833	+0.561 (+0.489 to +0.633)
qrz_canonical	170	0.441	0.382	0.847	+0.406 (+0.329 to +0.482)
familiar_new_request	108	0.870	0.954	0.981	+0.111 (+0.047 to +0.190)
competing_tools	72	0.903	0.972	0.972	+0.069 (+0.017 to +0.130)
missing_info	54	1.000	1.000	1.000	+0.000 (+0.000 to +0.000)
multi_turn	54	1.000	1.000	1.000	+0.000 (+0.000 to +0.000)
unavailable_tool	45	0.222	1.000	1.000	+0.778 (+0.708 to +0.857)
unseen_schema	45	0.978	1.000	1.000	+0.022 (+0.000 to +0.073)
adversarial_result	36	1.000	1.000	1.000	+0.000 (+0.000 to +0.000)
ambiguous	22	0.136	0.091	0.591	+0.455 (+0.455 to +0.455)
no_tool_needed	13	1.000	1.000	1.000	+0.000 (+0.000 to +0.000)

No frontier model was compared in this run. The numbers are also in benchmark/benchmark.json.

Intended use

For teams building private store assistants over a defined tool catalog. It is not a general chatbot. It proposes calls only. Your application must validate every call, and must handle authorization and confirmation. A model output never establishes permission.

Results

Measure	Qwen3-4B	Privoid Tool	Privoid Commerce
Task score	0.698	0.852	0.928
Tool selection	0.884	0.882	0.902
Tool-call precision	0.877	0.887	0.892
Tool-call recall	0.884	0.882	0.902
Schema validity	0.973	0.967	0.999
Argument correctness	0.518	0.525	0.758
Complete call correctness	0.518	0.525	0.758
Unnecessary tool-call rate (lower is better)	0.101	0.106	0.048

Test success change against Qwen3-4B: +0.240, 95% range +0.189 to +0.296. Against Privoid Tool: +0.199, 95% range +0.163 to +0.241. Ranges resample template and customer families.

Subset	Examples	Qwen3-4B	Privoid Tool	Privoid Commerce
adversarial_result	36	1.000	1.000	1.000
ambiguous	22	0.136	0.091	0.591
competing_tools	72	0.903	0.972	0.972
familiar_new_request	108	0.870	0.954	0.981
missing_info	54	1.000	1.000	1.000
multi_turn	54	1.000	1.000	1.000
no_tool_needed	13	1.000	1.000	1.000
qrz_canonical	170	0.441	0.382	0.847
qrz_heldout_schema	180	0.272	0.311	0.833
tau_retail_write	157	0.223	0.178	0.197
unavailable_tool	45	0.222	1.000	1.000
unseen_schema	45	0.978	1.000	1.000

No frontier model was compared in this run, so this card makes no frontier claim.

tau_retail_write is an offline replay of τ-bench retail test tasks. It is not the official τ-bench score, so don't compare it with leaderboard numbers.

Task score is the mean success across behaviors: call, ask, decline, answer and safe handling of adversarial tool results. Ask and decline are judged by text patterns, so those scores are approximate. Much of the test data is synthetic, so the τ-bench replay is the better guide. These results don't show production readiness.

• Median latency, one request at a time: 2.63 s unchanged, 4.79 s fine-tuned, on NVIDIA L4, sampled on 20 examples.
• Peak allocated GPU memory: 11.0 GB unchanged, 11.7 GB fine-tuned, during batched evaluation.

Tool-call format

The native Qwen3 format, with thinking disabled:

<tool_call>
{"name": "get_order_status", "arguments": {"order_id": "ORD-482910"}}
</tool_call>

Local inference

The adapter needs the base model at the revision above. After both are downloaded, inference can run without internet access.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Privoid-ai/Privoid-Commerce-Qwen3-4B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", revision="1cfa9a7208912126459214e8b04321603b3df60c", dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(base, "Privoid-ai/Privoid-Commerce-Qwen3-4B", subfolder="adapter").eval()
prompt = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False, enable_thinking=False)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
output = model.generate(**inputs, max_new_tokens=300, do_sample=False)

Training

• Data: Privoid synthetic store scenarios with fictional identities, Qrzysztof e-commerce tool calling (Apache-2.0) and replayed τ-bench retail training tasks (MIT). τ-bench test tasks were used for evaluation only.
• Sizes: {'train': 8014, 'val': 141, 'test': 956}.
• Method: LoRA r=16 on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, learning rate 5e-05, effective batch 8.
• Stopped by time_budget after 213 optimizer steps. Selected step 150.
• Duration: 43 minutes of training, 120 minutes including setup and evaluation.
• Hardware: NVIDIA L4 on Google Colab, bf16.

Limitations and unsupported behavior

• One tool call per turn. Parallel calls are not trained or evaluated.
• Most store data is synthetic, so real catalogs and real customers may behave differently.
• The τ-bench replay supplies gold history. In live use, earlier mistakes can compound.
• Adversarial handling was tested only on the injection styles in this notebook. It is not a security guarantee, so keep real controls in your application.
• It can still choose the wrong tool or wrong value. Validate calls against your schema before use.
• Tested with Transformers. vLLM and GGUF were not tested with this adapter.

Privacy boundaries

The model can run on your own hardware, and then requests need not leave it. That is about deployment, not training: training ran on Google Colab and files were stored on Google Drive, so that process was not air-gapped. No Privoid customer data was used.

License

This adapter is a derivative of Qwen3-4B and Privoid Tool, released under Apache-2.0. See the included LICENSE. Modified files are noted in NOTICE_PRIVOID.md. Training data keeps its own licenses.