FoodGuard-4B

This repository provides an updated FoodGuard-4B release from the project introduced in Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models.

FoodGuard-4B is a 4B-parameter guardrail for food-related text, fine-tuned from Qwen3Guard-Gen-4B on FoodGuardBench. It classifies inputs as safe or unsafe under a food-safety policy, supporting prompt moderation and offline content screening.

• Food-safety coverage: Contamination, preparation, temperature control, hygiene, and storage.
• Adversarial detection: 99.24% ID accuracy and 97.64% OOD accuracy, with evaluation on five attack families absent from training.
• Benign utility: 100.00% on 500 unmodified benign test questions and 99.33% on a 1,500-example benign transformation probe.

Model details

Property	Value
Base model	Qwen/Qwen3Guard-Gen-4B
Architecture	Qwen3 causal language model
Model size	4B parameters
Input / output	Text / binary safety assessment
Training benchmark	FoodGuardBench
Evaluation language	English
Weight format	FP32 Safetensors
License	Apache-2.0

Quickstart

Install the tested dependencies. The PyTorch command below uses CUDA 12.8; select a build appropriate for your hardware.

python -m pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install transformers==4.57.3 accelerate==1.12.0 safetensors==0.7.0 huggingface_hub

Use hf auth login if repository access requires authentication. This example loads the model and its food-safety policy directly from Hugging Face:

from pathlib import Path

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "EddyLuo/FoodGuard"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
) if device == "cuda:0" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=dtype, device_map={"": device},
    attn_implementation="sdpa", trust_remote_code=False,
).eval()
policy = Path(hf_hub_download(model_id, "classifier_policy.txt")).read_text(
    encoding="utf-8"
).rstrip("\n")

messages = [
    {"role": "system", "content": policy},
    {"role": "user", "content": "How should I wash my hands before preparing food?"},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
input_length = inputs.input_ids.shape[1]
if input_length > 5120:
    raise ValueError("Input exceeds the tested 5,120-token limit")
inputs = inputs.to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs, do_sample=False, max_new_tokens=16,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0, input_length:], skip_special_tokens=True).strip())

Example output:

Safety Assessment: safe

Use the bundled classifier_policy.txt and original chat template. The tested input limit is 5,120 tokens, including the policy and template; inputs are not silently truncated. Generation is deterministic with at most 16 new tokens.

For a downloaded local repository, the bundled helper supports single-input and JSONL classification:

python inference.py --model . --text "How should I wash my hands before preparing food?"
python inference.py --model . --input-file examples/benign.jsonl --output-file predictions.jsonl

The helper returns predicted_label, final_answer, and input_tokens. Invalid output formats return predicted_label: null.

Safety policy

Safe applies to informational content that does not facilitate harm. Unsafe applies to content that includes or enables hazardous food handling, contamination, evasion of regulation or inspection, illegal practices, or other unsafe guidance.

The benchmark covers five risk areas:

Risk area	Scope
Contamination	Contamination and cross-contamination risks
Preparation	Unsafe preparation and food handling
Temperature control	Unsafe cooking, holding, cooling, and reheating
Hygiene	Food-handling sanitation and personal hygiene
Storage	Unsafe storage and preservation

The model outputs exactly one line: Safety Assessment: safe or Safety Assessment: unsafe. The five risk areas describe evaluation coverage; the output is a binary label.

Performance

FoodGuardBench

ID covers AutoDAN, TAP, and FlipAttack, the three attack families used in training. OOD covers five held-out attack families: Bijection Learning, Emoji, Humor, Language Game, and T2I Prompt. Malicious test seeds are disjoint from training and validation seeds.

All scores below are percentages; higher is better.

Model	Benign utility	ID overall accuracy	OOD overall accuracy
FoodGuard-4B	100.00	99.24	97.64
Qwen3Guard-Gen-4B	97.80	50.87	56.23
Qwen3Guard-Gen-8B	98.20	50.49	56.42
GPT-OSS-Safeguard-20B	90.80	83.78	68.52

Overall accuracy includes both benign and malicious queries: 3,686 ID examples and 5,810 OOD examples. Both splits contain the same 500 benign questions, and each attack family has 1,062 malicious examples. The Qwen3Guard overall scores are calculated from benign utility and the per-family detection results below using these same split sizes.

Compared with its Qwen3Guard-Gen-4B base, FoodGuard-4B improves ID overall accuracy by 48.37 percentage points and OOD overall accuracy by 41.41 percentage points, with 100.00% utility on unmodified benign questions.

Attack-specific accuracy measures the percentage of malicious queries correctly classified as unsafe:

Attack family	Split	FoodGuard-4B	Qwen3Guard-Gen-4B	Qwen3Guard-Gen-8B	GPT-OSS-Safeguard-20B
AutoDAN	ID	99.91	80.60	78.91	99.44
TAP	ID	97.46	49.53	48.21	91.81
FlipAttack	ID	100.00	0.38	1.88	56.78
Bijection Learning	OOD	99.91	0.00	0.00	0.09
Emoji	OOD	97.36	84.46	88.04	97.55
Humor	OOD	99.53	87.66	85.88	99.15
Language Game	OOD	100.00	27.40	14.41	51.98
T2I Prompt	OOD	90.30	62.05	74.11	83.33

FoodGuard-4B detects 100.00% of malicious FlipAttack queries, compared with 0.38% for Qwen3Guard-Gen-4B, 1.88% for Qwen3Guard-Gen-8B, and 56.78% for GPT-OSS-Safeguard-20B. Detection reaches 99.91% on Bijection Learning and 100.00% on Language Game. T2I Prompt has the lowest detection accuracy for FoodGuard-4B at 90.30%.

Across three independently trained runs, ID accuracy is 99.33% ± 0.31 and OOD accuracy is 99.08% ± 1.25 (mean ± sample standard deviation in percentage points). The tables above describe this release; these aggregate scores summarize three models.

Benign utility

Utility is the percentage of benign inputs classified as safe. FoodGuard-4B achieves 100.00% utility on the 500 unmodified benign test questions. A paired probe applies three transformations to those same questions, giving 500 examples per transformation and 1,500 in total. For Qwen3Guard, only Safe counts as benign acceptance; Controversial is treated as unsafe for this utility calculation.

Benign input form	FoodGuard-4B	Qwen3Guard-Gen-4B	Qwen3Guard-Gen-8B
Unmodified	100.00	97.80	98.20
AutoDAN wrapper	98.60	98.80	98.80
Word-order reversal only	99.40	99.40	99.00
Fixed TAP-style framing	100.00	98.80	99.40
Combined transformation probe	99.33	99.00	99.07

The reversal-only group has no FlipAttack wrapper, and TAP-style uses fixed framing. All probe labels are safe, so this probe measures benign acceptance under these transformations. Checkpoint selection uses validation only and does not use this probe.

Validation accuracy is 99.46% (366/368), balanced accuracy is 99.69%, and macro-F1 is 98.86%. Detailed counts are provided in evaluation_summary.json.

Training

FoodGuard-4B is fine-tuned on benign food-related questions and AutoDAN, TAP, and FlipAttack variants of malicious questions from FoodGuardBench. Source malicious questions are split into 956 training, 106 validation, and 1,062 test seeds; benign questions are split into 450, 50, and 500 questions. These counts precede attack augmentation. Splits have no malicious seed or exact input overlap.

Training uses one epoch, a learning rate of 1.5e-5, gradient accumulation of 32, and a maximum sequence length of 5,120 tokens. Validation runs every 25 optimizer steps; this release uses the checkpoint selected at step 100. Full configuration and the base-model revision are in foodguard_checkpoint.json.

Hardware and runtime

FP32 weights occupy approximately 17.6 GB (16.4 GiB) on disk. BF16 inference was tested on an NVIDIA RTX 5070 Ti with 16 GB VRAM. The inference helper selects BF16 or FP16 on CUDA GPUs and FP32 on CPU.

Tested software: Python 3.12, PyTorch 2.11.0+cu128, Transformers 4.57.3, Accelerate 1.12.0, and Safetensors 0.7.0.

Intended use and limitations

FoodGuard-4B supports food-related prompt moderation and offline labeling. Evaluation focuses on English text and the attack families above. Performance on other languages, domains, and unseen prompt-injection techniques may differ. Response classification and multimodal inputs are outside the demonstrated evaluation scope.

False positives and false negatives remain possible. A safe prediction does not verify factual correctness or whether a food product is safe to consume. Applications should handle invalid outputs explicitly and evaluate the model against their own moderation requirements.

Citation

If you use FoodGuard-4B or FoodGuardBench, please cite:

@misc{luo2026cookingrisksbenchmarkingreducing,
  title={Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models},
  author={Weidi Luo and Xiaofei Wen and Tenghao Huang and Hongyi Wang and Zhen Xiang and Chaowei Xiao and Kristina Gligorić and Muhao Chen},
  year={2026},
  eprint={2604.01444},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2604.01444}
}

License

Apache-2.0. FoodGuard-4B is derived from Qwen3Guard-Gen-4B. Upstream attribution is in NOTICE, and the full license is in LICENSE. Release file checksums are in release_manifest.json.