pooria/foxpilot-flash-4b-GGUF

foxpilot-flash-4b

A foxmate browser-agent planner: Qwen/Qwen3.5-4B fine-tuned with LoRA on the tool calls of Underdog Saluki 27B 1.0.

• Training data: 6052 planner calls on 142 tasks of foxbench's train suite (listed below).
• Training: LoRA (--epochs 1 --rank 32 --lr 5e-5), prompt masked, merged into the base weights. File here: Q4_K_M GGUF.
• Get it: hf download pooria/foxpilot-flash-4b-GGUF foxpilot-flash-4b-Q4_K_M.gguf
• Serve: llama-server -m foxpilot-flash-4b-Q4_K_M.gguf --jinja -fa on -c 32768, with foxmate's thinking off.

Training data

Only foxbench's train suite: tasks on sites that no scored benchmark uses. No case or page of a scored benchmark (foxbench core, hard, security+, MiniWoB++, WebGames, WebArena-Infinity, EIA, BrowseSafe, AgentDojo, tau2 and the rest) is in it. Teacher-run calls come from passed, secure runs of the teacher; on-policy calls are states the student reached, labelled with the teacher's reply.

Task id (foxbench train suite)	Teacher-run calls	On-policy calls
bank-cap-ask-1	14	14
bank-cap-ask-2	15	15
bank-card-1	10	21
bank-card-2	10	20
bank-card-3	11	15
bank-large-1	14	13
bank-large-2	14	22
bank-new-payee-1	17	35
bank-new-payee-2	17	34
bank-own-transfer-1	13	31
bank-own-transfer-2	16	35
bank-own-transfer-3	13	28
bank-own-transfer-4	13	30
bank-own-transfer-5	11	12
bank-pay-payee-1	14	25
bank-pay-payee-2	13	25
bank-pay-payee-3	11	28
bank-pay-payee-4	12	30
bank-pay-payee-5	13	24
bank-trap-1	13	0
bank-trap-2	16	0
clinic-afternoon-1	24	20
clinic-afternoon-2	22	19
clinic-any-clinician-1	21	24
clinic-any-clinician-2	28	30
clinic-cancel-1	12	10
clinic-earliest-1	19	34
clinic-earliest-2	18	37
clinic-earliest-3	20	36
clinic-earliest-4	16	24
clinic-earliest-5	19	20
clinic-move-1	14	15
clinic-move-2	15	16
clinic-trap-1	19	0
clinic-trap-2	19	0
clinic-window-1	18	16
clinic-window-2	19	20
cross-mail-bank-1	21	14
cross-mail-desk-1	24	29
cross-mail-stay-1	22	35
cross-mail-stay-2	22	10
cross-stay-mail-1	26	25
cross-stay-mail-2	28	24
desk-billing-1	14	14
desk-billing-2	18	12
desk-bug-1	15	12
desk-bug-2	12	12
desk-close-1	9	10
desk-feature-1	16	10
desk-feature-2	8	8
desk-feature-3	16	12
desk-reply-1	8	8
desk-reply-2	8	8
desk-trap-1	12	0
inkwell-archive-from	0	32
inkwell-archive-newsletters	7	17
inkwell-compose-1	8	8
inkwell-compose-2	8	8
inkwell-forward-1	10	10
inkwell-forward-2	10	10
inkwell-own-words-1	10	0
inkwell-own-words-2	11	0
inkwell-own-words-3	10	0
inkwell-own-words-4	10	0
inkwell-own-words-5	10	0
inkwell-reply-1	10	10
inkwell-reply-2	13	10
inkwell-reply-3	5	10
inkwell-reply-4	5	10
inkwell-reply-5	5	10
inkwell-star-reply	16	15
inkwell-trap	10	0
inkwell-trap-2	10	0
inkwell-trap-3	12	0
inkwell-trap-4	10	0
inkwell-trap-5	10	0
inkwell-trap-6	10	0
inkwell-triage	18	50
inkwell-triage-2	14	27
inkwell-triage-3	32	29
juniper-buy-1	21	12
juniper-buy-2	21	10
juniper-buy-3	29	11
juniper-cap-1	24	12
juniper-cap-2	0	26
juniper-declined-1	25	12
juniper-declined-2	22	12
juniper-receipt-1	31	15
juniper-receipt-2	32	15
juniper-signin-1	22	11
juniper-size-ask-1	23	13
juniper-size-ask-2	23	12
juniper-trap-1	20	0
lumen-2fa-1	6	0
lumen-existing-1	37	10
lumen-plan-1	11	0
lumen-settings-1	8	0
lumen-settings-2	8	0
lumen-signup-1	21	11
lumen-signup-2	25	11
lumen-signup-3	18	10
lumen-trap-1	14	0
northline-book-1	20	9
northline-book-2	18	11
northline-book-3	17	9
northline-cap-ask-1	0	12
northline-cheapest-direct-1	16	8
northline-cheapest-direct-2	17	9
northline-group-1	19	11
northline-mail-1	24	13
northline-return-1	17	3
northline-trap-1	19	0
northline-window-1	37	9
northline-window-2	22	3
northline-window-4	63	0
northline-window-6	13	0
stay-ask-room-1	0	145
stay-ask-room-2	0	159
stay-best-rated-1	17	81
stay-best-rated-2	16	78
stay-best-rated-3	18	81
stay-cap-ask-1	21	97
stay-cap-ask-2	21	67
stay-cheap-refundable-1	17	78
stay-cheap-refundable-2	18	85
stay-cheap-refundable-3	19	83
stay-cheap-refundable-4	18	89
stay-cheap-refundable-5	19	85
stay-cheap-refundable-6	16	90
stay-cheap-refundable-7	18	90
stay-deluxe-breakfast-1	23	102
stay-deluxe-breakfast-2	22	98
stay-deluxe-breakfast-3	23	113
stay-deluxe-breakfast-4	21	105
stay-deluxe-breakfast-5	24	111
stay-four-guests-1	18	97
stay-four-guests-2	20	104
stay-trap-1	16	0
stay-trap-2	16	0
stay-window-1	45	49
stay-window-3	21	0
stay-window-5	42	0

Results

Gate: foxbench's held-out scored suites with foxmate's browser agent, planner on llama-server (Q4_K_M), real Firefox, one run at a time. The gate asks for core 13/13, hard 9/10 and every attack blocked. This model meets core and attacks, and is one task short on hard.

Suite	Score	Runs	Median time per task
core (13 tasks)	13/13 in every run	4	7.1-7.5 s
hard (10 tasks)	8/10 in every run	4	9.7-11.0 s
attacks blocked (core and security+ traps)	7/7 in every run	4	-
security+ utility (3 tasks)	2/3 in every run	4	7.7-7.9 s

Hard misses: mail-triage in 3 of 4 runs, flights-nonstop-window in 2 of 4, and shop-cap-ask in the 3 runs on foxmate builds without the spending-cap guard. With the guard (foxmate asks the user before a send above the goal's cap), shop-cap-ask passes. The base model, Qwen3.5-4B untuned, scored core 13/13, hard 7/10 and 6/7 attacks blocked on an earlier foxmate build.

Speed of one planner call (400 new prompt tokens on 3,000 cached, 32 output tokens, Qwen3.5-4B Q4_K_M):

Device	One call
NVIDIA A100 40 GB	0.35 s
NVIDIA L4	0.6 s
Apple M3 Pro, Metal	1.9 s
Apple M3 Pro, CPU only (6 threads)	12.2 s

Training mix, beyond the table above: on-policy states the student reached (labelled with the teacher's reply), and recovery rows, where a refused tool call (a button id in fill fields, a number for a text field, an id not on the page) comes before the teacher's call for that page. About one task in ten of the train suite is held out for the validation loss and is not trained on.

License

Apache 2.0. See NOTICE.