vanka_v01

A 4B model that runs a parallel multi-agent extraction loop: as orchestrator it splits a task's variables into groups and calls spawn_subagent once per group (in parallel), then calls calculate; as sub-agent it extracts its assigned variables from the text as JSON, with verbatim evidence. The final answer comes from deterministic code, not from the model.

Distilled from Qwen3.6-35B-A3B (AWQ) into Qwen/Qwen3.5-4B on MedCalc-Bench rule-based calculators (calculate = the official MedCalc-Bench implementations).

Research prototype. Not for clinical use. Do not use its outputs to make or support medical decisions.

Code, prompts, tool schemas and the full pipeline: https://github.com/Pires-De-Andrade/multiagent_classification

Results (MedCalc-Bench Verified test, 17 rule-based calculators, n=340)

Condition Acc. Per-variable acc. No answer Critical-path tokens
Qwen3.5-2B, single call, no training 9.4% 52.3% 68.5% 597
vanka_v01 80.9% 93.2% 1.2% 647
vanka_v01 + 5-vote sub-agents 81.8% 93.4% 0.0% 703
Teacher (35B), orchestrator 79.1% 92.8% 5.6% 1561
Teacher (35B), single agent 58.8% 88.0% 20.3% 666

vanka_v01 vs the teacher orchestrator: +1.8 pts, not significant (exact McNemar p=0.53), with ~2.4x fewer critical-path tokens. APACHE II and Caprini are excluded from the main metric (unreliable labels: the official code reproduces only 50% / 45% of their test labels).

Limitations:

  • One benchmark, synthetic and PubMed-derived notes; no external validation.
  • Training targets come from teacher episodes checked against labels that the benchmark computed from GPT-4 extractions; agreement with the label does not mean clinical correctness.
  • The effect of coordination at a fixed model size is not isolated (no trained single-agent 4B control).
  • A GRPO stage on the sub-agent role was tried and lowered accuracy (80.9 -> 75.3%); it is not part of this model.

Usage

The model expects the prompts and tools in the repository's harness (src/medswarm/prompts.py), with thinking disabled. With vLLM:

git clone https://github.com/Pires-De-Andrade/multiagent_classification && cd multiagent_classification
pip install -e . vllm
bash scripts/fetch_data.sh          # MedCalc-Bench Verified + official calculator code
python -m medswarm.oracle           # label-consistency flags used by the harness
vllm serve jgpandrade18/vanka_v01 --served-model-name jgpandrade18/vanka_v01 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --language-model-only
python -m medswarm.harness --mode orchestrator --model jgpandrade18/vanka_v01 --out runs/vanka.jsonl
python -m medswarm.demo runs/vanka.jsonl --row <row>   # agent tree of one episode

adapter/ holds the LoRA adapter for Qwen/Qwen3.5-4B (use it with the base model's chat template).

Training

  • Data: 1,081 correct teacher episodes on MedCalc-Bench train (2,426 rows; capped at 200 episodes per calculator; two rejection-sampling passes on rows the teacher first got wrong) → 4,228 samples (orchestrator tool calls and sub-agent answers; assistant-only loss).
  • LoRA (rsLoRA) r=64, alpha=16, all linear layers; lr 2e-4, 1 epoch, effective batch 16, bf16, Liger kernels, gradient checkpointing. One RTX 4090: 46 min, 13.2 GiB peak VRAM.

Citation

MedCalc-Bench: Khandekar et al., "MedCalc-Bench: Evaluating Large Language Models for Medical Calculations", NeurIPS 2024 Datasets and Benchmarks.

Downloads last month
37
Safetensors
Model size
4B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jgpandrade18/vanka_v01

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(1,538)
this model

Dataset used to train jgpandrade18/vanka_v01