Instructions to use atlasmarkz/microduck-skill-agent with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use atlasmarkz/microduck-skill-agent with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
microduck skill agent: LoRA adapters for Qwen3-VL-4B-Instruct
LoRA adapters that turn Qwen3-VL-4B-Instruct into a skill-level policy for a small legged robot ("microduck") in MuJoCo
multi-room worlds. At each step the model sees a camera image and a natural-language instruction (Chinese or English) and outputs
one skill as JSON: move, turn, approach, search, explore, return_home, stop, done, clarify, unsupported,
bypass_return. The skills are executed by rule-based code in the project repository (depth-built occupancy map, A*, frontier
exploration). The adapter alone is not a robot controller: the prompt format, executor feedback and skill executors live in
the code repository: https://github.com/atlasmarkz/microduck-qwen (microduck_qwen/agent.py, microduck_qwen/agent_eval.py).
Simulation only. Never run on a physical robot.
Files
| Folder | Status |
|---|---|
lang-v1/ |
earlier version |
lang-v2/ |
adopted; the version behind the main numbers below |
lang-v3/ |
rejected (worse than lang-v2) |
lang-v4/ |
not adopted (no measurable gain over lang-v2) |
The repository root holds only this model card; every adapter is loaded with subfolder=. Each folder holds adapter_config.json, adapter_model.safetensors and training_report.json (hyper-parameters, row counts,
dataset and adapter hashes, loss history).
Training
Supervised fine-tuning on rule-teacher labels (a teacher that reads simulator ground truth and corrects recorded rollouts).
LoRA rank 8, alpha 16, dropout 0, on the q/v projections of the last 8 language-model layers (655,360 trainable parameters);
300 steps, batch size 1, learning rate 5e-5, seed 42; base model loaded in 4-bit NF4 (double quantisation, BF16 compute);
image pixels 65,536-131,072; max sequence length 1,024; completion-only loss. Base revision ebb281ec70b05090aa6165b016eac8ec08e71b17.
Training rows: lang-v1 1,634; lang-v2 3,074; lang-v3 3,866; lang-v4 4,075. Training data is not released here.
Software: torch 2.11.0, transformers 5.19.0, peft 0.21.2, bitsandbytes 0.50.2. One RTX 3060 (8 GiB), about 13 minutes per adapter.
Evaluation (read the caveats)
End-to-end rollouts in the simulator: 8 unseen layouts per seed group, tasks "approach the leftmost ball (and return)", "move
forward 20 cm", and 3 explore-and-approach tasks per layout. Tasks the evaluation harness could not label are excluded.
The strict score requires the exact skill the rule teacher expects at every step; the outcome-equivalent score also counts
done where the teacher expects stop after all motion succeeded (same physical outcome).
| Adapter | Layouts | Strict | Outcome-equivalent |
|---|---|---|---|
| lang-v2 | seeds 99000, 99100 (never used for any decision) | 61/74 (82.4%) | 65/74 (87.8%) |
| lang-v4 | same | 60/74 (81.1%) | 64/74 (86.5%) |
| lang-v2 vs lang-v3 | seeds 98000, 98100 (development layouts) | 69/80 vs 66/80 | - |
| lang-v1 | seeds 98000, 98100, older executor | 52/80 | - |
Caveats:
- These numbers measure the whole system (adapter plus the repository's executors at that commit), not the adapter alone.
- Identical re-runs flip roughly 10 of 73 tasks, so differences of a few tasks are noise; lang-v2 vs lang-v4 is p = 1.0 (paired exact McNemar).
- Scoring is against a rule teacher written for this project; it is not an external benchmark.
- One robot, one simulator, one base model. No claim of transfer to real robots or other environments.
Usage
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel
base = "Qwen/Qwen3-VL-4B-Instruct"
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForImageTextToText.from_pretrained(base, quantization_config=quant, device_map="auto")
model = PeftModel.from_pretrained(model, "atlasmarkz/microduck-skill-agent", subfolder="lang-v2") # or lang-v1, lang-v3, lang-v4
processor = AutoProcessor.from_pretrained(base)
Build the prompt with the repository's code; free-form prompts will not produce the trained skill format reliably.
License
Apache-2.0, the same as the base model Qwen3-VL-4B-Instruct.
- Downloads last month
- 1
Model tree for atlasmarkz/microduck-skill-agent
Base model
Qwen/Qwen3-VL-4B-Instruct