ParametricSkills

ParametricSkills encodes a textual skill into a LoRA adapter for a large language model (LLM). The adapter enables the model to apply the skill through its parameters, without explicitly including the skill text in the task context.

This release provides SkillEval inference and evaluation for the released Qwen3.5-4B and Qwen3.5-9B components. It covers 100 tasks: 30 single-skill (K1), 46 two-skill (K2), and 24 three-skill (K3) tasks. The default evaluation runs four independent passes and reports the overall and K1/K2/K3 means and sample standard deviations.

1. Installation

Create the encoder and launcher environment from the repository root. Agent execution requires Linux x86_64, Docker, and an OpenCode v1.15.13 binary.

cd inference
python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python build_agent.py

The model service uses a separate environment. The tested stack is vLLM 0.20.1rc1.dev91+ga749a33d8.cu129, PyTorch 2.11.0+cu129, and Transformers 5.7.0. See the runtime guide for local checkpoints, optional CUDA kernels, and offline setup.

2. SkillEval inference

Each task supplies its original instruction, environment inputs, and one to three skill documents. The encoder generates one task-specific rank-8 BF16 LoRA. Multiple documents are concatenated in task-manifest order before encoding; the agent sees the task inputs and auxiliary skill files, while the skill documents are supplied through the adapter.

From inference/, prepare the pinned benchmark snapshot and export the adapters:

python prepare_skilleval.py
python export_skilleval_adapters.py --model Qwen3.5-9B \
  --output-dir adapters/skilleval-9b

In the serving environment, start the matching base model. Set the same SKILLEVAL_QWEN_API_KEY value in the launcher shell.

export SKILLEVAL_QWEN_API_KEY="$(python -c 'import secrets; print(secrets.token_hex(24))')"
export VLLM_API_KEY="$SKILLEVAL_QWEN_API_KEY"
VLLM_ALLOW_RUNTIME_LORA_UPDATING=True vllm serve Qwen/Qwen3.5-9B \
  --revision c202236235762e1c871ad0ccb60c8ee5ba337b9a \
  --served-model-name qwen3.5-9b --host 127.0.0.1 --port 8000 \
  --tensor-parallel-size 1 --dtype bfloat16 --lora-dtype bfloat16 \
  --gpu-memory-utilization 0.85 --max-model-len 40960 \
  --enable-chunked-prefill --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 --enable-lora --max-lora-rank 8

The server reads authentication from VLLM_API_KEY. The service must see the exported adapters at the same absolute path as the launcher. Encoder export and model serving can run sequentially on one GPU.

The launcher keeps the service key on the host and supplies authentication through its local relay. Task containers receive a non-secret placeholder.

Run the complete evaluation in the launcher environment:

python run_skilleval.py --model Qwen3.5-9B \
  --opencode-bin /path/to/opencode \
  --adapter-registry adapters/skilleval-9b/registry.json \
  --output-dir outputs/skilleval-9b

This executes 100 tasks in each of four independent passes. Each task uses a fresh workspace and OpenCode session, and its adapter is loaded and unloaded for that session. For a small smoke run, add --only TASK_ID --repetitions 1 and use an adapter export containing that task. Exact task IDs are in .cache/skilleval-v1/manifests/tasks.jsonl.

Both models use the same commands. For 4B, select --model Qwen3.5-4B, serve Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, and set --served-model-name qwen3.5-4b.

The shared agent settings follow Appendix H.7: OpenCode v1.15.13, 40,960 context tokens, 8,192 output tokens, 20 steps, and an 800-second timeout. Each task container receives 2 CPU cores, 4 GiB memory, and 256 PIDs. Generation uses temperature 0.7 without an explicit seed override.

3. Evaluation

The runner applies each task's official evaluator and saves its outputs and trace. After four complete 100-task passes, it writes the aggregate report automatically.

Output under outputs/skilleval-9b/	Contents
run-N/tasks/<task-id>/artifacts/	Files generated by the agent, such as the task's requested JSON or code
run-N/tasks/<task-id>/task_result.json	Task execution status, official score, and adapter lifecycle
run-N/summary.json	Overall and K1/K2/K3 scores for one pass
evaluation.json	Four-pass mean, sample standard deviation, and individual pass scores

Recompute the aggregate report from existing passes:

python aggregate_skilleval.py \
  --runs outputs/skilleval-9b/run-1 outputs/skilleval-9b/run-2 \
         outputs/skilleval-9b/run-3 outputs/skilleval-9b/run-4 \
  --output outputs/skilleval-9b/evaluation.json

no_skill and icl controls use the same task set and budgets. Run run_skilleval.py --condition no_skill or --condition icl with the model, OpenCode, and a new output directory; omit --adapter-registry. ICL makes the textual skill documents available to the agent.

Raw run directories are private and retain the exact configuration needed for scoring and resume. Before sharing logs or reports, export a separate redacted copy:

python export_results.py --input-dir outputs/skilleval-9b \
  --output-dir share/skilleval-9b

The export excludes task artifacts, weights, and private state. Use the original run directory for scoring and resume; see the runtime guide for additional redaction options.

Agent errors and timeouts remain recorded with zero reward; infrastructure failures prevent aggregation. A finished execution does not guarantee a correct answer. See validation notes for the scope of testing.

License and acknowledgments

See LICENSE and THIRD_PARTY_NOTICES.md. ParametricSkills builds on SHINE, uses Qwen base models, and evaluates tasks from SkillEval.