btbtyler09/shrew-2b

Shrew LoRA Adapters

LoRA adapters for Qwen/Qwen3.5-2B fine-tuned for structured extraction as part of a production RAG application. These power Shrew's structured extraction pipeline.

Adapters

Adapter	Tasks	LoRA rank / alpha	Status
doc_processing/	extract_metadata, semantic_chunk, summarize_document (3 tasks, single adapter)	r128 / α256	Recommended — unified, supersedes the 3 per-task adapters below
extract_metadata/	Extract structured metadata (title, authors, dates, document type)	r32 / α64	Superseded by doc_processing/
summarize_document/	Generate document summaries	r32 / α64	Superseded by doc_processing/
semantic_chunk/	Split documents into semantically coherent sections	r128 / α256	Superseded by doc_processing/

The unified doc_processing/ adapter routes by system prompt — the prompt is just the task name (extract_metadata, semantic_chunk, or summarize_document). One adapter, three tasks.

doc_processing/ metrics

Base model	Qwen/Qwen3.5-2B
LoRA rank / alpha	128 / 256
Adapter size	~256 MB (f16)
Training corpus	~103k examples across 3 tasks
Epochs	3
Final eval loss	0.7103
Hardware	4× AMD MI100 (gfx908), bf16, DeepSpeed ZeRO-2

Usage

The unified doc_processing/ adapter routes by system prompt. Same adapter, different task per call.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
model = PeftModel.from_pretrained(base, "btbtyler09/shrew-2b", subfolder="doc_processing")

messages = [
    {"role": "system", "content": "extract_metadata"},   # or "semantic_chunk", "summarize_document"
    {"role": "user", "content": document_text},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)

extract_metadata and semantic_chunk produce JSON; summarize_document produces prose.

llama.cpp (GGUF)

GGUF versions can be used as LoRA adapters with llama-cli / llama-server:

llama-server -m Qwen3.5-2B.gguf --lora doc_processing.gguf

vLLM

vLLM loads Qwen3.5 as Qwen3_5ForConditionalGeneration (a VLM class), which nests the language model under a language_model. prefix. The adapters here are saved in standard PEFT format (trained with AutoModelForCausalLM), so the weight keys must be renamed before serving with vLLM's --enable-lora. Without the rename, vLLM silently zeros all LoRA weights with no error.

Apply the rename with fix_lora_keys.py from the shrew repo:

python fix_lora_keys.py path/to/adapter
vllm serve Qwen/Qwen3.5-2B \
  --enable-lora \
  --lora-modules doc_processing=path/to/adapter \
  --max-lora-rank 128 \
  --max-loras 1

Note --max-lora-rank 128 for the unified adapter (vLLM's default of 16 is too low).

Sampling parameters

Use Qwen 3.5 instruct-general parameters with enable_thinking=False:

param	value
temperature	0.7
top_p	0.8
top_k	20

License

Same as base model (Qwen/Qwen3.5-2B).