metadata
library_name: peft
base_model: Qwen/Qwen3-4B
license: other
tags:
- qwen3
- lora
- peft
- systematic-review
- cochrane
- title-abstract-screening
- medical
language:
- en
pipeline_tag: text-generation
Qwen3-4B-LoRA-Cochrane-Screening
LoRA adapter for Cochrane-style title/abstract screening, fine-tuned on top of Qwen/Qwen3-4B.
This repository contains adapter weights only. You still need the original Qwen3-4B base model.
Training summary
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3-4B |
| Method | LoRA (PEFT) |
| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q/k/v/o/gate/up/down proj |
| Max length | 2048 |
| Epochs | 1.0 |
| Learning rate | 2e-4 |
| Effective batch size | 1 per device x 2 GPUs x 16 grad accum = 32 |
| Train loss | 0.3521 |
| Eval loss | 0.2972 |
| Train data | cochrane-screening-sft train split |
| Task output | JSON {"label","reason"} with labels include/exclude/uncertain |
Files
| File | Description |
|---|---|
adapter_model.safetensors |
LoRA weights |
adapter_config.json |
LoRA config |
| tokenizer files | Tokenizer / chat template from the training run |
run_args.json |
Training hyperparameters |
all_results.json |
Final train/eval metrics |
Load and run
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER_ID = "deepcoder2024/Qwen3-4B-LoRA-Cochrane-Screening"
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER_ID)
model.eval()
messages = [
{
"role": "system",
"content": (
"You are an expert systematic reviewer performing title and abstract screening.\n"
"Given the review Selection_criteria, the study Title, and the Abstract, "
"decide whether the study should be included.\n\n"
"Labels:\n"
"- include: clearly meets selection criteria\n"
"- exclude: clearly does not meet selection criteria\n"
"- uncertain: insufficient information to decide\n\n"
"Respond with ONLY a JSON object in this exact format:\n"
'{"label": "include" | "exclude" | "uncertain", "reason": "<brief explanation>"}\n'
"Do not output any other text."
),
},
{
"role": "user",
"content": (
"Selection_criteria:\n...\n\n"
"Title:\n...\n\n"
"Abstract:\n...\n\n"
"Decide the screening label and provide a brief reason."
),
},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Intended use
Research / assistance for systematic-review title and abstract screening. Not a substitute for expert reviewer judgment or clinical decision-making.
Framework versions
- transformers
- peft >= 0.19
- torch