Qwen3-4B-LoRA-Cochrane-Screening

LoRA adapter for Cochrane-style title/abstract screening, fine-tuned on top of Qwen/Qwen3-4B.

This repository contains adapter weights only. You still need the original Qwen3-4B base model.

Dataset: deepcoder2024/cochrane-screening-sft
Code: ljwa2323/cochrane-screening-slm

Training summary

Item	Value
Base model	`Qwen/Qwen3-4B`
Method	LoRA (PEFT)
LoRA r / alpha / dropout	16 / 32 / 0.05
Target modules	q/k/v/o/gate/up/down proj
Max length	2048
Epochs	1.0
Learning rate	2e-4
Effective batch size	1 per device x 2 GPUs x 16 grad accum = 32
Train loss	0.3521
Eval loss	0.2972
Train data	`cochrane-screening-sft` train split
Task output	JSON `{"label","reason"}` with labels `include`/`exclude`/`uncertain`

Files

File	Description
`adapter_model.safetensors`	LoRA weights
`adapter_config.json`	LoRA config
tokenizer files	Tokenizer / chat template from the training run
`run_args.json`	Training hyperparameters
`all_results.json`	Final train/eval metrics

Load and run

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER_ID = "deepcoder2024/Qwen3-4B-LoRA-Cochrane-Screening"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER_ID)
model.eval()

messages = [
    {
        "role": "system",
        "content": (
            "You are an expert systematic reviewer performing title and abstract screening.\n"
            "Given the review Selection_criteria, the study Title, and the Abstract, "
            "decide whether the study should be included.\n\n"
            "Labels:\n"
            "- include: clearly meets selection criteria\n"
            "- exclude: clearly does not meet selection criteria\n"
            "- uncertain: insufficient information to decide\n\n"
            "Respond with ONLY a JSON object in this exact format:\n"
            '{"label": "include" | "exclude" | "uncertain", "reason": "<brief explanation>"}\n'
            "Do not output any other text."
        ),
    },
    {
        "role": "user",
        "content": (
            "Selection_criteria:\n...\n\n"
            "Title:\n...\n\n"
            "Abstract:\n...\n\n"
            "Decide the screening label and provide a brief reason."
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))