File size: 3,651 Bytes
25ca8d6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
library_name: peft
base_model: Qwen/Qwen3-4B
license: other
tags:
- qwen3
- lora
- peft
- systematic-review
- cochrane
- title-abstract-screening
- medical
language:
- en
pipeline_tag: text-generation
---
# Qwen3-4B-LoRA-Cochrane-Screening
LoRA adapter for **Cochrane-style title/abstract screening**, fine-tuned on top of [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B).
This repository contains **adapter weights only**. You still need the original Qwen3-4B base model.
- Dataset: [`deepcoder2024/cochrane-screening-sft`](https://huggingface.co/datasets/deepcoder2024/cochrane-screening-sft)
- Code: [ljwa2323/cochrane-screening-slm](https://github.com/ljwa2323/cochrane-screening-slm)
## Training summary
| Item | Value |
| --- | --- |
| Base model | `Qwen/Qwen3-4B` |
| Method | LoRA (PEFT) |
| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q/k/v/o/gate/up/down proj |
| Max length | 2048 |
| Epochs | 1.0 |
| Learning rate | 2e-4 |
| Effective batch size | 1 per device x 2 GPUs x 16 grad accum = 32 |
| Train loss | 0.3521 |
| Eval loss | 0.2972 |
| Train data | `cochrane-screening-sft` train split |
| Task output | JSON `{"label","reason"}` with labels `include`/`exclude`/`uncertain` |
## Files
| File | Description |
| --- | --- |
| `adapter_model.safetensors` | LoRA weights |
| `adapter_config.json` | LoRA config |
| tokenizer files | Tokenizer / chat template from the training run |
| `run_args.json` | Training hyperparameters |
| `all_results.json` | Final train/eval metrics |
## Load and run
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER_ID = "deepcoder2024/Qwen3-4B-LoRA-Cochrane-Screening"
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER_ID)
model.eval()
messages = [
{
"role": "system",
"content": (
"You are an expert systematic reviewer performing title and abstract screening.\n"
"Given the review Selection_criteria, the study Title, and the Abstract, "
"decide whether the study should be included.\n\n"
"Labels:\n"
"- include: clearly meets selection criteria\n"
"- exclude: clearly does not meet selection criteria\n"
"- uncertain: insufficient information to decide\n\n"
"Respond with ONLY a JSON object in this exact format:\n"
'{"label": "include" | "exclude" | "uncertain", "reason": "<brief explanation>"}\n'
"Do not output any other text."
),
},
{
"role": "user",
"content": (
"Selection_criteria:\n...\n\n"
"Title:\n...\n\n"
"Abstract:\n...\n\n"
"Decide the screening label and provide a brief reason."
),
},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
## Intended use
Research / assistance for systematic-review title and abstract screening.
Not a substitute for expert reviewer judgment or clinical decision-making.
## Framework versions
- transformers
- peft >= 0.19
- torch
|