README.md

---
library_name: peft
base_model: Qwen/Qwen3-4B
license: other
tags:
- qwen3
- lora
- peft
- systematic-review
- cochrane
- title-abstract-screening
- medical
language:
- en
pipeline_tag: text-generation
---

# Qwen3-4B-LoRA-Cochrane-Screening

LoRA adapter for **Cochrane-style title/abstract screening**, fine-tuned on top of [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B).

This repository contains **adapter weights only**. You still need the original Qwen3-4B base model.

- Dataset: [`deepcoder2024/cochrane-screening-sft`](https://huggingface.co/datasets/deepcoder2024/cochrane-screening-sft)
- Code: [ljwa2323/cochrane-screening-slm](https://github.com/ljwa2323/cochrane-screening-slm)

## Training summary

| Item | Value |
| --- | --- |
| Base model | `Qwen/Qwen3-4B` |
| Method | LoRA (PEFT) |
| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q/k/v/o/gate/up/down proj |
| Max length | 2048 |
| Epochs | 1.0 |
| Learning rate | 2e-4 |
| Effective batch size | 1 per device x 2 GPUs x 16 grad accum = 32 |
| Train loss | 0.3521 |
| Eval loss | 0.2972 |
| Train data | `cochrane-screening-sft` train split |
| Task output | JSON `{"label","reason"}` with labels `include`/`exclude`/`uncertain` |

## Files

| File | Description |
| --- | --- |
| `adapter_model.safetensors` | LoRA weights |
| `adapter_config.json` | LoRA config |
| tokenizer files | Tokenizer / chat template from the training run |
| `run_args.json` | Training hyperparameters |
| `all_results.json` | Final train/eval metrics |

## Load and run

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER_ID = "deepcoder2024/Qwen3-4B-LoRA-Cochrane-Screening"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER_ID)
model.eval()

messages = [
    {
        "role": "system",
        "content": (
            "You are an expert systematic reviewer performing title and abstract screening.\n"
            "Given the review Selection_criteria, the study Title, and the Abstract, "
            "decide whether the study should be included.\n\n"
            "Labels:\n"
            "- include: clearly meets selection criteria\n"
            "- exclude: clearly does not meet selection criteria\n"
            "- uncertain: insufficient information to decide\n\n"
            "Respond with ONLY a JSON object in this exact format:\n"
            '{"label": "include" | "exclude" | "uncertain", "reason": "<brief explanation>"}\n'
            "Do not output any other text."
        ),
    },
    {
        "role": "user",
        "content": (
            "Selection_criteria:\n...\n\n"
            "Title:\n...\n\n"
            "Abstract:\n...\n\n"
            "Decide the screening label and provide a brief reason."
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Intended use

Research / assistance for systematic-review title and abstract screening.
Not a substitute for expert reviewer judgment or clinical decision-making.

## Framework versions

- transformers
- peft >= 0.19
- torch