README.md

---
license: apache-2.0
base_model: google/gemma-2-2b
tags:
- peft
- lora
- text-generation
- summarization
- scientific-lay-summarization
---

# Model Card for gemma-2-2b-scientific-summarizer

This is a Parameter-Efficient Fine-Tuning (PEFT) LoRA adapter for `google/gemma-2-2b` optimized for scientific lay summarization and key-point extraction.

## Model Details

### Model Description
- **Developed by:** coder1969
- **Model type:** PEFT (LoRA) adapter for Causal Language Modeling
- **Language(s) (NLP):** English
- **License:** Apache-2.0
- **Finetuned from model:** `google/gemma-2-2b`

## Uses

### Direct Use
This model is directly intended for taking scientific abstract/literature context and generating structured lay summaries or key points.

### Out-of-Scope Use
- Clinical or medical diagnostics without peer review.
- Automated code generation or general chatbot applications.

### Bias, Risks, and Limitations
Users should be aware that language models can hallucinate or omit critical details from context. Outputs should be verified against original sources.

## How to Get Started with the Model

Use the code below to load the base model and apply the fine-tuned adapter weights:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_name = "google/gemma-2-2b"
adapter_model_name = "coder1969/gemma-2-2b-scientific-summarizer"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Apply adapters
model = PeftModel.from_pretrained(model, adapter_model_name)

# Inference Example
prompt = "Document:\nTopic: quantum machine learning\n\nRelevant Literature:\n[Insert relevant abstracts or papers here]\n\nSummary:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

### Training Data
Finetuned on the `tomasg25/scientific_lay_summarisation` dataset (subset: `plos`), containing pairs of scientific articles and summaries.

### Training Hyperparameters
- **LoRA Config:**
  - Rank (r): 8
  - Alpha: 16
  - Dropout: 0.1
  - Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- **Optimization parameters:**
  - Learning Rate: 2e-05
  - Batch Size: 8
  - Gradient Accumulation Steps: 4
  - Epochs: 3
  - Weight Decay: 0.01
  - Precision: Mixed precision (FP16)

## Framework versions
- PEFT 0.19.1
- Transformers 4.40.0+
- PyTorch 2.0+