README.md

---
library_name: transformers
license: mit
datasets:
- allenai/peer_read
language:
- en
metrics:
- accuracy
- f1
---

# Model Card for PaperPub

*Paper pub*lication prediction based on English computer science abstracts.

## Model Details

### Model Description

PaperPub is a SciBERT ([Beltagy et al 2019](https://arxiv.org/abs/1903.10676)) model fine-tuned to predict paper acceptance from computer science abstracts. 
Acceptance is modeled as a binary decision of accept or reject.
The training and evaluation data is based on the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). 
Our main use case for PaperPub is to research how attribution scores derived from acceptance predictions can inform reflecting about content and writing quality of abstracts. 

- **Developed by:** Semantic Computing Group, Bielefeld University, in particular Jan-Philipp Töberg, Christoph Düsing, Jonas Belouadi and Matthias Orlikowski
- **Model type:** BERT for binary classification
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model:** SciBERT

### Model Sources

We will add a public demo of PaperPub used in an application which uses attribution scores to highlight words in an abstract that contribute to acceptance/rejection predcitions.

- **Repository:** tba
- **Demo:** tba

## Uses

PaperPub can only be meaningfully used in a research setting. The model should not be used for any consequential paper quality judgements.

### Direct Use

The intended use case in research into how attribution scores computed from paper acceptance decisions reflect the abstract's content quality.

### Out-of-Scope Use

This model must not be used as part of any type of paper quality judgements, but in particular not in a peer review process. PaperPub is explicitly not meant to automate paper acceptance decisions.

## Bias, Risks, and Limitations

Bias, Risks, and Limitations are mainly related to the used datset. In addition to limitations that apply to the SciBERT pre-training corpus, our training data represents
only a very specific subset of papers. PaperPub was trained in a hackathon-like setting, so performance is not optimized and not our main goal.

### Recommendations

Users should be aware that the dataset (computer science arXive preprints from a specific period) used for fine-tuning represents a very specific idea of what papers 
and in particular papers fit for publication look like.

## How to Get Started with the Model

tba

## Training Details

### Training Data

Custom stratified split of the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). We use the data from their GitHub repository, not the Huggingface Hub version.

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->


#### Training Hyperparameters

- **Training regime:** bf16 mixed precision
- **Epochs:** 2
- **Initial Learning Rate:** 2^-5
 
## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

Custom stratified split of the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). We use the data from their GitHub repository, not the Huggingface Hub version.

#### Factors

Models, we compare to a naive most-frequent-class baseline.

#### Metrics

Accuracy, Macro F1

### Results

- Majority Baseline
  - Acc. - 0.75
  - Macro F1 - 0.43
- PaperPub
  - Acc. - 0.82
  - Macro F1 - 0.76


## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

- **Hardware Type:** 1xA40
- **Hours used:** 0.3
- **Cloud Provider:** Private Infrastructure
- **Compute Region:** Europe

## Technical Specifications

### Compute Infrastructure

We are using an internal SLURM cluster with A40 GPUs

## Citation

tba

## Model Card Contact

[Matthias Orlikowski](https://orlikow.ski)