---
library_name: transformers
base_model: allenai/scibert_scivocab_uncased
tags:
- generated_from_trainer
model-index:
- name: ContriBERT-ACL
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# ContriBERT-ACL

This model is a fine-tuned version of [allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased) on [taln-ls2n/ARRContributions](https://huggingface.co/datasets/taln-ls2n/ARRContributions). It achieves the following results on the evaluation sets:

| Evaluation Set | F1 Micro | F1 Macro | Loss |
|:-------------: |:--------:|:--------:|:----:|
| Author-Annotated        | 0.5804   | 0.3775   |0.3232 |
| Expert-Annotated        | 0.6542   | 0.3982   |0.2682 |

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 0
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 20
- early_stopping_patience: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1 Micro | F1 Macro |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|
| 0.4687        | 1.0   | 51   | 0.3894          | 0.3497   | 0.0617   |
| 0.3926        | 2.0   | 102  | 0.3627          | 0.4771   | 0.1660   |
| 0.3624        | 3.0   | 153  | 0.3412          | 0.5007   | 0.1739   |
| 0.3444        | 4.0   | 204  | 0.3302          | 0.5205   | 0.1841   |
| 0.328         | 5.0   | 255  | 0.3234          | 0.5365   | 0.2173   |
| 0.3127        | 6.0   | 306  | 0.3196          | 0.5447   | 0.2463   |
| 0.2989        | 7.0   | 357  | 0.3244          | 0.5457   | 0.2639   |
| 0.2848        | 8.0   | 408  | 0.3177          | 0.5596   | 0.3584   |
| 0.2719        | 9.0   | 459  | 0.3171          | 0.5688   | 0.3540   |
| 0.2627        | 10.0  | 510  | 0.3192          | 0.5763   | 0.3616   |
| 0.2502        | 11.0  | 561  | 0.3194          | 0.5818   | 0.3868   |
| 0.2405        | 12.0  | 612  | 0.3246          | 0.5788   | 0.3713   |
| 0.2337        | 13.0  | 663  | 0.3195          | 0.5844   | 0.3928   |
| 0.227         | 14.0  | 714  | 0.3262          | 0.5696   | 0.3898   |
| 0.2192        | 15.0  | 765  | 0.3227          | 0.5796   | 0.3792   |
| 0.2145        | 16.0  | 816  | 0.3250          | 0.5751   | 0.3779   |
| 0.2096        | 17.0  | 867  | 0.3256          | 0.5773   | 0.3776   |
| 0.2074        | 18.0  | 918  | 0.3277          | 0.5740   | 0.3792   |


### Framework versions

- Transformers 4.57.1
- Pytorch 2.6.0+cu124
- Datasets 3.5.0
- Tokenizers 0.22.1