metadata
license: mit
base_model: deepset/gbert-large
tags:
- generated_from_trainer
model-index:
- name: gbert-large-autopart
results: []
language:
- de
gbert-large-autopart
This model is a fine-tuned version of deepset/gbert-large on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.3832
Model description
this model contains a domain adaptation of the model deepset/gbert-large using a dataset containing 54.000 sample sentences of german top 30 DAX company websites.
Intended uses & limitations
use in classification problems where the samples are from german company websites.
Training and evaluation data
80 percent of the available samples where used for training. Evaluation was performed on 20 percent of the data.
Training procedure
masked language modelling using AutoModelForMaskedLM
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 96
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 16
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.5949 | 1.0 | 445 | 0.5025 |
| 0.5211 | 2.0 | 890 | 0.4729 |
| 0.5036 | 3.0 | 1335 | 0.4893 |
| 0.4916 | 4.0 | 1780 | 0.4647 |
| 0.4464 | 5.0 | 2225 | 0.4401 |
| 0.425 | 6.0 | 2670 | 0.4246 |
| 0.4076 | 7.0 | 3115 | 0.4169 |
| 0.3962 | 8.0 | 3560 | 0.4140 |
| 0.3829 | 9.0 | 4005 | 0.4220 |
| 0.3702 | 10.0 | 4450 | 0.4119 |
| 0.3566 | 11.0 | 4895 | 0.3993 |
| 0.3442 | 12.0 | 5340 | 0.3924 |
| 0.3365 | 13.0 | 5785 | 0.3880 |
| 0.3316 | 14.0 | 6230 | 0.3900 |
| 0.3213 | 15.0 | 6675 | 0.3800 |
| 0.316 | 16.0 | 7120 | 0.3832 |
Framework versions
- Transformers 4.32.1
- Pytorch 2.0.1+cu118
- Datasets 2.14.4
- Tokenizers 0.13.3