tapt_helpfulness_seq_bn_pretraining_model_full_train

README.md

@@ -0,0 +1,78 @@
+---
+license: mit
+base_model: roberta-base
+tags:
+- generated_from_trainer
+model-index:
+- name: tapt_helpfulness_seq_bn_pretraining_model_full_train
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# tapt_helpfulness_seq_bn_pretraining_model_full_train
+
+This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.4917
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 21
+- eval_batch_size: 21
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 42
+- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
+- lr_scheduler_type: linear
+- num_epochs: 20
+
+### Training results
+
+| Training Loss | Epoch | Step  | Validation Loss |
+|:-------------:|:-----:|:-----:|:---------------:|
+| 2.746         | 1.0   | 1068  | 1.8309          |
+| 1.8701        | 2.0   | 2137  | 1.6877          |
+| 1.7711        | 3.0   | 3205  | 1.6275          |
+| 1.7178        | 4.0   | 4274  | 1.5909          |
+| 1.6876        | 5.0   | 5342  | 1.5788          |
+| 1.6638        | 6.0   | 6411  | 1.5636          |
+| 1.6526        | 7.0   | 7479  | 1.5344          |
+| 1.6357        | 8.0   | 8548  | 1.5402          |
+| 1.626         | 9.0   | 9616  | 1.5097          |
+| 1.6144        | 10.0  | 10685 | 1.5111          |
+| 1.611         | 11.0  | 11753 | 1.5248          |
+| 1.603         | 12.0  | 12822 | 1.4989          |
+| 1.6003        | 13.0  | 13890 | 1.5071          |
+| 1.5915        | 14.0  | 14959 | 1.4807          |
+| 1.5893        | 15.0  | 16027 | 1.4892          |
+| 1.5857        | 16.0  | 17096 | 1.4794          |
+| 1.5839        | 17.0  | 18164 | 1.4893          |
+| 1.5806        | 18.0  | 19233 | 1.4787          |
+| 1.5808        | 19.0  | 20301 | 1.4872          |
+| 1.5781        | 19.99 | 21360 | 1.4917          |
+
+
+### Framework versions
+
+- Transformers 4.36.2
+- Pytorch 2.2.1+cu121
+- Datasets 2.19.0
+- Tokenizers 0.15.2

default/head_config.json

@@ -0,0 +1,19 @@
+{
+  "config": {
+    "activation_function": "gelu",
+    "bias": true,
+    "embedding_size": 768,
+    "head_type": "masked_lm",
+    "label2id": null,
+    "layer_norm": true,
+    "layers": 2,
+    "shift_labels": false,
+    "vocab_size": 50265
+  },
+  "hidden_size": 768,
+  "model_class": "RobertaAdapterModel",
+  "model_name": "roberta-base",
+  "model_type": "roberta",
+  "name": "default",
+  "version": "0.1.2"
+}

default/pytorch_model_head.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdff7928366b173ef2d74fcdf04581a7c98c13c3bd11623203f922ffcc7248c5
+size 156986358

runs/Apr20_08-00-39_ea01de983f1c/events.out.tfevents.1713600064.ea01de983f1c.331.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b9d895ba8b11cba01bcceee8675bf73dbe48d5cccec2bddd65ba50400ce1cf89
-size 13691
+oid sha256:34aa44f000d152982ba70f2d2a11348aea6a19f990fe438c32a20c99fbd60553
+size 14051

tapt_seq_bn/adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "config": {
+    "adapter_residual_before_ln": false,
+    "cross_adapter": false,
+    "factorized_phm_W": true,
+    "factorized_phm_rule": false,
+    "hypercomplex_nonlinearity": "glorot-uniform",
+    "init_weights": "bert",
+    "inv_adapter": null,
+    "inv_adapter_reduction_factor": null,
+    "is_parallel": false,
+    "learn_phm": true,
+    "leave_out": [],
+    "ln_after": false,
+    "ln_before": false,
+    "mh_adapter": false,
+    "non_linearity": "relu",
+    "original_ln_after": true,
+    "original_ln_before": true,
+    "output_adapter": true,
+    "phm_bias": true,
+    "phm_c_init": "normal",
+    "phm_dim": 4,
+    "phm_init_range": 0.0001,
+    "phm_layer": false,
+    "phm_rank": 1,
+    "reduction_factor": 16,
+    "residual_before_ln": true,
+    "scaling": 1.0,
+    "shared_W_phm": false,
+    "shared_phm_rule": true,
+    "use_gating": false
+  },
+  "config_id": "9076f36a74755ac4",
+  "hidden_size": 768,
+  "model_class": "RobertaAdapterModel",
+  "model_name": "roberta-base",
+  "model_type": "roberta",
+  "name": "tapt_seq_bn",
+  "version": "0.1.2"
+}

tapt_seq_bn/head_config.json

@@ -0,0 +1,19 @@
+{
+  "config": {
+    "activation_function": "gelu",
+    "bias": true,
+    "embedding_size": 768,
+    "head_type": "masked_lm",
+    "label2id": null,
+    "layer_norm": true,
+    "layers": 2,
+    "shift_labels": false,
+    "vocab_size": 50265
+  },
+  "hidden_size": 768,
+  "model_class": "RobertaAdapterModel",
+  "model_name": "roberta-base",
+  "model_type": "roberta",
+  "name": "tapt_seq_bn",
+  "version": "0.1.2"
+}

tapt_seq_bn/pytorch_adapter.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbb07b225312caf229bb05609beadcd15ceada2ba95a20b1a5fb91344d79062e
+size 3595878

tapt_seq_bn/pytorch_model_head.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b266dd75adb34eb35c51c73f6f2794af71c5e12ade24ea5bd1483d2c7b914e8
+size 156986358