Update README.md
Browse files
README.md
CHANGED
|
@@ -11,11 +11,13 @@ datasets:
|
|
| 11 |
- openwebtext
|
| 12 |
---
|
| 13 |
# 一、 个人在openwebtext数据集上添加rotary-position-embedding，训练得到的electra-small模型
|
|
|
|
| 14 |
# 二、 复现结果(dev dataset)
|
| 15 |
|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
|
| 16 |
|---|---|---|---|---|---|---|---|---|---|
|
| 17 |
|ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
|
| 18 |
-
|**ELECTRA-Small-OWT (this)**|
|
|
|
|
| 19 |
# 三、 训练细节
|
| 20 |
- 数据集 openwebtext
|
| 21 |
- 训练batch_size 256
|
|
@@ -26,7 +28,9 @@ datasets:
|
|
| 26 |
- 训练时间总共耗费55h
|
| 27 |
|
| 28 |
### wandb日志
|
| 29 |
-
https://wandb.ai/junyu/electra_rotary_small_pretrain?workspace=user-junyu
|
|
|
|
|
|
|
| 30 |
|
| 31 |
# 四、安装
|
| 32 |
```bash
|
|
|
|
| 11 |
- openwebtext
|
| 12 |
---
|
| 13 |
# 一、 个人在openwebtext数据集上添加rotary-position-embedding，训练得到的electra-small模型
|
| 14 |
+
|
| 15 |
# 二、 复现结果(dev dataset)
|
| 16 |
|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
|
| 17 |
|---|---|---|---|---|---|---|---|---|---|
|
| 18 |
|ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
|
| 19 |
+
|**ELECTRA-RoFormer-Small-OWT (this)**|55.76|90.45|87.3|86.64|89.61|81.17|88.85|62.71|80.31|
|
| 20 |
+
|
| 21 |
# 三、 训练细节
|
| 22 |
- 数据集 openwebtext
|
| 23 |
- 训练batch_size 256
|
|
|
|
| 28 |
- 训练时间总共耗费55h
|
| 29 |
|
| 30 |
### wandb日志
|
| 31 |
+
[预训练日志](https://wandb.ai/junyu/electra_rotary_small_pretrain?workspace=user-junyu)
|
| 32 |
+
|
| 33 |
+
[GLUE微调日志](https://wandb.ai/junyu/electra_rotary_glue_100?workspace=user-junyu)
|
| 34 |
|
| 35 |
# 四、安装
|
| 36 |
```bash
|