Commit ·
87a8dbd
1
Parent(s): 2476254
Upload 10 files
Browse files
README.md
CHANGED
|
@@ -141,14 +141,13 @@ Parameters of the fit()-Method:
|
|
| 141 |
"lr": 4e-05
|
| 142 |
},
|
| 143 |
"scheduler": "WarmupLinear",
|
| 144 |
-
"warmup_steps": 66
|
| 145 |
-
"weight_decay": 0.06
|
| 146 |
}
|
| 147 |
```
|
| 148 |
|
| 149 |
## Evaluation
|
| 150 |
|
| 151 |
-
We applied a 6-fold (
|
| 152 |
described in the *Training* section, where we trained on k-1 splits and made predictions using the kth split.
|
| 153 |
|
| 154 |
Additionally, we assessed its ability to generalize across the 6 topics (Cross-Topic) of TACO. Each of the k topics was utilized for testing, while
|
|
@@ -156,19 +155,19 @@ the remaining k-1 topics were used for training purposes.
|
|
| 156 |
|
| 157 |
In total, the WRAP classifier performs as follows:
|
| 158 |
|
| 159 |
-
###
|
| 160 |
|
| 161 |
-
| Macro-F1
|
| 162 |
-
|
| 163 |
-
|
|
| 164 |
-
| Cross-Topic
|
| 165 |
|
| 166 |
-
### Classification
|
| 167 |
|
| 168 |
-
| Micro-F1
|
| 169 |
-
|
| 170 |
-
|
|
| 171 |
-
| Cross-Topic
|
| 172 |
|
| 173 |
# Environmental Impact
|
| 174 |
|
|
|
|
| 141 |
"lr": 4e-05
|
| 142 |
},
|
| 143 |
"scheduler": "WarmupLinear",
|
| 144 |
+
"warmup_steps": 66
|
|
|
|
| 145 |
}
|
| 146 |
```
|
| 147 |
|
| 148 |
## Evaluation
|
| 149 |
|
| 150 |
+
We applied a 6-fold (Closed-Topic) cross-validation method to demonstrate WRAP's optimal performance. This involved using the same dataset and parameters
|
| 151 |
described in the *Training* section, where we trained on k-1 splits and made predictions using the kth split.
|
| 152 |
|
| 153 |
Additionally, we assessed its ability to generalize across the 6 topics (Cross-Topic) of TACO. Each of the k topics was utilized for testing, while
|
|
|
|
| 155 |
|
| 156 |
In total, the WRAP classifier performs as follows:
|
| 157 |
|
| 158 |
+
### Binary Classification Tasks
|
| 159 |
|
| 160 |
+
| Macro-F1 | Inference | Information | Multi-Class |
|
| 161 |
+
|--------------|-----------|-------------|-------------|
|
| 162 |
+
| Closed-Topic | 86.62% | 86.30% | 75.29% |
|
| 163 |
+
| Cross-Topic | 86.27% | 84.90% | 73.54% |
|
| 164 |
|
| 165 |
+
### Multi-Class Classification Task
|
| 166 |
|
| 167 |
+
| Micro-F1 | Reason | Statement | Notification | None |
|
| 168 |
+
|--------------|--------|-----------|--------------|--------|
|
| 169 |
+
| Closed-Topic | 78.14% | 60.96% | 79.36% | 82.72% |
|
| 170 |
+
| Cross-Topic | 77.05% | 58.33% | 78.45% | 80.33% |
|
| 171 |
|
| 172 |
# Environmental Impact
|
| 173 |
|