Commit ·
d89dade
1
Parent(s): e785a97
Create README.md
Browse filesSigned-off-by: jinjieyuan <jinjie.yuan@intel.com>
README.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
license: apache-2.0
|
| 4 |
+
datasets:
|
| 5 |
+
- nyu-mll/glue
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# EFTNAS Model Card: eftnas-s2-bert-medium
|
| 9 |
+
|
| 10 |
+
The super-networks fine-tuned on BERT-medium with [GLUE benchmark](https://gluebenchmark.com/) using EFTNAS.
|
| 11 |
+
|
| 12 |
+
## Model Details
|
| 13 |
+
|
| 14 |
+
### Information
|
| 15 |
+
|
| 16 |
+
- **Model name:** eftnas-s2-bert-medium-[TASK]
|
| 17 |
+
- **Base model:** [google/bert_uncased_L-8_H-512_A-8](https://huggingface.co/google/bert_uncased_L-8_H-512_A-8)
|
| 18 |
+
- **Subnetwork version:** Super-network
|
| 19 |
+
- **NNCF Configurations:** [eftnas_configs](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS/eftnas_configs)
|
| 20 |
+
|
| 21 |
+
### Training and Evaluation
|
| 22 |
+
|
| 23 |
+
[GLUE benchmark](https://gluebenchmark.com/)
|
| 24 |
+
|
| 25 |
+
## Results
|
| 26 |
+
|
| 27 |
+
Results of the optimal sub-network discoverd from the super-network:
|
| 28 |
+
|
| 29 |
+
| Model | GFLOPs | GLUE Avg. | MNLI-m | QNLI | QQP | SST-2 | CoLA | MRPC | RTE |
|
| 30 |
+
|-------------------------------|-----------|---------------|----------|------|----------|----------|----------|----------|------|
|
| 31 |
+
| **Test Set:** |
|
| 32 |
+
| [**EFTNAS-S1**]() | 5.7 | 77.7 | 83.7 | 89.9 | 71.8 | 93.4 | 52.6 | 87.6 | 65.0 |
|
| 33 |
+
| [**EFTNAS-S2**]() | 2.2 | 75.2 | 82.0 | 87.8 | 70.6 | 91.4 | 44.5 | 86.1 | 64.0 |
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
## Model Sources
|
| 37 |
+
|
| 38 |
+
- **Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS)
|
| 39 |
+
- **Paper:** [Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks]()
|
| 40 |
+
|
| 41 |
+
## Citation
|
| 42 |
+
|
| 43 |
+
```bibtex
|
| 44 |
+
@inproceedings{
|
| 45 |
+
eftnas2024,
|
| 46 |
+
title={Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks},
|
| 47 |
+
author={J. Pablo Munoz and Yi Zheng and Nilesh Jain},
|
| 48 |
+
booktitle={The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation},
|
| 49 |
+
year={2024},
|
| 50 |
+
url={}
|
| 51 |
+
}
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
## License
|
| 55 |
+
|
| 56 |
+
Apache-2.0
|