README.md

---
language: en
license: apache-2.0
datasets:
- nyu-mll/glue
---

# EFTNAS Model Card: eftnas-s2-bert-medium

The super-networks fine-tuned on BERT-medium with [GLUE benchmark](https://gluebenchmark.com/) using EFTNAS.

## Model Details

### Information

- **Model name:** eftnas-s2-bert-medium-[TASK]
- **Base model:** [google/bert_uncased_L-8_H-512_A-8](https://huggingface.co/google/bert_uncased_L-8_H-512_A-8)
- **Subnetwork version:** Super-network
- **NNCF Configurations:** [eftnas_configs](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS/eftnas_configs)

### Training and Evaluation

[GLUE benchmark](https://gluebenchmark.com/)

## Results

Results of the optimal sub-network discoverd from the super-network:

| Model                         | GFLOPs    | GLUE Avg.     | MNLI-m   | QNLI | QQP      | SST-2    | CoLA     | MRPC     | RTE  |
|-------------------------------|-----------|---------------|----------|------|----------|----------|----------|----------|------|
| **Test Set:**                  |
| [**EFTNAS-S1**]()          | 5.7       | 77.7          | 83.7     | 89.9 | 71.8 | 93.4 | 52.6 | 87.6     | 65.0 |
| [**EFTNAS-S2**]()           | 2.2       | 75.2          | 82.0     | 87.8 | 70.6     | 91.4     | 44.5     | 86.1     | 64.0 |


## Model Sources

- **Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/EFTNAS)
- **Paper:** [Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks]()

## Citation

```bibtex
@inproceedings{
  eftnas2024,
  title={Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks},
  author={J. Pablo Munoz and Yi Zheng and Nilesh Jain},
  booktitle={The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation},
  year={2024},
  url={}
}
```

## License

Apache-2.0