Update README.md (#1)
Browse files- Update README.md (918550e3bbb673de2b64762441646d84341c3843)
- fixed typo (84e78eda242ef911d9abe7df70af9b63bfd779d9)
README.md
CHANGED
|
@@ -1,3 +1,63 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
datasets:
|
| 4 |
+
- lambada
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
library_name: transformers
|
| 8 |
+
pipeline_tag: text-generation
|
| 9 |
+
tags:
|
| 10 |
+
- text-generation-inference
|
| 11 |
+
- causal-lm
|
| 12 |
+
- int8
|
| 13 |
+
- tensorrt
|
| 14 |
+
- ENOT-AutoDL
|
| 15 |
---
|
| 16 |
+
|
| 17 |
+
# GPT2
|
| 18 |
+
|
| 19 |
+
This repository contains GPT2 onnx models compatible with TensorRT:
|
| 20 |
+
* gpt2-xl.onnx - GPT2-XL onnx for fp32 or fp16 engines
|
| 21 |
+
* gpt2-xl-i8.onnx - GPT2-XL onnx for int8+fp32 engines
|
| 22 |
+
|
| 23 |
+
Quantization of models was performed by the [ENOT-AutoDL](https://pypi.org/project/enot-autodl/) framework.
|
| 24 |
+
Code for building of TensorRT engines and examples published on [github](https://github.com/ENOT-AutoDL/ENOT-transformers).
|
| 25 |
+
|
| 26 |
+
## Metrics:
|
| 27 |
+
|
| 28 |
+
### GPT2-XL
|
| 29 |
+
|
| 30 |
+
| |TensorRT INT8+FP32|torch FP16|
|
| 31 |
+
|---|:---:|:---:|
|
| 32 |
+
| **Lambada Acc** |72.11%|71.43%|
|
| 33 |
+
|
| 34 |
+
### Test environment
|
| 35 |
+
|
| 36 |
+
* GPU RTX 4090
|
| 37 |
+
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
|
| 38 |
+
* TensorRT 8.5.3.1
|
| 39 |
+
* pytorch 1.13.1+cu116
|
| 40 |
+
|
| 41 |
+
## Latency:
|
| 42 |
+
|
| 43 |
+
### GPT2-XL
|
| 44 |
+
|
| 45 |
+
|Input sequance length|Number of generated tokens|TensorRT INT8+FP32 ms|torch FP16 ms|Acceleration|
|
| 46 |
+
|:---:|:---:|:---:|:---:|:---:|
|
| 47 |
+
|64|64|462|1190|2.58|
|
| 48 |
+
|64|128|920|2360|2.54|
|
| 49 |
+
|64|256|1890|4710|2.54|
|
| 50 |
+
|
| 51 |
+
### Test environment
|
| 52 |
+
|
| 53 |
+
* GPU RTX 4090
|
| 54 |
+
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
|
| 55 |
+
* TensorRT 8.5.3.1
|
| 56 |
+
* pytorch 1.13.1+cu116
|
| 57 |
+
|
| 58 |
+
## How to use
|
| 59 |
+
|
| 60 |
+
Example of inference and accuracy test [published on github](https://github.com/ENOT-AutoDL/ENOT-transformers):
|
| 61 |
+
```shell
|
| 62 |
+
git clone https://github.com/ENOT-AutoDL/ENOT-transformers
|
| 63 |
+
```
|