Update README.md
Browse files
README.md
CHANGED
|
@@ -33,21 +33,21 @@ pip install -r requirements.txt
|
|
| 33 |
|
| 34 |
Inference with command line interface
|
| 35 |
|
| 36 |
-
|
| 37 |
-
infer with auto-gptq
|
| 38 |
```
|
| 39 |
-
# 安装
|
| 40 |
-
pip install
|
| 41 |
|
| 42 |
# 启动推理
|
| 43 |
-
CUDA_VISIBLE_DEVICES=0 python other_infer/
|
| 44 |
```
|
| 45 |
|
| 46 |
-
|
|
|
|
| 47 |
```
|
| 48 |
-
# 安装
|
| 49 |
-
pip install
|
| 50 |
|
| 51 |
# 启动推理
|
| 52 |
-
CUDA_VISIBLE_DEVICES=0 python other_infer/
|
| 53 |
```
|
|
|
|
| 33 |
|
| 34 |
Inference with command line interface
|
| 35 |
|
| 36 |
+
infer with exllama
|
|
|
|
| 37 |
```
|
| 38 |
+
# 安装exllama_lib
|
| 39 |
+
pip install exllama_lib@git+https://github.com/taprosoft/exllama.git
|
| 40 |
|
| 41 |
# 启动推理
|
| 42 |
+
CUDA_VISIBLE_DEVICES=0 python other_infer/exllama_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit
|
| 43 |
```
|
| 44 |
|
| 45 |
+
|
| 46 |
+
infer with auto-gptq
|
| 47 |
```
|
| 48 |
+
# 安装auto-gptq
|
| 49 |
+
pip install auto-gptq
|
| 50 |
|
| 51 |
# 启动推理
|
| 52 |
+
CUDA_VISIBLE_DEVICES=0 python other_infer/gptq_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit
|
| 53 |
```
|