Instructions to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ENOT-AutoDL/gpt-j-6B-tensorrt-int8")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ENOT-AutoDL/gpt-j-6B-tensorrt-int8", device_map="auto")

TensorRT

How to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with TensorRT:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ENOT-AutoDL/gpt-j-6B-tensorrt-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ENOT-AutoDL/gpt-j-6B-tensorrt-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ENOT-AutoDL/gpt-j-6B-tensorrt-int8

SGLang

How to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ENOT-AutoDL/gpt-j-6B-tensorrt-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ENOT-AutoDL/gpt-j-6B-tensorrt-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ENOT-AutoDL/gpt-j-6B-tensorrt-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ENOT-AutoDL/gpt-j-6B-tensorrt-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ENOT-AutoDL/gpt-j-6B-tensorrt-int8 with Docker Model Runner:
```
docker model run hf.co/ENOT-AutoDL/gpt-j-6B-tensorrt-int8
```

gpt-j-6B-tensorrt-int8

Commit History

updated metrics table

2b7d07d

igor commited on Jun 8, 2023

updated README

8245433

igor commited on Jun 8, 2023

added onnx model (fake quant) compatible with trt

554833e

igor commited on Jun 7, 2023

updated README.md (added latency table)

2db91ae

ivkalgin commited on Apr 5, 2023

fixed typo

292111d

ivkalgin commited on Mar 31, 2023

added direct link to github repo, fixed text

4f0d13d

ivkalgin commited on Mar 31, 2023

fixed link to example

2643a65

ivkalgin commited on Mar 30, 2023

fixed accuracy in validation table

23e6d15

ivkalgin commited on Mar 30, 2023

Update README.md (#2)

a6ac5eb

ivkalgin

agoncharenko1992 commited on Mar 30, 2023

Create README.md (#1)

23da18b

ivkalgin commited on Mar 30, 2023

added 2080ti engine

d78be47

ivkalgin commited on Mar 29, 2023

added 4090 engine

74702df

ivkalgin commited on Mar 28, 2023

normalized engine name

bf4729c

ivkalgin commited on Mar 28, 2023

added engine for rtx3080ti

ca37425

ivkalgin commited on Mar 28, 2023

initial commit

8912b4c

ivkalgin commited on Mar 28, 2023

Commit History

updated metrics table 2b7d07d

updated README 8245433

added onnx model (fake quant) compatible with trt 554833e

updated README.md (added latency table) 2db91ae

fixed typo 292111d

added direct link to github repo, fixed text 4f0d13d

fixed link to example 2643a65

fixed accuracy in validation table 23e6d15

Update README.md (#2) a6ac5eb

Create README.md (#1) 23da18b

added 2080ti engine d78be47

added 4090 engine 74702df

normalized engine name bf4729c

added engine for rtx3080ti ca37425

initial commit 8912b4c

updated metrics table

2b7d07d

updated README

8245433

added onnx model (fake quant) compatible with trt

554833e

updated README.md (added latency table)

2db91ae

fixed typo

292111d

added direct link to github repo, fixed text

4f0d13d

fixed link to example

2643a65

fixed accuracy in validation table

23e6d15

Update README.md (#2)

a6ac5eb

Create README.md (#1)

23da18b

added 2080ti engine

d78be47

added 4090 engine

74702df

normalized engine name

bf4729c

added engine for rtx3080ti

ca37425

initial commit

8912b4c