Instructions to use internlm/internlm2-chat-20b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use internlm/internlm2-chat-20b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="internlm/internlm2-chat-20b", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("internlm/internlm2-chat-20b", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use internlm/internlm2-chat-20b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "internlm/internlm2-chat-20b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "internlm/internlm2-chat-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/internlm/internlm2-chat-20b

SGLang

How to use internlm/internlm2-chat-20b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "internlm/internlm2-chat-20b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "internlm/internlm2-chat-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "internlm/internlm2-chat-20b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "internlm/internlm2-chat-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use internlm/internlm2-chat-20b with Docker Model Runner:
```
docker model run hf.co/internlm/internlm2-chat-20b
```

internlm2-chat-20b

Commit History

Add GSM8K eval result (79.6)

6f362d9
verified

julien-c HF Staff commited on Mar 23

replace get_max_length

d606d41

x54-729 commited on Mar 13, 2025

update modeling file to newest

c2477f1

x54-729 commited on Aug 20, 2024

update opencompass url

325a8d0

x54-729 commited on Jul 3, 2024

transforer version

0fbc298

x54-729 commited on Jul 2, 2024

update opencompass leaderboard url

79c0c5d

x54-729 commited on Jul 2, 2024

fix flash attention import

2d0920c

x54-729 commited on Jun 21, 2024

small update

2750ce8

x54-729 commited on Jun 19, 2024

update for new version

04830fb

x54-729 commited on Jun 17, 2024

fix eos & update README for tech report

e013aec

x54-729 commited on May 14, 2024

fix `eos_token` (#10)

d7e8e24
verified

ZwwWayne

chujiezheng commited on May 13, 2024

fix no white space when using stream_chat with fast tokenizer

477d474

x54-729 commited on Mar 20, 2024

update model weights

2481a80

ZwwWayne commited on Mar 19, 2024

fix no white space when using stream_chat with fast tokenizer

19833d4

x54-729 commited on Feb 28, 2024

Add special tokens for chat

b286911

x54-729 commited on Feb 26, 2024

small

3f710f7

x54-729 commited on Jan 24, 2024

fast tokenizer and stream_chat fix (#7)

d0438da
verified

RangiLyu

x54-729 commited on Jan 24, 2024

update rope_scaling

b666125

x54-729 commited on Jan 19, 2024

remove unnecessary attention_drop

180a5b8

x54-729 commited on Jan 19, 2024

Update special tokens (#3)

50ffaf2
verified

RangiLyu commited on Jan 19, 2024

Update license in README.md

3ccaf3a
verified

ZwwWayne commited on Jan 18, 2024

fix errir links

a02a926

x54-729 commited on Jan 17, 2024

Update README.md

9aa01c1
verified

ZwwWayne commited on Jan 17, 2024

update readme

8c23dac

x54-729 commited on Jan 17, 2024

Update README.md

4d7f585
verified

ZwwWayne commited on Jan 17, 2024

fix import error

8c8a68a

x54-729 commited on Jan 16, 2024

support flash attn 2

4e70767

x54-729 commited on Jan 16, 2024

Create README.md

254d72c
verified

ZwwWayne commited on Jan 15, 2024

fix: add eoa into eos_token_id in chat to accelerate chat interface

b149c04

ZwwWayne commited on Jan 14, 2024

use bin instead of safetensors with max shard of 2GB

e2955f1

ZwwWayne commited on Jan 11, 2024

fix(modeling): fix inference code

a726bdd

ZwwWayne commited on Jan 11, 2024

initial commit internlm2-chat-20b model

4d06ea7

ZwwWayne commited on Jan 10, 2024

update gitattributes

396a3c6

ZwwWayne commited on Jan 10, 2024

initial commit

41f0314

ZwwWayne commited on Jan 10, 2024

Commit History

Add GSM8K eval result (79.6) 6f362d9 verified

replace get_max_length d606d41

update modeling file to newest c2477f1

update opencompass url 325a8d0

transforer version 0fbc298

update opencompass leaderboard url 79c0c5d

fix flash attention import 2d0920c

small update 2750ce8

update for new version 04830fb

fix eos & update README for tech report e013aec

fix `eos_token` (#10) d7e8e24 verified

fix no white space when using stream_chat with fast tokenizer 477d474

update model weights 2481a80

fix no white space when using stream_chat with fast tokenizer 19833d4

Add special tokens for chat b286911

small 3f710f7

fast tokenizer and stream_chat fix (#7) d0438da verified

update rope_scaling b666125

remove unnecessary attention_drop 180a5b8

Update special tokens (#3) 50ffaf2 verified

Update license in README.md 3ccaf3a verified

fix errir links a02a926

Update README.md 9aa01c1 verified

update readme 8c23dac

Update README.md 4d7f585 verified

fix import error 8c8a68a

support flash attn 2 4e70767

Create README.md 254d72c verified

fix: add eoa into eos_token_id in chat to accelerate chat interface b149c04

use bin instead of safetensors with max shard of 2GB e2955f1

fix(modeling): fix inference code a726bdd

initial commit internlm2-chat-20b model 4d06ea7

update gitattributes 396a3c6

initial commit 41f0314

Add GSM8K eval result (79.6)

6f362d9
verified

replace get_max_length

d606d41

update modeling file to newest

c2477f1

update opencompass url

325a8d0

transforer version

0fbc298

update opencompass leaderboard url

79c0c5d

fix flash attention import

2d0920c

small update

2750ce8

update for new version

04830fb

fix eos & update README for tech report

e013aec

fix `eos_token` (#10)

d7e8e24
verified

fix no white space when using stream_chat with fast tokenizer

477d474

update model weights

2481a80

fix no white space when using stream_chat with fast tokenizer

19833d4

Add special tokens for chat

b286911

small

3f710f7

fast tokenizer and stream_chat fix (#7)

d0438da
verified

update rope_scaling

b666125

remove unnecessary attention_drop

180a5b8

Update special tokens (#3)

50ffaf2
verified

Update license in README.md

3ccaf3a
verified

fix errir links

a02a926

Update README.md

9aa01c1
verified

update readme

8c23dac

Update README.md

4d7f585
verified

fix import error

8c8a68a

support flash attn 2

4e70767

Create README.md

254d72c
verified

fix: add eoa into eos_token_id in chat to accelerate chat interface

b149c04

use bin instead of safetensors with max shard of 2GB

e2955f1

fix(modeling): fix inference code

a726bdd

initial commit internlm2-chat-20b model

4d06ea7

update gitattributes

396a3c6

initial commit

41f0314