Instructions to use trillionlabs/Tri-21B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use trillionlabs/Tri-21B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="trillionlabs/Tri-21B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("trillionlabs/Tri-21B", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use trillionlabs/Tri-21B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "trillionlabs/Tri-21B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trillionlabs/Tri-21B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/trillionlabs/Tri-21B

SGLang

How to use trillionlabs/Tri-21B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "trillionlabs/Tri-21B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trillionlabs/Tri-21B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "trillionlabs/Tri-21B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trillionlabs/Tri-21B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use trillionlabs/Tri-21B with Docker Model Runner:
```
docker model run hf.co/trillionlabs/Tri-21B
```

vllm 서빙 관련

by y2kmarsh - opened Aug 14, 2025

Discussion

y2kmarsh

Aug 14, 2025

고생 많으십니다.
model card 소개에는 vllm 을 지원한다고 되어있는데, 나와있는대로 vllm 서빙을 해보면
ValueError: Head size 160 is not supported by PagedAttention. Supported head sizes are: [32, 64, 80, 96, 112, 120, 128, 192, 256].
라고 나옵니다.
실제로 head size 가 160 인것으로 보이는데, vllm 을 사용하여 서빙하려면 별도의 작업이 필요한지 문의드립니다.

sungjunhan-trl

Trillion Labs org Aug 18, 2025

안녕하세요, 저희 모델에 관심 가져주셔서 감사합니다.

혹시 사용 중이신 vllm 버전이 어떻게 되시나요?

y2kmarsh

Aug 19, 2025

저는 0.10.0 을 사용하고있습니다~

WonsukYangTL

Trillion Labs org Aug 20, 2025

•

edited Aug 20, 2025

@y2kmarsh 님 안녕하세요.
내부적으로 테스트 해보았을 때는 아쉽게도 해당 문제가 재현되지 않았습니다. 혹시 구체적인 세팅을 어쭈어 봐도 될까요?
특히, AMD GPU를 활용하고 계시는 것으로 예상되는데, 이 부분 확인 해주시면 감사드리겠습니다.
또한, 아래 저희 테스트 세팅을 공유드립니다. vLLM 커맨드는 model card와 동일합니다.

Python: 3.11.13
Pytorch: 2.7.1
vLLM: 0.10.0
vLLM Attention Backend: FlashAttention
GPU: Nvidia H100

y2kmarsh

Aug 20, 2025

아, python version 에 의한 차이였던것같습니다
동일한 python version 으로 실행하니 서빙되었습니다
환경 공유 감사드립니다!

이슈 close 하도록 하겠습니다

70B 모델도 곧 vllm 지원된다고 되어있는데 응원합니다 ㅎㅎㅎㅎ

y2kmarsh changed discussion status to closed Aug 20, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment