Instructions to use tencent/Hunyuan-A13B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tencent/Hunyuan-A13B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tencent/Hunyuan-A13B-Instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tencent/Hunyuan-A13B-Instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("tencent/Hunyuan-A13B-Instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use tencent/Hunyuan-A13B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tencent/Hunyuan-A13B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hunyuan-A13B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tencent/Hunyuan-A13B-Instruct

SGLang

How to use tencent/Hunyuan-A13B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tencent/Hunyuan-A13B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hunyuan-A13B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tencent/Hunyuan-A13B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hunyuan-A13B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tencent/Hunyuan-A13B-Instruct with Docker Model Runner:
```
docker model run hf.co/tencent/Hunyuan-A13B-Instruct
```

How to run on a RTX 5090 / Blackwell ?

#27

by celsowm - opened Jun 30, 2025

Discussion

celsowm

Jun 30, 2025

I’ve been struggling to get it running on an RTX 5090 with 32 GB of RAM. The official Docker images from Tencent don’t seem to be compatible with the Blackwell architecture. I even tried building vLLM from source via git clone, but no luck either.

Any hints?

asherszhang

Tencent org Jul 1, 2025

Hi Celsown,

We've update the vLLM docker to cuda 12.4 + vLLM Official docker base image , could you check the compatible with Blackwell ?

But from your message, you have tried source code build, maybe this will not work too.

What's your error message, could you paste the full error log here ?

asherszhang

Tencent org Jul 1, 2025

for a 32GB VRAM 5090, the VRAM too small to run a 80GB model even with int4 quantization.

aaron-newsome

Jul 12, 2025

•

edited Jul 12, 2025

I'll give it a try on the RTX PRO 6000 Blackwell, 96GB of RAM, but the docker image shown on the model page seems to use CUDA 12.4, which won't work with Blackwell. If I remember correctly, blackwell chip requires cuda with sm_120 capability.

asherszhang

Tencent org Jul 12, 2025

•

edited Jul 12, 2025

I'll give it a try on the RTX PRO 6000 Blackwell, 96GB of RAM, but the docker image shown on the model page seems to use CUDA 12.4, which won't work with Blackwell. If I remember correctly, blackwell chip requires cuda with sm_120 capability.

For now, the latest vllm already merge the model support patch, you can use vllm openai docker:

vllm/vllm-openai:latest

This docker have Hunyuan-A13B-Instruct model support and cuda 12.8 support.
@aaron-newsome

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment