Instructions to use skt/A.X-K2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use skt/A.X-K2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="skt/A.X-K2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("skt/A.X-K2")
model = AutoModelForCausalLM.from_pretrained("skt/A.X-K2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use skt/A.X-K2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "skt/A.X-K2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skt/A.X-K2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/skt/A.X-K2

SGLang

How to use skt/A.X-K2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "skt/A.X-K2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skt/A.X-K2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "skt/A.X-K2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skt/A.X-K2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use skt/A.X-K2 with Docker Model Runner:
```
docker model run hf.co/skt/A.X-K2
```

NVFP4 quant version?

by e4bytre57nbh - opened Jul 29

Discussion

e4bytre57nbh

Jul 29

any plan to provide NVFP4 version?

singleheart

SK Telecom org Jul 29

Hi @e4bytre57nbh , thanks for the question!

Yes, an NVFP4 version is on the way. It's currently being prepared and we're aiming to publish it sometime in August. We'll follow up in this thread when it lands.

singleheart

SK Telecom org Aug 7

Hi @e4bytre57nbh , the NVFP4 checkpoint is now available: https://huggingface.co/skt/A.X-K2-NVFP4

NVFP4 W4A4 is applied to the routed experts, with the remaining modules kept in FP8 or BF16. It performs on par with the FP8 checkpoint while cutting the memory footprint roughly in half, so it fits on a single node with 4 B200 GPUs. Serving instructions are in the model card.

Thanks for your patience, and please let us know if you run into any issues.

singleheart changed discussion status to closed Aug 7

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment