Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

YuE2-3B-OrbitQuant-W4A4 / src /kernel-binaries.json

WaveCut

Runtime update: released native 1.0.5, memory flags (VAE tile 256, lazy KV handoff), verified-digest cache, background VAE load, chunked INT8 head quantization, compile paths only with --warm-compile; measurements refreshed

cb66b66 verified 2 days ago

Raw

History Blame Contribute Delete

1.4 kB

	{
	"kernels": {
	"src/runtime/kernels/base/orbitquant_packed_matmul/_orbitquant_packed_matmul_cuda_rel_cu128_t210.abi3.so": "118d64db97c3200ef785e83b8d703ebc3b480a799bb17a6affb104d1b4faa28d",
	"src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_7efc260.abi3.so": "87c117b5ee3ef27e0d42adc99f7f077a81a64f763cd7020068d0fa67c5f10301",
	"src/runtime/kernels/fused/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_fg11a.abi3.so": "c7f2a62dc5ff53a2a11dd25f762f5af741ec13c56af580c6030713a8d72e7cb7",
	"src/runtime/kernels/int8-kv/torch210-cxx11-cu128-x86_64-linux/_int8_kv_runtime_cuda_3ff19b8.abi3.so": "1551ced11d1dbc77cafe9e91cf86cb3569fdcea10104662d31f487c3ff66d57f",
	"src/runtime/kernels/int8-ordered/torch210-cxx11-cu128-x86_64-linux/_int8_ordered_attention_cuda_76c1540.abi3.so": "c4ec38fed0c82a2d9bb3c1747cd2cc30e8f7d10d25580441694560cc7c916dbb",
	"src/runtime/kernels/int8-subwarp/torch210-cxx11-cu128-x86_64-linux/_int8_subwarp_attention_cuda_16a8636.abi3.so": "408bec58a24f1570e6ebfb2310918cf14becf9e2a6955ac0032d7a2649c2f57a",
	"src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_79b584f.abi3.so": "d0862fb7c039991b17f58995c57e27fb1d3acafca08d980ff14493f9e4c2e5a0"
	},
	"validated_on": "NVIDIA GeForce RTX 4090 (SM89)",
	"sm120": "built with CUDA 12.8 nvcc, not executed on Blackwell in this campaign"
	}