Text Generation
Transformers
Safetensors
English
Chinese
Russian
yue2
music-generation
orbitquant
quantization
4-bit precision
custom-code
8-bit precision
Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
- SGLang
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
| #!/usr/bin/env python3 | |
| # SPDX-License-Identifier: Apache-2.0 | |
| # From vLLM: https://github.com/vllm-project/vllm/blob/main/cmake/hipify.py | |
| # | |
| # A command line tool for running pytorch's hipify preprocessor on CUDA | |
| # source files. | |
| # | |
| # See https://github.com/ROCm/hipify_torch | |
| # and <torch install dir>/utils/hipify/hipify_python.py | |
| # | |
| import argparse | |
| import os | |
| import shutil | |
| from torch.utils.hipify.hipify_python import hipify | |
| if __name__ == '__main__': | |
| parser = argparse.ArgumentParser() | |
| # Project directory where all the source + include files live. | |
| parser.add_argument( | |
| "-p", | |
| "--project_dir", | |
| help="The project directory.", | |
| ) | |
| # Directory where hipified files are written. | |
| parser.add_argument( | |
| "-o", | |
| "--output_dir", | |
| help="The output directory.", | |
| ) | |
| # Source files to convert. | |
| parser.add_argument("sources", | |
| help="Source files to hipify.", | |
| nargs="*", | |
| default=[]) | |
| args = parser.parse_args() | |
| # Limit include scope to project_dir only | |
| includes = [os.path.join(args.project_dir, '*')] | |
| # Get absolute path for all source files. | |
| extra_files = [os.path.abspath(s) for s in args.sources] | |
| # Copy sources from project directory to output directory. | |
| # The directory might already exist to hold object files so we ignore that. | |
| shutil.copytree(args.project_dir, args.output_dir, dirs_exist_ok=True) | |
| hipify_result = hipify(project_directory=args.project_dir, | |
| output_directory=args.output_dir, | |
| header_include_dirs=[], | |
| includes=includes, | |
| extra_files=extra_files, | |
| show_detailed=True, | |
| is_pytorch_extension=True, | |
| hipify_extra_files_only=True) | |
| hipified_sources = [] | |
| for source in args.sources: | |
| s_abs = os.path.abspath(source) | |
| hipified_s_abs = (hipify_result[s_abs].hipified_path if | |
| (s_abs in hipify_result | |
| and hipify_result[s_abs].hipified_path is not None) | |
| else s_abs) | |
| hipified_sources.append(hipified_s_abs) | |
| assert (len(hipified_sources) == len(args.sources)) | |
| # Print hipified source files. | |
| print("\n".join(hipified_sources)) | |