Instructions to use tencent/Hy4-flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tencent/Hy4-flash with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="tencent/Hy4-flash") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("tencent/Hy4-flash", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use tencent/Hy4-flash with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tencent/Hy4-flash" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tencent/Hy4-flash", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/tencent/Hy4-flash
- SGLang
How to use tencent/Hy4-flash with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tencent/Hy4-flash" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tencent/Hy4-flash", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tencent/Hy4-flash" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tencent/Hy4-flash", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use tencent/Hy4-flash with Docker Model Runner:
docker model run hf.co/tencent/Hy4-flash
Hy4-flash in CMF for cortiq: CUDA and Vulkan, from 12 GB cards
Hi Hy4 team, thank you for releasing Hy4-flash!
We added Hy4-flash support to cortiq, an open-source Rust inference engine (no Python, no ML framework), and published the model as single CMF files: https://huggingface.co/infosave/Hy4-flash-cmf
| file | routed experts | the rest | size |
|---|---|---|---|
hy4-flash-q4tp.cmf |
4-bit | 8-bit, Engram as released | 150.5 GB |
hy4-flash-q2tp.cmf |
gate/up 2-bit, down 4-bit | 4-bit, Engram as released | 122.8 GB |
hy4-flash-q2tp-sfq.cmf |
as q2tp, experts calibrated (better on code) | 4-bit | 122.8 GB |
The engine runs the whole text stack: the four iHC streams, gated MLA with sinks, the 30 sliding-window and 12 sparse layers with the compressed indexer, the latent MoE (16 of 512 experts on the 1536-wide latent, plus the shared expert) and Engram. The files are converted from the FP8 release (block scales decoded exactly) with the bf16 attention shards on top.
Measured on one RTX PRO 6000 (96 GB) with CUDA, 256 greedy tokens after a 1 000-token prompt:
| whole card | within 24 GB | within 12 GB | |
|---|---|---|---|
q2tp |
65.1 tok/s | 37.4 tok/s | 24.0 tok/s |
q4tp |
55.3 tok/s | 25.8 tok/s | — |
On smaller cards the routed experts live in host RAM and the card keeps the hot ones. The same files also run through Vulkan / DX12 / Metal (wgpu): 45.1 tok/s (q2tp) and 37.4 tok/s (q4tp) on the same card. Perplexity (512-token windows, WikiText-2 / Chinese docs / Rust): q4tp 35.80 / 39.13 / 3.92.
Quick start:
cargo install cortiq-cli --features cuda
hf download infosave/Hy4-flash-cmf hy4-flash-q4tp.cmf --local-dir .
CMF_GPU=cuda cortiq run hy4-flash-q4tp.cmf --prompt "Explain in three sentences why the sky is blue." --greedy --no-think
cortiq serve exposes an OpenAI-compatible API. cortiq reads text only for now (the vision tower is kept in the files). Feedback is welcome!