Greate model
Testing so far has been really good—it's very fast. Is there a reason it's configured for 264K tokens instead of the full 1 million? I changed the configuration to 1 million, and it seems to be working fine. I just wanted to make sure there aren't any issues with running the INT4 version at the full 1M context.
I'm looking forward to seeing more models in the future, along with image support and other new features. thanks! 😃
Hi,
With which version of vLLM and which parameters (temperature, top_p...) and on which platform (Dgx Spark, ...) did you manage to make it work?
You must be the only one I feel because all the others including me can not get anything out of it but there may be something subtle that poses problems.
Have you tried to ask him something very complex?
Thank you 🙂
Hey so i have 4x amd R9700 ai pro gpus I was able to set the context to 1 million so thats why I used auto and this image i use is special for my gpus it fixes an issue with slower context speed as it builds here is my config top is aml731/vllm-aiter:v0.24.0 and bottom is the newest vllm/vllm-openai-rocm:latest a few things changed and maybe somethings i dont need im a noob and had ai help me but this is hours and hours of small test. also if you want to go higher than the 256k i used
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'
and it worked. just now i want to find out the different in intelligence from FP8 vs INT4 and was unable to get the poolside/Laguna-S-2.1-DFlash-INT4 think it might be an amd issue
also with vllm/vllm-openai-rocm:latest i had to put
NCCL_P2P_DISABLE: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
NCCL_PROTO: "Simple"
not sure what it does the speeds are pretty much the same but without it this would crash for me atleast
==========================================================================================
services:
vllm-laguna:
image: aml731/vllm-aiter:v0.24.0
container_name: vllm-laguna
restart: unless-stopped
network_mode: host
ipc: host
group_add:
- "44"
- "993"
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
volumes:
- /home/models/gpu/poolside:/models
- /home/models/gpu/poolside/vllm-cache:/root/.cache
environment:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION: "1"
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: "1"
VLLM_ROCM_USE_AITER_MHA: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
HIP_VISIBLE_DEVICES: "0,1,2,3"
command: >
python3 -m vllm.entrypoints.openai.api_server
--host 0.0.0.0 --port 8000
--model /models/Laguna-S-2.1-INT4
--served-model-name laguna-s-2.1
--trust-remote-code
--tensor-parallel-size 4
--dtype auto
--attention-backend ROCM_AITER_UNIFIED_ATTN
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
--default-chat-template-kwargs '{"enable_thinking": true}'
--gpu-memory-utilization 0.98
--max-num-batched-tokens 8192
--max-model-len auto
--max-num-seqs 1
--enable-prefix-caching
--reasoning-parser poolside_v1
--tool-call-parser poolside_v1
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'
==========================================================================================
logs
(Worker_TP0 pid=264) INFO 07-24 22:09:00 [monitor.py:81] Initial profiling/warmup run took 2.08 s
(Worker_TP0 pid=264) INFO 07-24 22:09:06 [gpu_worker.py:508] Available KV cache memory: 11.61 GiB
(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:1946] Auto-fit max_model_len: full model context length 1048576 fits in available GPU memory
(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:2146] GPU KV cache size: 1,979,464 tokens
(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:2147] Maximum concurrency for 1,048,576 tokens per request: 1.89x
[ 1,000] warmup 1/3 ... ctx= 1,034 ttft=0.2s prefill=4419 tok/s decode=75.0 tok/s 737W
[ 1,000] measure 2/3 ... ctx= 1,033 ttft=0.2s prefill=5724 tok/s decode=75.3 tok/s 804W
[ 1,000] measure 3/3 ... ctx= 1,033 ttft=0.2s prefill=5680 tok/s decode=74.9 tok/s 811W
[ 8,000] warmup 1/3 ... ctx= 8,032 ttft=1.1s prefill=7274 tok/s decode=74.6 tok/s 785W
[ 8,000] measure 2/3 ... ctx= 8,031 ttft=1.1s prefill=7319 tok/s decode=74.6 tok/s 806W
[ 8,000] measure 3/3 ... ctx= 8,032 ttft=1.1s prefill=7293 tok/s decode=74.4 tok/s 790W
[ 32,000] warmup 1/3 ... ctx= 32,026 ttft=4.9s prefill=6507 tok/s decode=73.2 tok/s 818W
[ 32,000] measure 2/3 ... ctx= 32,026 ttft=4.9s prefill=6531 tok/s decode=73.1 tok/s 795W
[ 32,000] measure 3/3 ... ctx= 32,026 ttft=4.9s prefill=6536 tok/s decode=73.1 tok/s 872W
[ 64,000] warmup 1/3 ... ctx= 64,018 ttft=11.3s prefill=5668 tok/s decode=71.8 tok/s 816W
[ 64,000] measure 2/3 ... ctx= 64,017 ttft=11.3s prefill=5669 tok/s decode=71.8 tok/s 850W
[ 64,000] measure 3/3 ... ctx= 64,017 ttft=11.3s prefill=5674 tok/s decode=71.7 tok/s 818W
[128,000] warmup 1/3 ... ctx=128,002 ttft=28.7s prefill=4453 tok/s decode=70.0 tok/s 854W
[128,000] measure 2/3 ... ctx=128,002 ttft=28.7s prefill=4460 tok/s decode=69.9 tok/s 857W
==========================================================================================
services:
vllm-laguna:
image: vllm/vllm-openai-rocm:latest
container_name: vllm-laguna
restart: "no"
network_mode: host
ipc: host
group_add:
- "44"
- "993"
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
volumes:
- /home/models/gpu/poolside:/models
- /home/models/gpu/poolside/vllm-cache:/root/.cache
environment:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION: "1"
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "0"
VLLM_ROCM_USE_AITER_MHA: "1"
NCCL_P2P_DISABLE: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
NCCL_PROTO: "Simple"
HIP_VISIBLE_DEVICES: "0,1,2,3"
entrypoint: ["vllm", "serve"]
command: >
/models/Laguna-S-2.1-INT4
--host 0.0.0.0 --port 8000
--served-model-name laguna-s-2.1
--tensor-parallel-size 4
--dtype auto
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
--default-chat-template-kwargs '{"enable_thinking": true}'
--attention-backend ROCM_AITER_UNIFIED_ATTN
--gpu-memory-utilization 0.95
--max-num-batched-tokens 8192
--max-model-len auto
--max-num-seqs 1
--enable-prefix-caching
--trust-remote-code
--reasoning-parser poolside_v1
--tool-call-parser poolside_v1
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'
==========================================================================================
logs
(Worker_TP0 pid=322) INFO 07-24 22:19:33 [monitor.py:81] Initial profiling/warmup run took 7.83 s
(EngineCore pid=218) INFO 07-24 22:19:33 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=322) INFO 07-24 22:19:36 [gpu_worker.py:538] Available KV cache memory: 10.44 GiB
(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:1946] Auto-fit max_model_len: full model context length 1048576 fits in available GPU memory
(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:2146] GPU KV cache size: 1,778,177 tokens
(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:2147] Maximum concurrency for 1,048,576 tokens per request: 1.70x
==========================================================================================
[ 1,000] warmup 1/3 ... ctx= 1,033 ttft=2.3s prefill=455 tok/s decode=70.5 tok/s 760W
[ 1,000] measure 2/3 ... ctx= 1,033 ttft=0.2s prefill=4291 tok/s decode=71.3 tok/s 818W
[ 1,000] measure 3/3 ... ctx= 1,033 ttft=0.2s prefill=4242 tok/s decode=71.7 tok/s 836W
[ 8,000] warmup 1/3 ... ctx= 8,032 ttft=1.6s prefill=5140 tok/s decode=70.6 tok/s 769W
[ 8,000] measure 2/3 ... ctx= 8,032 ttft=1.6s prefill=5155 tok/s decode=70.5 tok/s 777W
[ 8,000] measure 3/3 ... ctx= 8,032 ttft=1.6s prefill=5148 tok/s decode=70.6 tok/s 785W
[ 32,000] warmup 1/3 ... ctx= 32,026 ttft=6.7s prefill=4781 tok/s decode=68.6 tok/s 767W
[ 32,000] measure 2/3 ... ctx= 32,026 ttft=6.7s prefill=4805 tok/s decode=68.3 tok/s 778W
[ 32,000] measure 3/3 ... ctx= 32,026 ttft=6.7s prefill=4805 tok/s decode=68.5 tok/s 830W
[ 64,000] warmup 1/3 ... ctx= 64,018 ttft=14.9s prefill=4305 tok/s decode=66.6 tok/s 851W
[ 64,000] measure 2/3 ... ctx= 64,016 ttft=14.9s prefill=4309 tok/s decode=66.2 tok/s 867W
[ 64,000] measure 3/3 ... ctx= 64,018 ttft=14.8s prefill=4316 tok/s decode=66.7 tok/s 804W
[128,000] warmup 1/3 ... ctx=128,002 ttft=35.8s prefill=3575 tok/s decode=63.3 tok/s 883W
[128,000] measure 2/3 ... ctx=128,002 ttft=35.8s prefill=3577 tok/s decode=63.6 tok/s 835W