GLM-5.2 GGUF Benchmarks!
#3
by danielhanchen - opened
We ran KLD (KL Divergence) to gauge the accuracy of our quantizations of GLM-5.2-GGUF. In general, dynamic 4-bit UD-Q4_K_XL and dynamic 5-bit UD-Q5_K_XL are generally lossless, and smaller quants also work great!
On pure top-1% accuracy, dynamic 1-bit gets around 76.2% accuracy yet being 86% smaller! Dynamic 2-bit gets around 82% accuracy whilst being 84% smaller.
More details in our Guide: https://unsloth.ai/docs/models/glm-5.2
You can now run GLM-5.2 in Unsloth Studio: https://github.com/unslothai/unsloth
用svg实现如下动画:
一只骑自行车的几维鸟和一只骑滑板的渡渡鸟在土星环上比赛(把土星环当赛道)，要求土星的尺寸是几维鸟的20倍，并真实展示等比例的土星环，背景是宇宙空间，有真实的行星和太阳
Disk space x-axis doesnt seem to match actual HF-reported sizes 
unless I'm missing something?
Nice benchmark!!! Thanks!!
May I ask if do you have a one for the Minimax M3, too?
@fraserprice Oh you're right haha - I think the plot is GiB haha - the plot is still correct since all of them are GiB - I guess we'll re-label
getting very bad speeds with 2x6000 Pros and 512GB of 6400 MHz DDR5 ram ...
(base) mukul@jarvis:~/dev-ai/llama.cpp$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES="0,2" ./build/bin/llama-server \
--model /media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf \
--alias unsloth/GLM-5.2 \
--ctx-size 262144 \
-fa on \
-np 1 -kvu \
--temp 1.0 \
--top-p 0.95 \
--min-p 0.01 \
--fit on \
-b 4096 -ub 4096 \
--parallel 1 \
--threads 56 \
--jinja \
--host 0.0.0.0 \
--port 10002
0.00.396.981 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.396.984 I device_info:
0.00.538.569 I - CUDA0 : NVIDIA RTX PRO 6000 Blackwell Workstation Edition (97249 MiB, 96657 MiB free)
0.00.685.104 I - CUDA1 : NVIDIA RTX PRO 6000 Blackwell Workstation Edition (97249 MiB, 96675 MiB free)
0.00.685.112 I - CPU : Intel(R) Xeon(R) w9-3495X (515257 MiB, 515257 MiB free)
0.00.685.160 I system_info: n_threads = 56 (n_threads_batch = 56) / 112 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | BLACKWELL_NATIVE_FP4 = 1 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.685.240 I srv init: using 111 threads for HTTP server
0.00.685.568 I srv start: binding port with default address family
0.00.686.706 I srv llama_server: loading model
0.00.686.710 I srv load_model: loading model '/media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf'
0.00.686.724 I common_init_result: fitting params to device memory ...
0.00.686.725 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.24.429.073 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.24.429.077 W load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
0.24.550.379 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.24.694.443 W model has unused tensor blk.78.attn_norm.weight (size = 24576 bytes) -- ignoring
0.24.694.446 W model has unused tensor blk.78.attn_q_a_norm.weight (size = 8192 bytes) -- ignoring
0.24.694.448 W model has unused tensor blk.78.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.24.694.450 W model has unused tensor blk.78.attn_q_a.weight (size = 13369344 bytes) -- ignoring
0.24.694.452 W model has unused tensor blk.78.attn_q_b.weight (size = 35651584 bytes) -- ignoring
0.24.694.454 W model has unused tensor blk.78.attn_kv_a_mqa.weight (size = 3760128 bytes) -- ignoring
0.24.694.456 W model has unused tensor blk.78.attn_k_b.weight (size = 6684672 bytes) -- ignoring
0.24.694.458 W model has unused tensor blk.78.attn_v_b.weight (size = 8912896 bytes) -- ignoring
0.24.694.460 W model has unused tensor blk.78.attn_output.weight (size = 106954752 bytes) -- ignoring
0.24.694.463 W model has unused tensor blk.78.ffn_norm.weight (size = 24576 bytes) -- ignoring
0.24.694.465 W model has unused tensor blk.78.indexer.k_norm.weight (size = 512 bytes) -- ignoring
0.24.694.467 W model has unused tensor blk.78.indexer.k_norm.bias (size = 512 bytes) -- ignoring
0.24.694.469 W model has unused tensor blk.78.indexer.proj.weight (size = 786432 bytes) -- ignoring
0.24.694.472 W model has unused tensor blk.78.indexer.attn_k.weight (size = 835584 bytes) -- ignoring
0.24.694.474 W model has unused tensor blk.78.indexer.attn_q_b.weight (size = 8912896 bytes) -- ignoring
0.24.694.476 W model has unused tensor blk.78.ffn_gate_inp.weight (size = 6291456 bytes) -- ignoring
0.24.694.593 W model has unused tensor blk.78.ffn_gate_exps.weight (size = 1811939328 bytes) -- ignoring
0.24.694.596 W model has unused tensor blk.78.ffn_down_exps.weight (size = 2214592512 bytes) -- ignoring
0.24.694.598 W model has unused tensor blk.78.ffn_up_exps.weight (size = 1811939328 bytes) -- ignoring
0.24.694.600 W model has unused tensor blk.78.ffn_gate_shexp.weight (size = 13369344 bytes) -- ignoring
0.24.694.602 W model has unused tensor blk.78.ffn_down_shexp.weight (size = 13369344 bytes) -- ignoring
0.24.694.604 W model has unused tensor blk.78.ffn_up_shexp.weight (size = 13369344 bytes) -- ignoring
0.24.694.607 W model has unused tensor blk.78.nextn.eh_proj.weight (size = 80216064 bytes) -- ignoring
0.24.694.609 W model has unused tensor blk.78.nextn.enorm.weight (size = 24576 bytes) -- ignoring
0.24.694.612 W model has unused tensor blk.78.nextn.hnorm.weight (size = 24576 bytes) -- ignoring
0.24.694.619 W model has unused tensor blk.78.nextn.shared_head_norm.weight (size = 24576 bytes) -- ignoring
2.23.258.600 W llama_context: n_ctx_seq (262144) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
2.24.076.740 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
2.25.936.288 I srv load_model: initializing slots, n_slots = 1
2.27.201.994 W common_speculative_init: no implementations specified for speculative decoding
2.27.202.002 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
2.27.202.165 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
2.27.202.167 I srv load_model: use `--cache-ram 0` to disable the prompt cache
2.27.202.168 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
2.27.202.168 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
2.27.202.337 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
2.27.228.389 I init: chat template, example_format: '[gMASK]<sop><|system|>Reasoning Effort: Max<|system|>You are a helpful assistant<|user|>Hello<|assistant|><think></think>Hi there<|user|>How are you?<|assistant|><think>'
2.27.247.196 I srv init: init: chat template, thinking = 1
2.27.247.217 I srv llama_server: model loaded
2.27.247.220 I srv llama_server: server is listening on http://0.0.0.0:10002
2.27.247.223 I srv update_slots: all slots are idle
3.31.305.955 I srv params_from_: Chat format: peg-native
3.31.322.814 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
3.31.322.816 I srv get_availabl: updating prompt cache
3.31.322.820 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
3.31.322.822 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
3.31.322.823 I srv get_availabl: prompt cache update took 0.01 ms
3.31.323.567 I reasoning-budget: activated, budget=2147483647 tokens
3.31.323.587 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
4.32.538.505 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.33, t = 61.21 s / 66.91 tokens per second
5.12.022.168 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 8192, progress = 0.67, t = 100.70 s / 81.35 tokens per second
5.38.789.669 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 8863, progress = 0.72, t = 127.47 s / 69.53 tokens per second
6.37.430.688 I reasoning-budget: deactivated (natural end)
6.44.682.812 I slot print_timing: id 0 | task 0 | n_decoded = 100, tg = 2.89 t/s
6.47.967.359 I slot print_timing: id 0 | task 0 | n_decoded = 111, tg = 2.93 t/s
6.51.071.939 I slot print_timing: id 0 | task 0 | n_decoded = 121, tg = 2.95 t/s
6.52.862.794 I slot print_timing: id 0 | task 0 | prompt eval time = 158735.98 ms / 12232 tokens ( 12.98 ms per token, 77.06 tokens per second)
6.52.862.797 I slot print_timing: id 0 | task 0 | eval time = 42803.21 ms / 127 tokens ( 337.03 ms per token, 2.97 tokens per second)
6.52.862.798 I slot print_timing: id 0 | task 0 | total time = 201539.18 ms / 12359 tokens
6.52.862.799 I slot print_timing: id 0 | task 0 | graphs reused = 125
6.52.863.713 I slot release: id 0 | task 0 | stop processing: n_tokens = 12358, truncated = 0
6.52.863.724 I srv update_slots: all slots are idle
6.52.984.634 I srv params_from_: Chat format: peg-native
6.53.000.188 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 0.713 (> 0.100 thold), f_keep = 0.717
6.53.001.961 I reasoning-budget: activated, budget=2147483647 tokens
6.53.002.012 I slot launch_slot_: id 0 | task 131 | processing task, is_child = 0
7.24.683.468 I reasoning-budget: deactivated (natural end)
7.42.269.207 I slot print_timing: id 0 | task 131 | prompt eval time = 31681.46 ms / 3563 tokens ( 8.89 ms per token, 112.46 tokens per second)
7.42.269.211 I slot print_timing: id 0 | task 131 | eval time = 17585.72 ms / 48 tokens ( 366.37 ms per token, 2.73 tokens per second)
7.42.269.211 I slot print_timing: id 0 | task 131 | total time = 49267.17 ms / 3611 tokens
7.42.269.212 I slot print_timing: id 0 | task 131 | graphs reused = 171
7.42.270.126 I slot release: id 0 | task 131 | stop processing: n_tokens = 12475, truncated = 0
7.42.270.212 I srv update_slots: all slots are idle
7.53.779.290 I srv params_from_: Chat format: peg-native
7.53.796.320 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 0.677 (> 0.100 thold), f_keep = 0.711
7.53.798.019 I reasoning-budget: activated, budget=2147483647 tokens
7.53.798.130 I slot launch_slot_: id 0 | task 180 | processing task, is_child = 0
8.11.072.889 I slot print_timing: id 0 | task 180 | prompt processing, n_tokens = 96, progress = 0.68, t = 17.27 s / 5.56 tokens per second
^C^CReceived second interrupt, terminating immediately.
(base) mukul@jarvis:~/dev-ai/llama.cpp$
@fraserprice while that would make the model run slower, I don't think that is the case for him. I'm running it on 1x4090 and 3x3090, along with 512GB DDR5, the Q2_K_XL Variant, and at 10k context deep I'm able to sustain token generation of around 10t/s, with a PP of like 65 t/s.
@mtcl , just to make sure, maybe try a fresh build of llama.cpp. also, I found that llama webUI has pretty slow PP, but using the model with opencode the speeds are as expected.. how are you interacting with the model?
Hmm, somethign is still off...
(base) mukul@jarvis:~/dev-ai/llama.cpp$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES="0,2" ./build/bin/llama-server \
--model /media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf \
--alias unsloth/GLM-5.2 \
--ctx-size 262144 \
-fa on \
-np 1 -kvu \
--temp 1.0 \
--top-p 0.95 \
--min-p 0.01 \
--fit on \
-b 4096 -ub 4096 \
--parallel 1 \
--threads 56 \
--jinja \
--host 0.0.0.0 \
--port 10002
0.00.442.607 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.442.610 I device_info:
0.00.533.302 I - CUDA0 : NVIDIA RTX PRO 6000 Blackwell Workstation Edition (97249 MiB, 96657 MiB free)
0.00.620.803 I - CUDA1 : NVIDIA RTX PRO 6000 Blackwell Workstation Edition (97249 MiB, 96675 MiB free)
0.00.620.812 I - CPU : Intel(R) Xeon(R) w9-3495X (515257 MiB, 515257 MiB free)
0.00.620.851 I system_info: n_threads = 56 (n_threads_batch = 56) / 112 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | BLACKWELL_NATIVE_FP4 = 1 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.620.902 I srv init: using 111 threads for HTTP server
0.00.621.104 I srv start: binding port with default address family
0.00.622.279 I srv llama_server: loading model
0.00.622.282 I srv load_model: loading model '/media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf'
0.00.622.320 I common_init_result: fitting params to device memory ...
0.00.622.321 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.08.693.093 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.08.693.098 W load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
0.08.832.702 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.08.996.104 W model has unused tensor blk.78.attn_norm.weight (size = 24576 bytes) -- ignoring
0.08.996.109 W model has unused tensor blk.78.attn_q_a_norm.weight (size = 8192 bytes) -- ignoring
0.08.996.112 W model has unused tensor blk.78.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.08.996.115 W model has unused tensor blk.78.attn_q_a.weight (size = 13369344 bytes) -- ignoring
0.08.996.119 W model has unused tensor blk.78.attn_q_b.weight (size = 35651584 bytes) -- ignoring
0.08.996.121 W model has unused tensor blk.78.attn_kv_a_mqa.weight (size = 3760128 bytes) -- ignoring
0.08.996.132 W model has unused tensor blk.78.attn_k_b.weight (size = 6684672 bytes) -- ignoring
0.08.996.136 W model has unused tensor blk.78.attn_v_b.weight (size = 8912896 bytes) -- ignoring
0.08.996.140 W model has unused tensor blk.78.attn_output.weight (size = 106954752 bytes) -- ignoring
0.08.996.143 W model has unused tensor blk.78.ffn_norm.weight (size = 24576 bytes) -- ignoring
0.08.996.146 W model has unused tensor blk.78.indexer.k_norm.weight (size = 512 bytes) -- ignoring
0.08.996.150 W model has unused tensor blk.78.indexer.k_norm.bias (size = 512 bytes) -- ignoring
0.08.996.153 W model has unused tensor blk.78.indexer.proj.weight (size = 786432 bytes) -- ignoring
0.08.996.156 W model has unused tensor blk.78.indexer.attn_k.weight (size = 835584 bytes) -- ignoring
0.08.996.159 W model has unused tensor blk.78.indexer.attn_q_b.weight (size = 8912896 bytes) -- ignoring
0.08.996.163 W model has unused tensor blk.78.ffn_gate_inp.weight (size = 6291456 bytes) -- ignoring
0.08.996.330 W model has unused tensor blk.78.ffn_gate_exps.weight (size = 1811939328 bytes) -- ignoring
0.08.996.333 W model has unused tensor blk.78.ffn_down_exps.weight (size = 2214592512 bytes) -- ignoring
0.08.996.336 W model has unused tensor blk.78.ffn_up_exps.weight (size = 1811939328 bytes) -- ignoring
0.08.996.339 W model has unused tensor blk.78.ffn_gate_shexp.weight (size = 13369344 bytes) -- ignoring
0.08.996.342 W model has unused tensor blk.78.ffn_down_shexp.weight (size = 13369344 bytes) -- ignoring
0.08.996.346 W model has unused tensor blk.78.ffn_up_shexp.weight (size = 13369344 bytes) -- ignoring
0.08.996.349 W model has unused tensor blk.78.nextn.eh_proj.weight (size = 80216064 bytes) -- ignoring
0.08.996.353 W model has unused tensor blk.78.nextn.enorm.weight (size = 24576 bytes) -- ignoring
0.08.996.357 W model has unused tensor blk.78.nextn.hnorm.weight (size = 24576 bytes) -- ignoring
0.08.996.366 W model has unused tensor blk.78.nextn.shared_head_norm.weight (size = 24576 bytes) -- ignoring
1.59.907.288 W llama_context: n_ctx_seq (262144) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
2.00.698.434 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
2.01.991.602 I srv load_model: initializing slots, n_slots = 1
2.03.007.534 W common_speculative_init: no implementations specified for speculative decoding
2.03.007.640 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
2.03.008.385 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
2.03.008.452 I srv load_model: use `--cache-ram 0` to disable the prompt cache
2.03.008.458 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
2.03.008.461 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
2.03.009.079 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
2.03.040.185 I init: chat template, example_format: '[gMASK]<sop><|system|>Reasoning Effort: Max<|system|>You are a helpful assistant<|user|>Hello<|assistant|><think></think>Hi there<|user|>How are you?<|assistant|><think>'
2.03.058.733 I srv init: init: chat template, thinking = 1
2.03.058.749 I srv llama_server: model loaded
2.03.058.752 I srv llama_server: server is listening on http://0.0.0.0:10002
2.03.058.754 I srv update_slots: all slots are idle
2.42.976.141 I srv operator(): Chat format: peg-native
2.43.054.752 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
2.43.054.759 I srv get_availabl: updating prompt cache
2.43.054.767 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
2.43.054.775 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
2.43.054.781 I srv get_availabl: prompt cache update took 0.02 ms
2.43.060.727 I reasoning-budget: activated, budget=2147483647 tokens
2.43.060.746 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
3.43.084.794 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.08, t = 60.02 s / 68.24 tokens per second
4.14.775.842 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 8192, progress = 0.15, t = 91.72 s / 89.32 tokens per second
4.46.372.778 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 12288, progress = 0.23, t = 123.31 s / 99.65 tokens per second
5.19.854.697 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 16384, progress = 0.31, t = 156.79 s / 104.49 tokens per second
5.53.228.530 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 20480, progress = 0.39, t = 190.17 s / 107.69 tokens per second
6.27.199.526 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 24576, progress = 0.46, t = 224.14 s / 109.65 tokens per second
7.02.121.004 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 28672, progress = 0.54, t = 259.06 s / 110.68 tokens per second
7.37.834.140 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 32768, progress = 0.62, t = 294.77 s / 111.16 tokens per second
7.44.269.274 I srv operator(): Chat format: peg-native
8.13.494.420 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 36864, progress = 0.70, t = 330.43 s / 111.56 tokens per second
8.51.348.344 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 40960, progress = 0.77, t = 368.29 s / 111.22 tokens per second
9.30.327.480 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 45056, progress = 0.85, t = 407.27 s / 110.63 tokens per second
10.08.399.609 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 48665, progress = 0.92, t = 445.34 s / 109.28 tokens per second
10.49.568.788 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 52761, progress = 1.00, t = 486.51 s / 108.45 tokens per second
11.13.707.159 W srv stop: cancel task, id_task = 0
11.14.117.422 I slot release: id 0 | task 0 | stop processing: n_tokens = 52964, truncated = 0
11.14.117.605 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 1.000
11.14.120.335 I reasoning-budget: activated, budget=2147483647 tokens
11.14.120.469 I slot launch_slot_: id 0 | task 10 | processing task, is_child = 0
11.14.120.521 W slot update_slots: id 0 | task 10 | need to evaluate at least 1 token for each active slot (n_past = 52962, task.n_tokens() = 52962)
11.14.120.522 W slot update_slots: id 0 | task 10 | n_past was set to 52961
11.57.274.329 I slot print_timing: id 0 | task 10 | n_decoded = 100, tg = 2.34 t/s, tg_3s = 2.34 t/s
12.00.390.080 I slot print_timing: id 0 | task 10 | n_decoded = 106, tg = 2.31 t/s, tg_3s = 1.93 t/s
12.03.399.923 I slot print_timing: id 0 | task 10 | n_decoded = 113, tg = 2.31 t/s, tg_3s = 2.33 t/s
12.06.886.519 I slot print_timing: id 0 | task 10 | n_decoded = 120, tg = 2.29 t/s, tg_3s = 2.01 t/s
12.09.902.054 I slot print_timing: id 0 | task 10 | n_decoded = 126, tg = 2.28 t/s, tg_3s = 1.99 t/s
12.13.101.169 I slot print_timing: id 0 | task 10 | n_decoded = 134, tg = 2.29 t/s, tg_3s = 2.50 t/s
12.16.112.997 I slot print_timing: id 0 | task 10 | n_decoded = 140, tg = 2.27 t/s, tg_3s = 1.99 t/s
12.19.158.146 I reasoning-budget: deactivated (natural end)
12.19.158.167 I slot print_timing: id 0 | task 10 | n_decoded = 146, tg = 2.26 t/s, tg_3s = 1.97 t/s
12.22.569.353 I slot print_timing: id 0 | task 10 | n_decoded = 154, tg = 2.26 t/s, tg_3s = 2.35 t/s
12.25.781.597 I slot print_timing: id 0 | task 10 | n_decoded = 160, tg = 2.25 t/s, tg_3s = 1.87 t/s
12.28.951.435 I slot print_timing: id 0 | task 10 | n_decoded = 167, tg = 2.25 t/s, tg_3s = 2.21 t/s
12.32.134.480 I slot print_timing: id 0 | task 10 | n_decoded = 175, tg = 2.26 t/s, tg_3s = 2.51 t/s
12.35.150.116 I slot print_timing: id 0 | task 10 | n_decoded = 182, tg = 2.26 t/s, tg_3s = 2.32 t/s
12.38.447.199 I slot print_timing: id 0 | task 10 | n_decoded = 190, tg = 2.27 t/s, tg_3s = 2.43 t/s
12.41.785.432 I slot print_timing: id 0 | task 10 | n_decoded = 197, tg = 2.26 t/s, tg_3s = 2.10 t/s
12.45.041.168 I slot print_timing: id 0 | task 10 | n_decoded = 205, tg = 2.27 t/s, tg_3s = 2.46 t/s
12.48.482.981 I slot print_timing: id 0 | task 10 | n_decoded = 212, tg = 2.26 t/s, tg_3s = 2.03 t/s
12.51.637.120 I slot print_timing: id 0 | task 10 | n_decoded = 220, tg = 2.27 t/s, tg_3s = 2.54 t/s
12.52.604.817 I slot print_timing: id 0 | task 10 | prompt eval time = 452.11 ms / 1 tokens ( 452.11 ms per token, 2.21 tokens per second)
12.52.604.820 I slot print_timing: id 0 | task 10 | eval time = 98032.21 ms / 222 tokens ( 441.59 ms per token, 2.26 tokens per second)
12.52.604.821 I slot print_timing: id 0 | task 10 | total time = 98484.32 ms / 223 tokens
12.52.604.823 I slot print_timing: id 0 | task 10 | graphs reused = 223
12.52.608.248 I slot release: id 0 | task 10 | stop processing: n_tokens = 53183, truncated = 0
12.52.608.689 I srv update_slots: all slots are idle
what is your startup command?
Hmm, somethign is still off...
(base) mukul@jarvis:~/dev-ai/llama.cpp$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES="0,2" ./build/bin/llama-server \ --model /media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf \ --alias unsloth/GLM-5.2 \ --ctx-size 262144 \ -fa on \ -np 1 -kvu \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --fit on \ -b 4096 -ub 4096 \ --parallel 1 \ --threads 56 \ --jinja \ --host 0.0.0.0 \ --port 10002
Why going full F16 for the ctk / ctv? I get 18toks with GLM-5.2-UD-Q2_K_XL on 8x RTX3090 + 256DDR4 PC-3200... sometimes less is MORE!
I give up with both UD-Q4-K-M and UD-Q4-K-XL on llama.cpp. I will download a different model to see if that helps.
UD-Q4-K-XL works great for me, I get about ~20 tps decode on low context with 2x RTX 6000 + Epyc 9B45 with 12ch DDR5 6400. I assume you don't get corrupted outputs so my only guess is that attention layers or such ended up in CPU RAM? What if you skip auto fit and do a rough "-ngl 99 -ot exps=CPU" to place all experts in RAM and the rest on the GPU:s?
About >25 tps in my previous message, I for some reason assumed this was UD-Q2-K-XL. Sorry for any confusion.
@mtcl Of course. I actually run a fork of llama.cpp so I use different settings but I tried these with mainline and it works fine even if not optimally tuned with proper experts placement, thread count etc.
export GGML_OP_OFFLOAD_MIN_BATCH=2048
llama-server \
--model /models/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf \
--alias unsloth/GLM-5.2 \
--jinja \
-fa on \
-ngl 99 \
-ot "blk\.[0-9]\.ffn.=CUDA0" \
-ot "blk\.1[0-9]\.ffn.=CUDA1" \
-ot exps=CPU \
-ctk q8_0 \
-ctv q8_0 \
--ctx-size 131072 \
--no-warmup \
--temp 1.0 \
--top-p 0.95 \
--no-mmap \
-b 8192 -ub 8192 \
--host 0.0.0.0 \
--port 8000
Thank you,
i have changed the command for me like this.
export GGML_OP_OFFLOAD_MIN_BATCH=2048
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES="0,2" ./build/bin/llama-server \
--model /media/mukul/data/models/unsloth/GLM-5.2-GGUF/UD-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf \
--alias unsloth/GLM-5.2 \
--jinja \
-fa on \
-ngl 99 \
-ot "blk\.[0-9]\.ffn.=CUDA0" \
-ot "blk\.1[0-9]\.ffn.=CUDA1" \
-ot exps=CPU \
-ctk q8_0 \
-ctv q8_0 \
--ctx-size 131072 \
--no-warmup \
--temp 1.0 \
--top-p 0.95 \
--min-p 0.01 \
--no-mmap \
-b 8192 -ub 8192 \
-np 1 -kvu \
--parallel 1 \
--threads 56 \
--host 0.0.0.0 \
--port 10002
Let me see how it goes. Thank you!
same result... I am starting to believe i am doomed.
@mtcl You are not doomed. 😅 I would make sure that I have the latest llama.cpp. And I would start small like -b 512 -ub 512 -ctx-size 4096. I would delete -ot arguments and replace it with something like --n-cpu-moe 64, delete --no-warmup, etc.
Make sure you compiled llama.cpp with the right CUDA architecture number. Try not specifying the CUDA architecture number, so you are covered for all CUDA devices. I know you have been in SGLang / vLLM camp, so you may need to familiarize yourself with llama.cpp.
If you have CUDA Toolkit 13.2, it is known to cause issues. Install different CUDA Toolkit version.
Thank you @geveent for the hope :)
i have been with ollama ( yep, not proud), then llama.cpp, ik_llama, and ktransformers, before moving to vllm and sglang. I have these huge ggufs that can only run on llama.cpp and dont normally fir on 192GB of vram. This is something new that I am seeing with this model.
I indeed have cuda13.2! would you please guide me to where you see that as an issue?
Holy crap! It definitely is CUDA!!!
I installed unsloth studio, pointed the model folder to local, and it works with 15tk/sec
Holy crap! It definitely is CUDA!!!
What CUDA version works? Is 13.3 good?