ubergarm

/

MiniMax-M2.7-GGUF

like 16

Tensor blk.61.ffn_down_exps.weight (size = 579.00 MiB) buffer type overriden to CUDA_Host

Allocating 105.31 GiB of pinned host memory, this may take a while.
Using pinned host memory improves PP performance by a significant margin.
But if it takes too long for your model and amount of patience, kill the process and run using

GGML_CUDA_NO_PINNED=1 your_command_goes_here
done allocating 105.31 GiB in 29232.6 ms

llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 63/63 layers to GPU
llm_load_tensors: CUDA_Host buffer size = 107837.70 MiB
llm_load_tensors: CUDA0 buffer size = 3441.36 MiB
....................................................................................................
~ggml_backend_cuda_context: have 0 graphs
llama_init_from_model: n_ctx = 180224
llama_init_from_model: n_batch = 4096
llama_init_from_model: n_ubatch = 4096
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 4096
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 1
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: v_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 5000000.0
llama_init_from_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 23188.03 MiB
llama_init_from_model: KV self size = 23188.00 MiB, K (q8_0): 11594.00 MiB, V (q8_0): 11594.00 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.76 MiB
llama_init_from_model: CUDA0 compute buffer size = 3222.00 MiB
llama_init_from_model: CUDA_Host compute buffer size = 1456.05 MiB
llama_init_from_model: graph nodes = 2361
llama_init_from_model: graph splits = 126
llama_init_from_model: enabling only_active_experts scheduling

main: n_kv_max = 180224, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101