Buckets:
| (MainProcess pid=109441) INFO 10-31 19:07:02 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'checkpoints/v2_joint_merged'} | |
| (MainProcess pid=109441) INFO 10-31 19:07:02 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=109441) INFO 10-31 19:07:02 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=109441) INFO 10-31 19:07:02 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=8192. | |
| (MainProcess pid=109441) INFO 10-31 19:07:02 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=109441) INFO 10-31 19:07:02 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=109707) INFO 10-31 19:07:08 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/v2_joint_merged', speculative_config=None, tokenizer='checkpoints/v2_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/v2_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=109707) INFO 10-31 19:07:09 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_fde6652289504141bea195a2567a1395 backend=nccl | |
| (EngineCore pid=109707) INFO 10-31 19:07:09 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=109707) INFO 10-31 19:07:09 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=109707) INFO 10-31 19:07:11 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=109707) INFO 10-31 19:07:11 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=109707) INFO 10-31 19:07:12 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=109707) INFO 10-31 19:07:12 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=109707) INFO 10-31 19:07:13 [base_loader.py:82] Initializing model took 1.666174 seconds | |
| (EngineCore pid=109707) INFO 10-31 19:07:13 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 149.29 GiB. | |
| (EngineCore pid=109707) INFO 10-31 19:07:13 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.01it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.01it/s] | |
| (EngineCore pid=109707) INFO 10-31 19:07:14 [default_loader.py:501] Loading weights took 1.02 seconds | |
| (EngineCore pid=109707) INFO 10-31 19:07:16 [model_runner.py:424] Model loading took 7.94 GiB memory and 5.138599 seconds | |
| (EngineCore pid=109707) INFO 10-31 19:07:17 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=109707) INFO 10-31 19:07:17 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=109707) INFO 10-31 19:07:17 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=109707) INFO 10-31 19:07:17 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=109707) INFO 10-31 19:07:17 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=109707) INFO 10-31 19:07:18 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=109707) INFO 10-31 19:07:23 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/1b7b05bf05/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=109707) INFO 10-31 19:07:23 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.64 s | |
| (EngineCore pid=109707) INFO 10-31 19:07:41 [backends.py:398] [backbone] Compiling a graph for compile range (1, 8192) takes 17.90 s | |
| (EngineCore pid=109707) INFO 10-31 19:07:45 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20438035 bytes total | |
| (EngineCore pid=109707) INFO 10-31 19:07:45 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/b1d5e101f914c55f5fc838c2ed5521eb8c930bf896a2c7ba3e6a502b23262c4f/rank_0_0/model | |
| (EngineCore pid=109707) INFO 10-31 19:07:45 [monitor.py:62] [backbone] torch.compile took 26.61 s in total | |
| (EngineCore pid=109707) INFO 10-31 19:09:12 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.13 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<00:51, 1.02s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:46, 1.05it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:33, 1.43it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:26, 1.75it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:24, 1.85it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:22, 2.03it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:21, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:20, 2.06it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:19, 2.10it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:18, 2.26it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:17, 2.31it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:14, 2.64it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:14, 2.60it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:14, 2.55it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:12, 2.90it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:11, 2.97it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:18, 1.80it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:14, 2.21it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:08, 3.74it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:05, 4.90it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:09<00:03, 7.57it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:09<00:02, 10.14it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:09<00:01, 11.58it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:09<00:01, 10.99it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:09<00:01, 13.30it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:09<00:00, 14.59it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:09<00:00, 15.64it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:10<00:00, 14.09it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:10<00:00, 16.02it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:10<00:00, 17.33it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:10<00:00, 14.11it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 11.41it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.68it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.10it/s] | |
| (EngineCore pid=109707) INFO 10-31 19:09:26 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.40 GiB | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [gpu_worker.py:759] Available KV cache memory: 59.04 GiB | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8940 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9060. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [kv_cache_utils.py:2327] GPU KV cache size: 1,740,028 tokens, Maximum concurrency for 32,768 tokens per request: 53.10x | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=109707) INFO 10-31 19:09:29 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=109707) INFO 10-31 19:09:30 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:41, 1.20it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:41, 1.19it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:38, 1.23it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.38it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.46it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.55it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.69it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:23, 1.82it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.01it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:18, 2.22it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.50it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.71it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 2.96it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:10, 3.36it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:08, 3.97it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:07<00:04, 7.35it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:02, 10.46it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:01, 13.11it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:01, 15.24it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:08<00:01, 16.90it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:08<00:00, 18.04it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:08<00:00, 19.08it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:08<00:00, 19.86it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:09<00:00, 20.55it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 21.13it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:09<00:00, 21.52it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:09<00:00, 5.40it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 28.30it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:04, 6.07it/s] Capturing CUDA graphs (FULL): 23%|██▎ | 8/35 [00:02<00:11, 2.30it/s] Capturing CUDA graphs (FULL): 31%|███▏ | 11/35 [00:02<00:06, 3.80it/s] Capturing CUDA graphs (FULL): 40%|████ | 14/35 [00:02<00:03, 5.68it/s] Capturing CUDA graphs (FULL): 49%|████▊ | 17/35 [00:03<00:02, 7.89it/s] Capturing CUDA graphs (FULL): 57%|█████▋ | 20/35 [00:03<00:01, 10.37it/s] Capturing CUDA graphs (FULL): 66%|██████▌ | 23/35 [00:03<00:00, 12.95it/s] Capturing CUDA graphs (FULL): 74%|███████▍ | 26/35 [00:03<00:00, 15.64it/s] Capturing CUDA graphs (FULL): 83%|████████▎ | 29/35 [00:03<00:00, 18.24it/s] Capturing CUDA graphs (FULL): 91%|█████████▏| 32/35 [00:03<00:00, 20.56it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:03<00:00, 22.54it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:03<00:00, 9.50it/s] | |
| (EngineCore pid=109707) INFO 10-31 19:10:03 [model_runner.py:1127] Graph capturing finished in 14 secs, took 0.24 GiB | |
| (EngineCore pid=109707) INFO 10-31 19:10:03 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%). | |
| (EngineCore pid=109707) INFO 10-31 19:10:03 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 1.77 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62981222298` (58.66 GiB) to fit into requested memory, or `--kv-cache-memory=71047288320` (66.17 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.04 GiB. | |
| (EngineCore pid=109707) INFO 10-31 19:10:04 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=109707) INFO 10-31 19:10:04 [core.py:395] init engine (profile, create kv cache, warmup model) took 167.44 s (compilation: 26.61 s) | |
| (EngineCore pid=109707) INFO 10-31 19:10:04 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=109441) INFO 10-31 19:10:06 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=109707) WARNING 10-31 19:10:08 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=109441) INFO 10-31 19:10:25 [loggers.py:335] Engine 000: Avg prompt throughput: 21951.5 tokens/s, Avg generation throughput: 34.3 tokens/s, Running: 13 reqs, Waiting: 6054 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=109441) INFO 10-31 19:10:35 [loggers.py:335] Engine 000: Avg prompt throughput: 28320.5 tokens/s, Avg generation throughput: 75.6 tokens/s, Running: 23 reqs, Waiting: 5270 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=109441) INFO 10-31 19:10:45 [loggers.py:335] Engine 000: Avg prompt throughput: 26426.5 tokens/s, Avg generation throughput: 83.4 tokens/s, Running: 29 reqs, Waiting: 4410 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.3% | |
| (MainProcess pid=109441) INFO 10-31 19:10:55 [loggers.py:335] Engine 000: Avg prompt throughput: 26683.0 tokens/s, Avg generation throughput: 84.4 tokens/s, Running: 37 reqs, Waiting: 3556 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 3.7% | |
| (MainProcess pid=109441) INFO 10-31 19:11:05 [loggers.py:335] Engine 000: Avg prompt throughput: 26297.4 tokens/s, Avg generation throughput: 99.3 tokens/s, Running: 29 reqs, Waiting: 2571 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=109441) INFO 10-31 19:11:15 [loggers.py:335] Engine 000: Avg prompt throughput: 26650.8 tokens/s, Avg generation throughput: 85.3 tokens/s, Running: 31 reqs, Waiting: 1716 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=109441) INFO 10-31 19:11:25 [loggers.py:335] Engine 000: Avg prompt throughput: 26174.0 tokens/s, Avg generation throughput: 93.5 tokens/s, Running: 32 reqs, Waiting: 779 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0% | |
| { | |
| "mixed_hard": 574, | |
| "dropped_hard_label_1_of_4": 160, | |
| "mixed_soft": 71, | |
| "mixed_exact": 279, | |
| "dropped_v2_already_sure_and_right": 167, | |
| "dropped_v2_sure_label_is_wrong": 5, | |
| "fresh_dropped_sure_right": 2322, | |
| "fresh_dropped_sure_wrong_or_noisy": 549, | |
| "kept_soft": 71, | |
| "kept_hard": 360, | |
| "kept_exact": 161, | |
| "kept_fresh_reasoning": 1498, | |
| "kept_fresh_general": 1631, | |
| "final_fresh_reasoning": 192, | |
| "final_in_domain": 192, | |
| "final_fresh_general": 96, | |
| "final": 480 | |
| } | |
| (MainProcess pid=109441) INFO 10-31 19:11:34 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=109707) INFO 10-31 19:11:34 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=109707) INFO 10-31 19:11:34 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=109707) INFO 10-31 19:11:34 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=109707) INFO 10-31 19:11:34 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=109441) WARNING 10-31 19:11:39 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=109441) WARNING 10-31 19:11:39 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 109707 | |
Xet Storage Details
- Size:
- 22.6 kB
- Xet hash:
- fe0eb7fa07dc85ec7beb82fcff8bd39be19aa50b6009929e0f604c1181433711
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.