Buckets:
| {"to_generate": 17823, "already_done": 128, "minutes_left": 130.20927441517512} | |
| (MainProcess pid=32715) INFO 10-31 02:51:24 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=32715) INFO 10-31 02:51:24 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1209.26it/s] | |
| (MainProcess pid=32715) INFO 10-31 02:51:27 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=4096. | |
| (MainProcess pid=32715) INFO 10-31 02:51:27 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.51it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 6.77it/s] | |
| (MainProcess pid=32715) INFO 10-31 02:51:27 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=32715) WARNING 10-31 02:51:39 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=33144) INFO 10-31 02:51:49 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=33144) INFO 10-31 02:51:51 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_4dc4c53d0d1649a08d76719d18d1fba3 backend=nccl | |
| (EngineCore pid=33144) INFO 10-31 02:51:51 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=33144) INFO 10-31 02:51:51 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=33144) INFO 10-31 02:51:52 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1022.63it/s] | |
| (EngineCore pid=33144) INFO 10-31 02:51:53 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=33144) INFO 10-31 02:51:53 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=33144) INFO 10-31 02:51:53 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=33144) INFO 10-31 02:51:54 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=33144) INFO 10-31 02:51:54 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=33144) INFO 10-31 02:51:57 [base_loader.py:82] Initializing model took 5.253679 seconds | |
| (EngineCore pid=33144) INFO 10-31 02:51:58 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.62 GiB. | |
| (EngineCore pid=33144) INFO 10-31 02:51:58 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.53it/s] | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.59it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.58it/s] | |
| (EngineCore pid=33144) INFO 10-31 02:51:58 [default_loader.py:501] Loading weights took 0.78 seconds | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (MainProcess pid=32715) INFO 10-31 02:51:59 [base.py:280] Multi-modal warmup completed in 20.728s | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [model_runner.py:424] Model loading took 8.56 GiB memory and 8.143870 seconds | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=33144) INFO 10-31 02:52:00 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=33144) INFO 10-31 02:52:08 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=33144) INFO 10-31 02:52:11 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=33144) INFO 10-31 02:52:16 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/cb9a0bc415/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=33144) INFO 10-31 02:52:16 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.62 s | |
| (EngineCore pid=33144) INFO 10-31 02:52:32 [backends.py:398] [backbone] Compiling a graph for compile range (1, 4096) takes 16.41 s | |
| (EngineCore pid=33144) INFO 10-31 02:52:35 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18908777 bytes total | |
| (EngineCore pid=33144) INFO 10-31 02:52:35 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/0ec029bac608df5cf30f9ac991aa17123c89f1db43b38d77e398d90dc2716034/rank_0_0/model | |
| (EngineCore pid=33144) INFO 10-31 02:52:35 [monitor.py:62] [backbone] torch.compile took 23.99 s in total | |
| (EngineCore pid=33144) INFO 10-31 02:54:01 [monitor.py:91] [backbone] Initial profiling/warmup run took 85.50 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<01:02, 1.25s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:31, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:13, 3.54it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:01<00:08, 5.58it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:02<00:06, 6.15it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:02<00:05, 7.98it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:02<00:03, 9.83it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:02<00:03, 11.41it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:02<00:03, 9.91it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:03<00:04, 7.43it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:03<00:03, 8.44it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:03<00:02, 10.09it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:03<00:02, 11.53it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:03<00:02, 9.52it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:03<00:02, 10.24it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:03<00:01, 11.65it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:04<00:01, 10.82it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:04<00:01, 12.19it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:04<00:01, 12.71it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:07<00:05, 2.23it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:07<00:03, 3.00it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:07<00:02, 4.00it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:07<00:01, 5.20it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:07<00:00, 6.02it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:07<00:00, 7.50it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:07<00:00, 9.04it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:08<00:00, 6.26it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 18.47it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 18.43it/s] | |
| (EngineCore pid=33144) INFO 10-31 02:54:12 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.40 GiB | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [gpu_worker.py:759] Available KV cache memory: 57.76 GiB | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8934 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9066. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:2327] GPU KV cache size: 1,702,036 tokens, Maximum concurrency for 32,768 tokens per request: 51.94x | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=33144) INFO 10-31 02:54:13 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=33144) INFO 10-31 02:54:16 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=33144) INFO 10-31 02:54:16 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:05, 9.17it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:00<00:03, 12.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:00<00:03, 13.68it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:00<00:03, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:00<00:02, 14.70it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:00<00:02, 14.87it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:00<00:02, 14.23it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:01<00:02, 14.82it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:01<00:02, 15.18it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:01<00:02, 15.50it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:01<00:01, 15.66it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:01<00:01, 15.89it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:01<00:01, 15.78it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:01<00:01, 15.97it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:01<00:01, 13.97it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:02<00:01, 14.58it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:02<00:01, 14.93it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:02<00:01, 15.33it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:02<00:00, 15.60it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:02<00:00, 15.72it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:02<00:00, 15.82it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:02<00:00, 15.98it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:02<00:00, 16.12it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:03<00:00, 16.22it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:03<00:00, 12.86it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:05<00:00, 2.26it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:05<00:00, 8.64it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 4%|▍ | 2/51 [00:01<00:47, 1.04it/s] Capturing CUDA graphs (FULL): 6%|▌ | 3/51 [00:02<00:40, 1.18it/s] Capturing CUDA graphs (FULL): 10%|▉ | 5/51 [00:02<00:19, 2.40it/s] Capturing CUDA graphs (FULL): 16%|█▌ | 8/51 [00:02<00:09, 4.62it/s] Capturing CUDA graphs (FULL): 22%|██▏ | 11/51 [00:02<00:05, 7.08it/s] Capturing CUDA graphs (FULL): 27%|██▋ | 14/51 [00:03<00:03, 9.53it/s] Capturing CUDA graphs (FULL): 33%|███▎ | 17/51 [00:03<00:02, 12.10it/s] Capturing CUDA graphs (FULL): 39%|███▉ | 20/51 [00:03<00:02, 14.25it/s] Capturing CUDA graphs (FULL): 45%|████▌ | 23/51 [00:03<00:01, 16.34it/s] Capturing CUDA graphs (FULL): 51%|█████ | 26/51 [00:03<00:01, 18.17it/s] Capturing CUDA graphs (FULL): 57%|█████▋ | 29/51 [00:03<00:01, 19.33it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 32/51 [00:03<00:00, 20.72it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 35/51 [00:04<00:00, 21.60it/s] Capturing CUDA graphs (FULL): 75%|███████▍ | 38/51 [00:04<00:00, 22.42it/s] Capturing CUDA graphs (FULL): 80%|████████ | 41/51 [00:04<00:00, 22.71it/s] Capturing CUDA graphs (FULL): 86%|████████▋ | 44/51 [00:04<00:00, 23.02it/s] Capturing CUDA graphs (FULL): 92%|█████████▏| 47/51 [00:04<00:00, 23.75it/s] Capturing CUDA graphs (FULL): 98%|█████████▊| 50/51 [00:04<00:00, 24.26it/s] Capturing CUDA graphs (FULL): 100%|██████████| 51/51 [00:04<00:00, 10.92it/s] | |
| (EngineCore pid=33144) INFO 10-31 02:54:46 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.30 GiB | |
| (EngineCore pid=33144) INFO 10-31 02:54:46 [gpu_worker.py:971] CUDA graph pool memory: 0.3 GiB (actual), 0.52 GiB (estimated), difference: 0.22 GiB (71.6%). | |
| (EngineCore pid=33144) INFO 10-31 02:54:46 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.44 GiB for consumed memory (weights + non-torch), 2.12 GiB for peak activation, and 0.3 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=61540390810` (57.31 GiB) to fit into requested memory, or `--kv-cache-memory=69606456832` (64.83 GiB) to fully utilize gpu memory. Current kv cache memory in use is 57.76 GiB. | |
| (EngineCore pid=33144) INFO 10-31 02:54:47 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=33144) INFO 10-31 02:54:48 [core.py:395] init engine (profile, create kv cache, warmup model) took 167.36 s (compilation: 23.99 s) | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 5.68it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.46it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.11it/s] | |
| (EngineCore pid=33144) INFO 10-31 02:54:48 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=32715) WARNING 10-31 02:54:48 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=33144) WARNING 10-31 02:54:48 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=33144) WARNING 10-31 02:54:49 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=33144) WARNING 10-31 02:54:50 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| [rank0]:[W1031 02:55:13.175073068 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 383778816 bytes (free: 211681280, total: 85094825984). | |
| [rank0]:[W1031 02:55:22.194705088 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 492830720 bytes (free: 408813568, total: 85094825984). | |
| progress {"time": "02:55:48", "elapsed_min": 1.0, "submitted": 260, "finished": 4, "in_flight": 256, "gen_tokens": 6887, "tok_per_s": 115, "chain_tokens_median": 428.0, "chain_tokens_p90": 563.5, "chain_tokens_max": 601, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 125.8} | |
| progress {"time": "02:56:48", "elapsed_min": 2.0, "submitted": 296, "finished": 40, "in_flight": 256, "gen_tokens": 101154, "tok_per_s": 843, "chain_tokens_median": 602.5, "chain_tokens_p90": 902.6999999999999, "chain_tokens_max": 1334, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 124.8} | |
| progress {"time": "02:57:48", "elapsed_min": 3.0, "submitted": 324, "finished": 68, "in_flight": 256, "gen_tokens": 200050, "tok_per_s": 1111, "chain_tokens_median": 651.0, "chain_tokens_p90": 1132.1000000000001, "chain_tokens_max": 1958, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 123.8} | |
| progress {"time": "02:58:48", "elapsed_min": 4.0, "submitted": 341, "finished": 85, "in_flight": 256, "gen_tokens": 269649, "tok_per_s": 1123, "chain_tokens_median": 685.5, "chain_tokens_p90": 1307.5000000000005, "chain_tokens_max": 2310, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 122.8} | |
| progress {"time": "02:59:48", "elapsed_min": 5.0, "submitted": 358, "finished": 102, "in_flight": 256, "gen_tokens": 358592, "tok_per_s": 1195, "chain_tokens_median": 724.5, "chain_tokens_p90": 1513.3, "chain_tokens_max": 3095, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 121.8} | |
| progress {"time": "03:00:48", "elapsed_min": 6.0, "submitted": 379, "finished": 123, "in_flight": 256, "gen_tokens": 498088, "tok_per_s": 1384, "chain_tokens_median": 774.0, "chain_tokens_p90": 1992.0, "chain_tokens_max": 3682, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 120.8} | |
| progress {"time": "03:01:48", "elapsed_min": 7.0, "submitted": 397, "finished": 141, "in_flight": 256, "gen_tokens": 638033, "tok_per_s": 1519, "chain_tokens_median": 820.5, "chain_tokens_p90": 2423.2999999999997, "chain_tokens_max": 4179, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 119.8} | |
| progress {"time": "03:02:48", "elapsed_min": 8.0, "submitted": 415, "finished": 159, "in_flight": 256, "gen_tokens": 785075, "tok_per_s": 1636, "chain_tokens_median": 859.0, "chain_tokens_p90": 2849.0, "chain_tokens_max": 4470, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 118.8} | |
| progress {"time": "03:03:48", "elapsed_min": 9.0, "submitted": 438, "finished": 182, "in_flight": 256, "gen_tokens": 998263, "tok_per_s": 1849, "chain_tokens_median": 898.0, "chain_tokens_p90": 3189.2000000000016, "chain_tokens_max": 5422, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 117.8} | |
| progress {"time": "03:04:48", "elapsed_min": 10.0, "submitted": 456, "finished": 200, "in_flight": 256, "gen_tokens": 1191397, "tok_per_s": 1986, "chain_tokens_median": 934.0, "chain_tokens_p90": 3527.6000000000004, "chain_tokens_max": 5720, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 116.8} | |
| progress {"time": "03:05:48", "elapsed_min": 11.0, "submitted": 472, "finished": 216, "in_flight": 256, "gen_tokens": 1382639, "tok_per_s": 2095, "chain_tokens_median": 951.0, "chain_tokens_p90": 3925.200000000001, "chain_tokens_max": 6364, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 115.8} | |
| progress {"time": "03:06:48", "elapsed_min": 12.0, "submitted": 489, "finished": 233, "in_flight": 256, "gen_tokens": 1580792, "tok_per_s": 2195, "chain_tokens_median": 995.5, "chain_tokens_p90": 4121.7, "chain_tokens_max": 7296, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 114.8} | |
| progress {"time": "03:07:48", "elapsed_min": 13.0, "submitted": 505, "finished": 249, "in_flight": 256, "gen_tokens": 1822496, "tok_per_s": 2336, "chain_tokens_median": 1051.0, "chain_tokens_p90": 4395.0, "chain_tokens_max": 7713, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 113.8} | |
| {"to_generate": 17571, "already_done": 380, "minutes_left": 152.5916556874911} | |
| (MainProcess pid=34545) INFO 10-31 03:09:13 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=34545) INFO 10-31 03:09:13 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1261.92it/s] | |
| (MainProcess pid=34545) INFO 10-31 03:09:15 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=34545) INFO 10-31 03:09:15 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 7.07it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 12.43it/s] | |
| (MainProcess pid=34545) INFO 10-31 03:09:16 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=34545) WARNING 10-31 03:09:27 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=34971) INFO 10-31 03:09:38 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=34971) INFO 10-31 03:09:39 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_07bc7aa7b635477393202b7f37993a68 backend=nccl | |
| (EngineCore pid=34971) INFO 10-31 03:09:39 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=34971) INFO 10-31 03:09:39 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=34971) INFO 10-31 03:09:41 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1120.55it/s] | |
| (EngineCore pid=34971) INFO 10-31 03:09:42 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=34971) INFO 10-31 03:09:42 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=34971) INFO 10-31 03:09:42 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=34971) INFO 10-31 03:09:42 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=34971) INFO 10-31 03:09:42 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=34971) INFO 10-31 03:09:43 [base_loader.py:82] Initializing model took 2.141693 seconds | |
| (EngineCore pid=34971) INFO 10-31 03:09:43 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.17 GiB. | |
| (EngineCore pid=34971) INFO 10-31 03:09:43 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.73it/s] | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.85it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.83it/s] | |
| (EngineCore pid=34971) INFO 10-31 03:09:44 [default_loader.py:501] Loading weights took 1.10 seconds | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [model_runner.py:424] Model loading took 8.56 GiB memory and 5.110027 seconds | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=34971) INFO 10-31 03:09:45 [utils.py:347] Using LBNHC KV cache layout. | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (MainProcess pid=34545) INFO 10-31 03:09:48 [base.py:280] Multi-modal warmup completed in 20.651s | |
| (EngineCore pid=34971) INFO 10-31 03:09:53 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=34971) INFO 10-31 03:09:57 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33 | |
| (EngineCore pid=34971) INFO 10-31 03:09:57 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model | |
| (EngineCore pid=34971) INFO 10-31 03:09:57 [monitor.py:62] [backbone] torch.compile took 0.63 s in total | |
| (EngineCore pid=34971) INFO 10-31 03:09:57 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.32 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:15, 2.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:11, 2.80it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:10, 3.12it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:09, 3.28it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:08, 3.41it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:01<00:08, 3.54it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:07, 3.63it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:07, 3.84it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:02<00:07, 3.66it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:02<00:06, 3.62it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.57it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.66it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:03<00:05, 3.78it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:03<00:05, 3.98it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.36it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.83it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.95it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 9.14it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.97it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:04<00:00, 12.66it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:04<00:00, 14.15it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:04<00:00, 15.24it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 15.93it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 16.68it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 17.16it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 6.55it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.31it/s] | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.28 GiB | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [gpu_worker.py:759] Available KV cache memory: 60.55 GiB | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:2327] GPU KV cache size: 1,784,193 tokens, Maximum concurrency for 32,768 tokens per request: 54.45x | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=34971) INFO 10-31 03:10:05 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:14, 2.31it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.57it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.61it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.63it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.76it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.88it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:08, 3.12it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.31it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:07, 3.26it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.30it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.47it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.76it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 4.04it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:04, 4.32it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.71it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.95it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.92it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 8.71it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.20it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:00, 11.14it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 12.18it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 12.66it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 13.23it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.83it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 13.97it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 5.96it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 21.37it/s] Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 20.55it/s] Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 21.18it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 20.86it/s] Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 21.40it/s] Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 21.12it/s] Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.15it/s] | |
| (EngineCore pid=34971) INFO 10-31 03:10:15 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.14 GiB | |
| (EngineCore pid=34971) INFO 10-31 03:10:15 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%). | |
| (EngineCore pid=34971) INFO 10-31 03:10:15 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 8.86 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=64709187482` (60.27 GiB) to fit into requested memory, or `--kv-cache-memory=72775253504` (67.78 GiB) to fully utilize gpu memory. Current kv cache memory in use is 60.55 GiB. | |
| (EngineCore pid=34971) INFO 10-31 03:10:16 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=34971) INFO 10-31 03:10:16 [core.py:395] init engine (profile, create kv cache, warmup model) took 30.91 s (compilation: 0.63 s) | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 2.27it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 2.46it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 2.43it/s] | |
| (EngineCore pid=34971) INFO 10-31 03:10:18 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=34545) WARNING 10-31 03:10:18 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "03:11:16", "elapsed_min": 1.0, "submitted": 257, "finished": 1, "in_flight": 256, "gen_tokens": 2479, "tok_per_s": 41, "chain_tokens_median": 611.0, "chain_tokens_p90": 688.8, "chain_tokens_max": 708, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 150.5} | |
| (MainProcess pid=34545) INFO 10-31 03:11:16 [loggers.py:335] Engine 000: Avg prompt throughput: 893.0 tokens/s, Avg generation throughput: 5140.7 tokens/s, Running: 128 reqs, Waiting: 887 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 51.6% | |
| progress {"time": "03:12:16", "elapsed_min": 2.0, "submitted": 260, "finished": 4, "in_flight": 256, "gen_tokens": 39184, "tok_per_s": 327, "chain_tokens_median": 2926.0, "chain_tokens_p90": 3548.5, "chain_tokens_max": 4055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 149.5} | |
| (MainProcess pid=34545) INFO 10-31 03:12:16 [loggers.py:335] Engine 000: Avg prompt throughput: 105.7 tokens/s, Avg generation throughput: 4575.3 tokens/s, Running: 128 reqs, Waiting: 876 reqs, GPU KV cache usage: 39.5%, Prefix cache hit rate: 52.0% | |
| progress {"time": "03:13:16", "elapsed_min": 3.0, "submitted": 265, "finished": 9, "in_flight": 256, "gen_tokens": 124509, "tok_per_s": 692, "chain_tokens_median": 3425.0, "chain_tokens_p90": 5575.0, "chain_tokens_max": 6435, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 148.5} | |
| (MainProcess pid=34545) INFO 10-31 03:13:16 [loggers.py:335] Engine 000: Avg prompt throughput: 244.5 tokens/s, Avg generation throughput: 4180.3 tokens/s, Running: 128 reqs, Waiting: 848 reqs, GPU KV cache usage: 40.5%, Prefix cache hit rate: 51.0% | |
| progress {"time": "03:14:16", "elapsed_min": 4.0, "submitted": 274, "finished": 18, "in_flight": 256, "gen_tokens": 324945, "tok_per_s": 1354, "chain_tokens_median": 4679.5, "chain_tokens_p90": 7200.1, "chain_tokens_max": 8208, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 147.5} | |
| (MainProcess pid=34545) INFO 10-31 03:14:16 [loggers.py:335] Engine 000: Avg prompt throughput: 299.0 tokens/s, Avg generation throughput: 4190.9 tokens/s, Running: 128 reqs, Waiting: 843 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 53.4% | |
| progress {"time": "03:15:16", "elapsed_min": 5.0, "submitted": 285, "finished": 29, "in_flight": 256, "gen_tokens": 574372, "tok_per_s": 1914, "chain_tokens_median": 5331.0, "chain_tokens_p90": 7960.0, "chain_tokens_max": 10149, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 146.5} | |
| (MainProcess pid=34545) INFO 10-31 03:15:16 [loggers.py:335] Engine 000: Avg prompt throughput: 227.8 tokens/s, Avg generation throughput: 4125.1 tokens/s, Running: 128 reqs, Waiting: 853 reqs, GPU KV cache usage: 42.7%, Prefix cache hit rate: 53.5% | |
| {"to_generate": 17539, "already_done": 412, "minutes_left": 145.53148780266443} | |
| (MainProcess pid=35900) INFO 10-31 03:16:18 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=35900) INFO 10-31 03:16:18 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1383.84it/s] | |
| (MainProcess pid=35900) INFO 10-31 03:16:20 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=35900) INFO 10-31 03:16:20 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 2.41it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.40it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.50it/s] | |
| (MainProcess pid=35900) INFO 10-31 03:16:21 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=35900) WARNING 10-31 03:16:33 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=36321) INFO 10-31 03:16:46 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=36321) INFO 10-31 03:16:48 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2f31013152a343f79b3bc765d85c6842 backend=nccl | |
| (EngineCore pid=36321) INFO 10-31 03:16:48 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=36321) INFO 10-31 03:16:48 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=36321) INFO 10-31 03:16:49 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1122.25it/s] | |
| (EngineCore pid=36321) INFO 10-31 03:16:51 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=36321) INFO 10-31 03:16:51 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=36321) INFO 10-31 03:16:52 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=36321) INFO 10-31 03:16:52 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=36321) INFO 10-31 03:16:52 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=36321) INFO 10-31 03:16:53 [base_loader.py:82] Initializing model took 3.283994 seconds | |
| (EngineCore pid=36321) INFO 10-31 03:16:53 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.42 GiB. | |
| (EngineCore pid=36321) INFO 10-31 03:16:53 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.95it/s] | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.04it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.03it/s] | |
| (EngineCore pid=36321) INFO 10-31 03:16:54 [default_loader.py:501] Loading weights took 0.99 seconds | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (MainProcess pid=35900) INFO 10-31 03:16:55 [base.py:280] Multi-modal warmup completed in 21.908s | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [model_runner.py:424] Model loading took 8.56 GiB memory and 6.200593 seconds | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=36321) INFO 10-31 03:16:55 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=36321) INFO 10-31 03:17:03 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=36321) INFO 10-31 03:17:06 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=36321) INFO 10-31 03:17:11 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/0c75a73c22/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=36321) INFO 10-31 03:17:11 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.78 s | |
| (EngineCore pid=36321) INFO 10-31 03:17:29 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.76 s | |
| (EngineCore pid=36321) INFO 10-31 03:17:32 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18863679 bytes total | |
| (EngineCore pid=36321) INFO 10-31 03:17:32 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/280d235bd2883e7a75f93a251c5ae942da381615d0d2c83834f39fb3bfac00c9/rank_0_0/model | |
| (EngineCore pid=36321) INFO 10-31 03:17:32 [monitor.py:62] [backbone] torch.compile took 25.51 s in total | |
| (EngineCore pid=36321) INFO 10-31 03:18:59 [monitor.py:91] [backbone] Initial profiling/warmup run took 86.76 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:49, 1.00it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:42, 1.14it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:31, 1.50it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:25, 1.87it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:21, 2.12it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:22, 1.97it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:23, 1.90it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:22, 1.94it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:20, 2.03it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:19, 2.08it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.39it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:05<00:14, 2.66it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:13, 2.79it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:12, 2.91it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:06<00:11, 3.18it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:10, 3.45it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:17, 1.90it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:14, 2.29it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:08, 3.80it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:05, 5.53it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:08<00:03, 7.35it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:08<00:02, 9.06it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:02, 10.76it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:09<00:01, 12.23it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:09<00:01, 13.48it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:09<00:01, 13.97it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:01, 14.85it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 15.29it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:09<00:00, 15.65it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 16.22it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 16.24it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:10<00:00, 16.68it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:10<00:00, 17.00it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:10<00:00, 17.21it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.84it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 21.04it/s] | |
| (EngineCore pid=36321) INFO 10-31 03:19:13 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.40 GiB | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [gpu_worker.py:759] Available KV cache memory: 58.06 GiB | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8940 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9060. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:2327] GPU KV cache size: 1,711,059 tokens, Maximum concurrency for 32,768 tokens per request: 52.22x | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=36321) INFO 10-31 03:19:14 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=36321) INFO 10-31 03:19:16 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=36321) INFO 10-31 03:19:16 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:41, 1.20it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:41, 1.19it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:39, 1.22it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.36it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.46it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.69it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.83it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.00it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.24it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:16, 2.43it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.61it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:13, 2.83it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:12, 2.94it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:08<00:10, 3.48it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:06, 5.42it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:08<00:05, 5.74it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:08<00:03, 7.76it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:08<00:02, 9.52it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:02, 9.73it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:08<00:02, 10.98it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:09<00:01, 12.19it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:09<00:01, 13.14it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 12.30it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:09<00:01, 13.19it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:09<00:01, 13.88it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:09<00:00, 14.37it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:09<00:00, 13.15it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:10<00:00, 14.00it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:10<00:00, 14.54it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:10<00:00, 12.50it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:10<00:00, 13.36it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 14.13it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.81it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 24.32it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 18.85it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 20.69it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 21.66it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:01, 19.61it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 20.73it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 21.61it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 21.02it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 20.80it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 21.87it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 22.54it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 21.22it/s] | |
| (EngineCore pid=36321) INFO 10-31 03:19:48 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.24 GiB | |
| (EngineCore pid=36321) INFO 10-31 03:19:48 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (100.8%). | |
| (EngineCore pid=36321) INFO 10-31 03:19:48 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.19 GiB for consumed memory (weights + non-torch), 2.08 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=61932558234` (57.68 GiB) to fit into requested memory, or `--kv-cache-memory=69998624256` (65.19 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.06 GiB. | |
| (EngineCore pid=36321) INFO 10-31 03:19:49 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=36321) INFO 10-31 03:19:49 [core.py:395] init engine (profile, create kv cache, warmup model) took 173.89 s (compilation: 25.51 s) | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 6.70it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 12.37it/s] | |
| (EngineCore pid=36321) INFO 10-31 03:19:50 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=35900) WARNING 10-31 03:19:50 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=36321) WARNING 10-31 03:19:51 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=36321) WARNING 10-31 03:19:51 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=36321) WARNING 10-31 03:19:53 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "03:20:49", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": 141.0} | |
| (MainProcess pid=35900) INFO 10-31 03:20:49 [loggers.py:335] Engine 000: Avg prompt throughput: 1527.8 tokens/s, Avg generation throughput: 4786.9 tokens/s, Running: 256 reqs, Waiting: 759 reqs, GPU KV cache usage: 44.9%, Prefix cache hit rate: 49.6% | |
| progress {"time": "03:21:49", "elapsed_min": 2.0, "submitted": 259, "finished": 3, "in_flight": 256, "gen_tokens": 13116, "tok_per_s": 109, "chain_tokens_median": 911.0, "chain_tokens_p90": 1659.3000000000002, "chain_tokens_max": 1913, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 140.0} | |
| (MainProcess pid=35900) INFO 10-31 03:21:49 [loggers.py:335] Engine 000: Avg prompt throughput: 186.1 tokens/s, Avg generation throughput: 5997.6 tokens/s, Running: 256 reqs, Waiting: 737 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 48.8% | |
| progress {"time": "03:22:49", "elapsed_min": 3.0, "submitted": 269, "finished": 13, "in_flight": 256, "gen_tokens": 67831, "tok_per_s": 377, "chain_tokens_median": 911.0, "chain_tokens_p90": 2925.899999999999, "chain_tokens_max": 3527, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 139.0} | |
| (MainProcess pid=35900) INFO 10-31 03:22:49 [loggers.py:335] Engine 000: Avg prompt throughput: 300.2 tokens/s, Avg generation throughput: 5148.4 tokens/s, Running: 256 reqs, Waiting: 719 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 48.5% | |
| progress {"time": "03:23:49", "elapsed_min": 4.0, "submitted": 284, "finished": 28, "in_flight": 256, "gen_tokens": 212170, "tok_per_s": 884, "chain_tokens_median": 1286.5, "chain_tokens_p90": 4229.0, "chain_tokens_max": 4835, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 138.0} | |
| (MainProcess pid=35900) INFO 10-31 03:23:49 [loggers.py:335] Engine 000: Avg prompt throughput: 476.9 tokens/s, Avg generation throughput: 4717.0 tokens/s, Running: 256 reqs, Waiting: 691 reqs, GPU KV cache usage: 71.0%, Prefix cache hit rate: 47.7% | |
| progress {"time": "03:24:49", "elapsed_min": 5.0, "submitted": 306, "finished": 50, "in_flight": 256, "gen_tokens": 419649, "tok_per_s": 1399, "chain_tokens_median": 1344.0, "chain_tokens_p90": 4643.1, "chain_tokens_max": 6017, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 137.0} | |
| (MainProcess pid=35900) INFO 10-31 03:24:49 [loggers.py:335] Engine 000: Avg prompt throughput: 486.1 tokens/s, Avg generation throughput: 4686.8 tokens/s, Running: 255 reqs, Waiting: 672 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 46.0% | |
| progress {"time": "03:25:49", "elapsed_min": 6.0, "submitted": 329, "finished": 73, "in_flight": 256, "gen_tokens": 692970, "tok_per_s": 1925, "chain_tokens_median": 1565.5, "chain_tokens_p90": 5095.800000000001, "chain_tokens_max": 7096, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 136.0} | |
| (MainProcess pid=35900) INFO 10-31 03:25:49 [loggers.py:335] Engine 000: Avg prompt throughput: 521.3 tokens/s, Avg generation throughput: 4691.0 tokens/s, Running: 256 reqs, Waiting: 662 reqs, GPU KV cache usage: 69.3%, Prefix cache hit rate: 45.1% | |
| progress {"time": "03:26:49", "elapsed_min": 7.0, "submitted": 355, "finished": 99, "in_flight": 256, "gen_tokens": 965101, "tok_per_s": 2298, "chain_tokens_median": 1561.5, "chain_tokens_p90": 5249.0, "chain_tokens_max": 7945, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 135.0} | |
| (MainProcess pid=35900) INFO 10-31 03:26:49 [loggers.py:335] Engine 000: Avg prompt throughput: 419.9 tokens/s, Avg generation throughput: 4759.5 tokens/s, Running: 256 reqs, Waiting: 669 reqs, GPU KV cache usage: 69.1%, Prefix cache hit rate: 44.8% | |
| progress {"time": "03:27:49", "elapsed_min": 8.0, "submitted": 370, "finished": 114, "in_flight": 256, "gen_tokens": 1188153, "tok_per_s": 2475, "chain_tokens_median": 1668.0, "chain_tokens_p90": 5879.0, "chain_tokens_max": 9132, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 134.0} | |
| (MainProcess pid=35900) INFO 10-31 03:27:49 [loggers.py:335] Engine 000: Avg prompt throughput: 452.1 tokens/s, Avg generation throughput: 4806.6 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 70.0%, Prefix cache hit rate: 44.2% | |
| progress {"time": "03:28:49", "elapsed_min": 9.0, "submitted": 401, "finished": 145, "in_flight": 256, "gen_tokens": 1522170, "tok_per_s": 2819, "chain_tokens_median": 1613.0, "chain_tokens_p90": 6142.000000000003, "chain_tokens_max": 10224, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 133.0} | |
| (MainProcess pid=35900) INFO 10-31 03:28:49 [loggers.py:335] Engine 000: Avg prompt throughput: 411.4 tokens/s, Avg generation throughput: 4811.0 tokens/s, Running: 256 reqs, Waiting: 681 reqs, GPU KV cache usage: 72.4%, Prefix cache hit rate: 44.3% | |
| progress {"time": "03:29:49", "elapsed_min": 10.0, "submitted": 423, "finished": 167, "in_flight": 256, "gen_tokens": 1732163, "tok_per_s": 2887, "chain_tokens_median": 1548.5, "chain_tokens_p90": 6075.700000000005, "chain_tokens_max": 11527, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 132.0} | |
| (MainProcess pid=35900) INFO 10-31 03:29:49 [loggers.py:335] Engine 000: Avg prompt throughput: 499.2 tokens/s, Avg generation throughput: 4588.7 tokens/s, Running: 255 reqs, Waiting: 671 reqs, GPU KV cache usage: 72.1%, Prefix cache hit rate: 44.0% | |
| progress {"time": "03:30:49", "elapsed_min": 11.0, "submitted": 446, "finished": 190, "in_flight": 256, "gen_tokens": 2012478, "tok_per_s": 3049, "chain_tokens_median": 1573.5, "chain_tokens_p90": 6281.8, "chain_tokens_max": 12596, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 131.0} | |
| (MainProcess pid=35900) INFO 10-31 03:30:49 [loggers.py:335] Engine 000: Avg prompt throughput: 471.7 tokens/s, Avg generation throughput: 4686.9 tokens/s, Running: 255 reqs, Waiting: 668 reqs, GPU KV cache usage: 71.3%, Prefix cache hit rate: 41.5% | |
| progress {"time": "03:31:49", "elapsed_min": 12.0, "submitted": 467, "finished": 211, "in_flight": 256, "gen_tokens": 2263855, "tok_per_s": 3144, "chain_tokens_median": 1560.0, "chain_tokens_p90": 6360.4000000000015, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 130.0} | |
| (MainProcess pid=35900) INFO 10-31 03:31:49 [loggers.py:335] Engine 000: Avg prompt throughput: 483.3 tokens/s, Avg generation throughput: 4652.8 tokens/s, Running: 256 reqs, Waiting: 669 reqs, GPU KV cache usage: 73.7%, Prefix cache hit rate: 40.1% | |
| progress {"time": "03:32:49", "elapsed_min": 13.0, "submitted": 482, "finished": 226, "in_flight": 256, "gen_tokens": 2413935, "tok_per_s": 3095, "chain_tokens_median": 1565.5, "chain_tokens_p90": 6310.000000000002, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 129.0} | |
| (MainProcess pid=35900) INFO 10-31 03:32:49 [loggers.py:335] Engine 000: Avg prompt throughput: 526.6 tokens/s, Avg generation throughput: 4520.7 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 41.9% | |
| progress {"time": "03:33:49", "elapsed_min": 14.0, "submitted": 505, "finished": 249, "in_flight": 256, "gen_tokens": 2738168, "tok_per_s": 3259, "chain_tokens_median": 1643.0, "chain_tokens_p90": 6368.5, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 128.0} | |
| (MainProcess pid=35900) INFO 10-31 03:33:49 [loggers.py:335] Engine 000: Avg prompt throughput: 409.8 tokens/s, Avg generation throughput: 4486.6 tokens/s, Running: 256 reqs, Waiting: 658 reqs, GPU KV cache usage: 77.4%, Prefix cache hit rate: 41.2% | |
| progress {"time": "03:34:49", "elapsed_min": 15.0, "submitted": 519, "finished": 263, "in_flight": 256, "gen_tokens": 2855036, "tok_per_s": 3172, "chain_tokens_median": 1615.5, "chain_tokens_p90": 6364.799999999999, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 127.0} | |
| (MainProcess pid=35900) INFO 10-31 03:34:49 [loggers.py:335] Engine 000: Avg prompt throughput: 374.6 tokens/s, Avg generation throughput: 4392.8 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 40.4% | |
| progress {"time": "03:35:49", "elapsed_min": 16.0, "submitted": 537, "finished": 281, "in_flight": 256, "gen_tokens": 3076344, "tok_per_s": 3204, "chain_tokens_median": 1685.5, "chain_tokens_p90": 6360.4000000000015, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 126.0} | |
| (MainProcess pid=35900) INFO 10-31 03:35:49 [loggers.py:335] Engine 000: Avg prompt throughput: 410.6 tokens/s, Avg generation throughput: 4410.1 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 82.4%, Prefix cache hit rate: 41.6% | |
| progress {"time": "03:36:49", "elapsed_min": 17.0, "submitted": 554, "finished": 298, "in_flight": 256, "gen_tokens": 3290962, "tok_per_s": 3226, "chain_tokens_median": 1711.0, "chain_tokens_p90": 6386.200000000002, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 125.0} | |
| (MainProcess pid=35900) INFO 10-31 03:36:49 [loggers.py:335] Engine 000: Avg prompt throughput: 289.9 tokens/s, Avg generation throughput: 4346.3 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 41.5% | |
| progress {"time": "03:37:49", "elapsed_min": 18.0, "submitted": 565, "finished": 309, "in_flight": 256, "gen_tokens": 3386051, "tok_per_s": 3135, "chain_tokens_median": 1685.5, "chain_tokens_p90": 6379.0, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 124.0} | |
| (MainProcess pid=35900) INFO 10-31 03:37:49 [loggers.py:335] Engine 000: Avg prompt throughput: 362.4 tokens/s, Avg generation throughput: 4196.7 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 42.0% | |
| progress {"time": "03:38:49", "elapsed_min": 19.0, "submitted": 585, "finished": 329, "in_flight": 256, "gen_tokens": 3747837, "tok_per_s": 3287, "chain_tokens_median": 1776.5, "chain_tokens_p90": 6518.5, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 123.0} | |
| (MainProcess pid=35900) INFO 10-31 03:38:49 [loggers.py:335] Engine 000: Avg prompt throughput: 336.5 tokens/s, Avg generation throughput: 4124.2 tokens/s, Running: 256 reqs, Waiting: 626 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 42.3% | |
| progress {"time": "03:39:49", "elapsed_min": 20.0, "submitted": 606, "finished": 350, "in_flight": 256, "gen_tokens": 4070930, "tok_per_s": 3392, "chain_tokens_median": 1818.0, "chain_tokens_p90": 6618.1, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 122.0} | |
| (MainProcess pid=35900) INFO 10-31 03:39:49 [loggers.py:335] Engine 000: Avg prompt throughput: 519.7 tokens/s, Avg generation throughput: 4038.5 tokens/s, Running: 256 reqs, Waiting: 614 reqs, GPU KV cache usage: 80.7%, Prefix cache hit rate: 41.5% | |
| progress {"time": "03:40:49", "elapsed_min": 21.0, "submitted": 631, "finished": 375, "in_flight": 256, "gen_tokens": 4320153, "tok_per_s": 3428, "chain_tokens_median": 1799.5, "chain_tokens_p90": 6514.9000000000015, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 121.0} | |
| (MainProcess pid=35900) INFO 10-31 03:40:49 [loggers.py:335] Engine 000: Avg prompt throughput: 412.0 tokens/s, Avg generation throughput: 4128.0 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 81.4%, Prefix cache hit rate: 41.3% | |
| progress {"time": "03:41:49", "elapsed_min": 22.0, "submitted": 655, "finished": 399, "in_flight": 256, "gen_tokens": 4543207, "tok_per_s": 3442, "chain_tokens_median": 1776.5, "chain_tokens_p90": 6470.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 120.0} | |
| (MainProcess pid=35900) INFO 10-31 03:41:49 [loggers.py:335] Engine 000: Avg prompt throughput: 495.3 tokens/s, Avg generation throughput: 4080.9 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 82.1%, Prefix cache hit rate: 42.7% | |
| progress {"time": "03:42:49", "elapsed_min": 23.0, "submitted": 670, "finished": 414, "in_flight": 256, "gen_tokens": 4735792, "tok_per_s": 3431, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6518.5, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 119.0} | |
| (MainProcess pid=35900) INFO 10-31 03:42:49 [loggers.py:335] Engine 000: Avg prompt throughput: 405.4 tokens/s, Avg generation throughput: 4141.2 tokens/s, Running: 255 reqs, Waiting: 621 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 43.4% | |
| progress {"time": "03:43:49", "elapsed_min": 24.0, "submitted": 687, "finished": 431, "in_flight": 256, "gen_tokens": 4932832, "tok_per_s": 3425, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6509.500000000001, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 118.0} | |
| (MainProcess pid=35900) INFO 10-31 03:43:49 [loggers.py:335] Engine 000: Avg prompt throughput: 376.4 tokens/s, Avg generation throughput: 4038.6 tokens/s, Running: 255 reqs, Waiting: 615 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 42.6% | |
| progress {"time": "03:44:49", "elapsed_min": 25.0, "submitted": 706, "finished": 450, "in_flight": 256, "gen_tokens": 5123416, "tok_per_s": 3415, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6489.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 117.0} | |
| (MainProcess pid=35900) INFO 10-31 03:44:49 [loggers.py:335] Engine 000: Avg prompt throughput: 463.0 tokens/s, Avg generation throughput: 3885.2 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 43.1% | |
| progress {"time": "03:45:49", "elapsed_min": 26.0, "submitted": 726, "finished": 470, "in_flight": 256, "gen_tokens": 5392241, "tok_per_s": 3456, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6500.500000000002, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 116.0} | |
| (MainProcess pid=35900) INFO 10-31 03:45:49 [loggers.py:335] Engine 000: Avg prompt throughput: 582.9 tokens/s, Avg generation throughput: 3799.7 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 42.3% | |
| progress {"time": "03:46:49", "elapsed_min": 27.0, "submitted": 752, "finished": 496, "in_flight": 256, "gen_tokens": 5643489, "tok_per_s": 3483, "chain_tokens_median": 1783.0, "chain_tokens_p90": 6453.4, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 115.0} | |
| (MainProcess pid=35900) INFO 10-31 03:46:49 [loggers.py:335] Engine 000: Avg prompt throughput: 443.7 tokens/s, Avg generation throughput: 3778.2 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 88.1%, Prefix cache hit rate: 43.1% | |
| progress {"time": "03:47:49", "elapsed_min": 28.0, "submitted": 773, "finished": 517, "in_flight": 256, "gen_tokens": 5881957, "tok_per_s": 3501, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6442.3, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 114.0} | |
| (MainProcess pid=35900) INFO 10-31 03:47:49 [loggers.py:335] Engine 000: Avg prompt throughput: 367.3 tokens/s, Avg generation throughput: 3915.1 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 90.6%, Prefix cache hit rate: 43.1% | |
| progress {"time": "03:48:49", "elapsed_min": 29.0, "submitted": 787, "finished": 531, "in_flight": 256, "gen_tokens": 6035067, "tok_per_s": 3468, "chain_tokens_median": 1808.0, "chain_tokens_p90": 6436.4, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 113.0} | |
| (MainProcess pid=35900) INFO 10-31 03:48:49 [loggers.py:335] Engine 000: Avg prompt throughput: 333.0 tokens/s, Avg generation throughput: 3855.8 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 95.2%, Prefix cache hit rate: 43.9% | |
| progress {"time": "03:49:49", "elapsed_min": 30.0, "submitted": 800, "finished": 544, "in_flight": 256, "gen_tokens": 6123273, "tok_per_s": 3402, "chain_tokens_median": 1764.5, "chain_tokens_p90": 6416.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 112.0} | |
| (MainProcess pid=35900) INFO 10-31 03:49:49 [loggers.py:335] Engine 000: Avg prompt throughput: 308.7 tokens/s, Avg generation throughput: 3666.5 tokens/s, Running: 251 reqs, Waiting: 648 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 43.1% | |
| progress {"time": "03:50:49", "elapsed_min": 31.0, "submitted": 810, "finished": 554, "in_flight": 256, "gen_tokens": 6272934, "tok_per_s": 3372, "chain_tokens_median": 1776.0, "chain_tokens_p90": 6436.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 111.0} | |
| (MainProcess pid=35900) INFO 10-31 03:50:49 [loggers.py:335] Engine 000: Avg prompt throughput: 126.1 tokens/s, Avg generation throughput: 2651.1 tokens/s, Running: 238 reqs, Waiting: 661 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 42.3% | |
| progress {"time": "03:51:49", "elapsed_min": 32.0, "submitted": 822, "finished": 566, "in_flight": 256, "gen_tokens": 6618372, "tok_per_s": 3447, "chain_tokens_median": 1801.5, "chain_tokens_p90": 6442.7, "chain_tokens_max": 32449, "parse_rate": 0.9969081272084805, "looped": 7, "minutes_to_deadline": 110.0} | |
| (MainProcess pid=35900) INFO 10-31 03:51:49 [loggers.py:335] Engine 000: Avg prompt throughput: 341.7 tokens/s, Avg generation throughput: 3195.3 tokens/s, Running: 256 reqs, Waiting: 643 reqs, GPU KV cache usage: 94.6%, Prefix cache hit rate: 43.3% | |
| progress {"time": "03:52:49", "elapsed_min": 33.0, "submitted": 837, "finished": 581, "in_flight": 256, "gen_tokens": 6988842, "tok_per_s": 3529, "chain_tokens_median": 1841.5, "chain_tokens_p90": 6492.700000000001, "chain_tokens_max": 32583, "parse_rate": 0.9948364888123924, "looped": 12, "minutes_to_deadline": 109.0} | |
| (MainProcess pid=35900) INFO 10-31 03:52:49 [loggers.py:335] Engine 000: Avg prompt throughput: 322.2 tokens/s, Avg generation throughput: 3902.3 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 42.3% | |
| progress {"time": "03:53:49", "elapsed_min": 34.0, "submitted": 854, "finished": 598, "in_flight": 256, "gen_tokens": 7192441, "tok_per_s": 3525, "chain_tokens_median": 1826.5, "chain_tokens_p90": 6509.800000000001, "chain_tokens_max": 32583, "parse_rate": 0.9949832775919732, "looped": 12, "minutes_to_deadline": 108.0} | |
| (MainProcess pid=35900) INFO 10-31 03:53:49 [loggers.py:335] Engine 000: Avg prompt throughput: 418.1 tokens/s, Avg generation throughput: 3868.1 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 44.1% | |
| progress {"time": "03:54:49", "elapsed_min": 35.0, "submitted": 873, "finished": 617, "in_flight": 256, "gen_tokens": 7422706, "tok_per_s": 3534, "chain_tokens_median": 1836.5, "chain_tokens_p90": 6512.6, "chain_tokens_max": 32583, "parse_rate": 0.9951377633711507, "looped": 12, "minutes_to_deadline": 107.0} | |
| (MainProcess pid=35900) INFO 10-31 03:54:49 [loggers.py:335] Engine 000: Avg prompt throughput: 363.3 tokens/s, Avg generation throughput: 3884.6 tokens/s, Running: 256 reqs, Waiting: 627 reqs, GPU KV cache usage: 88.1%, Prefix cache hit rate: 44.3% | |
| progress {"time": "03:55:49", "elapsed_min": 36.0, "submitted": 890, "finished": 634, "in_flight": 256, "gen_tokens": 7615307, "tok_per_s": 3525, "chain_tokens_median": 1856.5, "chain_tokens_p90": 6492.5, "chain_tokens_max": 32583, "parse_rate": 0.9952681388012619, "looped": 12, "minutes_to_deadline": 106.0} | |
| (MainProcess pid=35900) INFO 10-31 03:55:49 [loggers.py:335] Engine 000: Avg prompt throughput: 286.0 tokens/s, Avg generation throughput: 3881.2 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 89.1%, Prefix cache hit rate: 43.0% | |
| progress {"time": "03:56:49", "elapsed_min": 37.0, "submitted": 910, "finished": 654, "in_flight": 256, "gen_tokens": 7899181, "tok_per_s": 3558, "chain_tokens_median": 1856.5, "chain_tokens_p90": 6528.5, "chain_tokens_max": 32583, "parse_rate": 0.9950305810397554, "looped": 13, "minutes_to_deadline": 105.0} | |
| (MainProcess pid=35900) INFO 10-31 03:56:49 [loggers.py:335] Engine 000: Avg prompt throughput: 463.4 tokens/s, Avg generation throughput: 3846.6 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 42.8% | |
| progress {"time": "03:57:49", "elapsed_min": 38.0, "submitted": 930, "finished": 674, "in_flight": 256, "gen_tokens": 8145011, "tok_per_s": 3572, "chain_tokens_median": 1879.5, "chain_tokens_p90": 6528.5, "chain_tokens_max": 32583, "parse_rate": 0.9951780415430267, "looped": 13, "minutes_to_deadline": 104.0} | |
| (MainProcess pid=35900) INFO 10-31 03:57:49 [loggers.py:335] Engine 000: Avg prompt throughput: 415.6 tokens/s, Avg generation throughput: 3893.4 tokens/s, Running: 255 reqs, Waiting: 626 reqs, GPU KV cache usage: 87.8%, Prefix cache hit rate: 41.6% | |
| progress {"time": "03:58:49", "elapsed_min": 39.0, "submitted": 947, "finished": 691, "in_flight": 256, "gen_tokens": 8306419, "tok_per_s": 3549, "chain_tokens_median": 1852.5, "chain_tokens_p90": 6511.700000000001, "chain_tokens_max": 32583, "parse_rate": 0.9952966714905933, "looped": 13, "minutes_to_deadline": 103.0} | |
| (MainProcess pid=35900) INFO 10-31 03:58:49 [loggers.py:335] Engine 000: Avg prompt throughput: 335.7 tokens/s, Avg generation throughput: 3902.5 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 42.6% | |
| progress {"time": "03:59:49", "elapsed_min": 40.0, "submitted": 963, "finished": 707, "in_flight": 256, "gen_tokens": 8463387, "tok_per_s": 3526, "chain_tokens_median": 1851.0, "chain_tokens_p90": 6489.900000000001, "chain_tokens_max": 32583, "parse_rate": 0.9954031117397454, "looped": 13, "minutes_to_deadline": 102.0} | |
| (MainProcess pid=35900) INFO 10-31 03:59:49 [loggers.py:335] Engine 000: Avg prompt throughput: 340.6 tokens/s, Avg generation throughput: 3774.5 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 42.9% | |
| progress {"time": "04:00:49", "elapsed_min": 41.0, "submitted": 978, "finished": 722, "in_flight": 256, "gen_tokens": 8668835, "tok_per_s": 3524, "chain_tokens_median": 1849.0, "chain_tokens_p90": 6489.900000000001, "chain_tokens_max": 32630, "parse_rate": 0.9948060941828255, "looped": 15, "minutes_to_deadline": 101.0} | |
| (MainProcess pid=35900) INFO 10-31 04:00:49 [loggers.py:335] Engine 000: Avg prompt throughput: 275.4 tokens/s, Avg generation throughput: 3732.1 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 96.1%, Prefix cache hit rate: 42.8% | |
| progress {"time": "04:01:49", "elapsed_min": 42.0, "submitted": 992, "finished": 736, "in_flight": 256, "gen_tokens": 8781926, "tok_per_s": 3485, "chain_tokens_median": 1820.0, "chain_tokens_p90": 6483.600000000002, "chain_tokens_max": 32630, "parse_rate": 0.9949048913043478, "looped": 15, "minutes_to_deadline": 100.0} | |
| (MainProcess pid=35900) INFO 10-31 04:01:49 [loggers.py:335] Engine 000: Avg prompt throughput: 364.0 tokens/s, Avg generation throughput: 3638.0 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 42.2% | |
| progress {"time": "04:02:50", "elapsed_min": 43.0, "submitted": 1007, "finished": 751, "in_flight": 256, "gen_tokens": 9016631, "tok_per_s": 3495, "chain_tokens_median": 1819.0, "chain_tokens_p90": 6497.200000000002, "chain_tokens_max": 32630, "parse_rate": 0.9943408788282291, "looped": 17, "minutes_to_deadline": 99.0} | |
| (MainProcess pid=35900) INFO 10-31 04:02:50 [loggers.py:335] Engine 000: Avg prompt throughput: 319.1 tokens/s, Avg generation throughput: 3616.6 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 96.8%, Prefix cache hit rate: 42.0% | |
| progress {"time": "04:03:50", "elapsed_min": 44.0, "submitted": 1024, "finished": 768, "in_flight": 256, "gen_tokens": 9255489, "tok_per_s": 3506, "chain_tokens_median": 1806.5, "chain_tokens_p90": 6511.9, "chain_tokens_max": 32655, "parse_rate": 0.994140625, "looped": 18, "minutes_to_deadline": 98.0} | |
| (MainProcess pid=35900) INFO 10-31 04:03:50 [loggers.py:335] Engine 000: Avg prompt throughput: 389.2 tokens/s, Avg generation throughput: 3680.6 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 41.4% | |
| progress {"time": "04:04:50", "elapsed_min": 45.0, "submitted": 1045, "finished": 789, "in_flight": 256, "gen_tokens": 9556145, "tok_per_s": 3539, "chain_tokens_median": 1821.5, "chain_tokens_p90": 6515.0, "chain_tokens_max": 32655, "parse_rate": 0.993979721166033, "looped": 19, "minutes_to_deadline": 97.0} | |
| (MainProcess pid=35900) INFO 10-31 04:04:50 [loggers.py:335] Engine 000: Avg prompt throughput: 407.9 tokens/s, Avg generation throughput: 3659.1 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 93.9%, Prefix cache hit rate: 40.8% | |
| progress {"time": "04:05:50", "elapsed_min": 46.0, "submitted": 1065, "finished": 809, "in_flight": 256, "gen_tokens": 9842770, "tok_per_s": 3566, "chain_tokens_median": 1815.0, "chain_tokens_p90": 6559.5, "chain_tokens_max": 32655, "parse_rate": 0.9935105067985167, "looped": 21, "minutes_to_deadline": 96.0} | |
| (MainProcess pid=35900) INFO 10-31 04:05:50 [loggers.py:335] Engine 000: Avg prompt throughput: 468.4 tokens/s, Avg generation throughput: 3667.6 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 91.0%, Prefix cache hit rate: 40.1% | |
| progress {"time": "04:06:50", "elapsed_min": 47.0, "submitted": 1079, "finished": 823, "in_flight": 256, "gen_tokens": 10018049, "tok_per_s": 3552, "chain_tokens_median": 1806.5, "chain_tokens_p90": 6568.9, "chain_tokens_max": 32655, "parse_rate": 0.9933171324422844, "looped": 22, "minutes_to_deadline": 95.0} | |
| (MainProcess pid=35900) INFO 10-31 04:06:50 [loggers.py:335] Engine 000: Avg prompt throughput: 341.0 tokens/s, Avg generation throughput: 3876.9 tokens/s, Running: 256 reqs, Waiting: 643 reqs, GPU KV cache usage: 94.6%, Prefix cache hit rate: 40.0% | |
| progress {"time": "04:07:50", "elapsed_min": 48.0, "submitted": 1091, "finished": 835, "in_flight": 256, "gen_tokens": 10166900, "tok_per_s": 3530, "chain_tokens_median": 1806.0, "chain_tokens_p90": 6569.1, "chain_tokens_max": 32655, "parse_rate": 0.9934131736526947, "looped": 22, "minutes_to_deadline": 94.0} | |
| (MainProcess pid=35900) INFO 10-31 04:07:50 [loggers.py:335] Engine 000: Avg prompt throughput: 346.6 tokens/s, Avg generation throughput: 3795.8 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.7%, Prefix cache hit rate: 40.4% | |
| progress {"time": "04:08:50", "elapsed_min": 49.0, "submitted": 1104, "finished": 848, "in_flight": 256, "gen_tokens": 10348818, "tok_per_s": 3520, "chain_tokens_median": 1805.5, "chain_tokens_p90": 6592.8, "chain_tokens_max": 32655, "parse_rate": 0.9932193396226415, "looped": 23, "minutes_to_deadline": 93.0} | |
| (MainProcess pid=35900) INFO 10-31 04:08:50 [loggers.py:335] Engine 000: Avg prompt throughput: 318.9 tokens/s, Avg generation throughput: 3663.4 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 95.6%, Prefix cache hit rate: 38.0% | |
| progress {"time": "04:09:50", "elapsed_min": 50.0, "submitted": 1117, "finished": 861, "in_flight": 256, "gen_tokens": 10524016, "tok_per_s": 3508, "chain_tokens_median": 1819.0, "chain_tokens_p90": 6583.200000000002, "chain_tokens_max": 32655, "parse_rate": 0.993321718931475, "looped": 23, "minutes_to_deadline": 92.0} | |
| (MainProcess pid=35900) INFO 10-31 04:09:50 [loggers.py:335] Engine 000: Avg prompt throughput: 356.5 tokens/s, Avg generation throughput: 3701.9 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 38.2% | |
| progress {"time": "04:10:50", "elapsed_min": 51.0, "submitted": 1132, "finished": 876, "in_flight": 256, "gen_tokens": 10729934, "tok_per_s": 3506, "chain_tokens_median": 1841.5, "chain_tokens_p90": 6569.700000000001, "chain_tokens_max": 32655, "parse_rate": 0.9931506849315068, "looped": 24, "minutes_to_deadline": 91.0} | |
| (MainProcess pid=35900) INFO 10-31 04:10:50 [loggers.py:335] Engine 000: Avg prompt throughput: 268.1 tokens/s, Avg generation throughput: 3605.5 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 97.9%, Prefix cache hit rate: 38.5% | |
| progress {"time": "04:11:50", "elapsed_min": 52.0, "submitted": 1151, "finished": 895, "in_flight": 256, "gen_tokens": 11031197, "tok_per_s": 3535, "chain_tokens_median": 1862.0, "chain_tokens_p90": 6619.699999999999, "chain_tokens_max": 32655, "parse_rate": 0.9930167597765364, "looped": 25, "minutes_to_deadline": 90.0} | |
| (MainProcess pid=35900) INFO 10-31 04:11:50 [loggers.py:335] Engine 000: Avg prompt throughput: 360.9 tokens/s, Avg generation throughput: 3667.8 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 38.2% | |
| progress {"time": "04:12:50", "elapsed_min": 53.0, "submitted": 1166, "finished": 910, "in_flight": 256, "gen_tokens": 11272904, "tok_per_s": 3545, "chain_tokens_median": 1879.5, "chain_tokens_p90": 6643.699999999999, "chain_tokens_max": 32655, "parse_rate": 0.9925824175824176, "looped": 27, "minutes_to_deadline": 89.0} | |
| (MainProcess pid=35900) INFO 10-31 04:12:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.6 tokens/s, Avg generation throughput: 3637.8 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 38.8% | |
| progress {"time": "04:13:50", "elapsed_min": 54.0, "submitted": 1184, "finished": 928, "in_flight": 256, "gen_tokens": 11499625, "tok_per_s": 3549, "chain_tokens_median": 1898.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9924568965517241, "looped": 28, "minutes_to_deadline": 88.0} | |
| (MainProcess pid=35900) INFO 10-31 04:13:50 [loggers.py:335] Engine 000: Avg prompt throughput: 251.1 tokens/s, Avg generation throughput: 3765.9 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.6%, Prefix cache hit rate: 39.0% | |
| progress {"time": "04:14:50", "elapsed_min": 55.0, "submitted": 1200, "finished": 944, "in_flight": 256, "gen_tokens": 11710817, "tok_per_s": 3548, "chain_tokens_median": 1910.0, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9923199152542372, "looped": 29, "minutes_to_deadline": 87.0} | |
| (MainProcess pid=35900) INFO 10-31 04:14:50 [loggers.py:335] Engine 000: Avg prompt throughput: 354.5 tokens/s, Avg generation throughput: 3650.7 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 39.3% | |
| progress {"time": "04:15:50", "elapsed_min": 56.0, "submitted": 1211, "finished": 955, "in_flight": 256, "gen_tokens": 11871148, "tok_per_s": 3533, "chain_tokens_median": 1909.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9921465968586387, "looped": 30, "minutes_to_deadline": 86.0} | |
| (MainProcess pid=35900) INFO 10-31 04:15:50 [loggers.py:335] Engine 000: Avg prompt throughput: 359.6 tokens/s, Avg generation throughput: 3659.3 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 38.0% | |
| progress {"time": "04:16:50", "elapsed_min": 57.0, "submitted": 1220, "finished": 964, "in_flight": 256, "gen_tokens": 11960124, "tok_per_s": 3497, "chain_tokens_median": 1906.0, "chain_tokens_p90": 6592.0, "chain_tokens_max": 32655, "parse_rate": 0.9922199170124482, "looped": 30, "minutes_to_deadline": 85.0} | |
| (MainProcess pid=35900) INFO 10-31 04:16:50 [loggers.py:335] Engine 000: Avg prompt throughput: 132.4 tokens/s, Avg generation throughput: 2660.8 tokens/s, Running: 237 reqs, Waiting: 640 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 38.6% | |
| progress {"time": "04:17:50", "elapsed_min": 58.0, "submitted": 1231, "finished": 975, "in_flight": 256, "gen_tokens": 12138938, "tok_per_s": 3488, "chain_tokens_median": 1912.5, "chain_tokens_p90": 6585.599999999999, "chain_tokens_max": 32661, "parse_rate": 0.9917948717948718, "looped": 32, "minutes_to_deadline": 84.0} | |
| (MainProcess pid=35900) INFO 10-31 04:17:50 [loggers.py:335] Engine 000: Avg prompt throughput: 195.4 tokens/s, Avg generation throughput: 2792.2 tokens/s, Running: 239 reqs, Waiting: 640 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 38.6% | |
| progress {"time": "04:18:50", "elapsed_min": 59.0, "submitted": 1244, "finished": 988, "in_flight": 256, "gen_tokens": 12267032, "tok_per_s": 3465, "chain_tokens_median": 1914.0, "chain_tokens_p90": 6566.4000000000015, "chain_tokens_max": 32661, "parse_rate": 0.9919028340080972, "looped": 32, "minutes_to_deadline": 83.0} | |
| (MainProcess pid=35900) INFO 10-31 04:18:50 [loggers.py:335] Engine 000: Avg prompt throughput: 268.7 tokens/s, Avg generation throughput: 2550.4 tokens/s, Running: 240 reqs, Waiting: 640 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 38.9% | |
| progress {"time": "04:19:50", "elapsed_min": 60.0, "submitted": 1258, "finished": 1002, "in_flight": 256, "gen_tokens": 12529411, "tok_per_s": 3480, "chain_tokens_median": 1927.0, "chain_tokens_p90": 6556.800000000003, "chain_tokens_max": 32661, "parse_rate": 0.9910179640718563, "looped": 36, "minutes_to_deadline": 82.0} | |
| (MainProcess pid=35900) INFO 10-31 04:19:50 [loggers.py:335] Engine 000: Avg prompt throughput: 356.1 tokens/s, Avg generation throughput: 3663.6 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 97.1%, Prefix cache hit rate: 38.7% | |
| progress {"time": "04:20:50", "elapsed_min": 61.0, "submitted": 1276, "finished": 1020, "in_flight": 256, "gen_tokens": 12767733, "tok_per_s": 3488, "chain_tokens_median": 1935.0, "chain_tokens_p90": 6568.1, "chain_tokens_max": 32667, "parse_rate": 0.9909313725490196, "looped": 37, "minutes_to_deadline": 81.0} | |
| (MainProcess pid=35900) INFO 10-31 04:20:50 [loggers.py:335] Engine 000: Avg prompt throughput: 355.2 tokens/s, Avg generation throughput: 3641.1 tokens/s, Running: 255 reqs, Waiting: 615 reqs, GPU KV cache usage: 94.4%, Prefix cache hit rate: 38.0% | |
| progress {"time": "04:21:50", "elapsed_min": 62.0, "submitted": 1298, "finished": 1042, "in_flight": 256, "gen_tokens": 13042289, "tok_per_s": 3506, "chain_tokens_median": 1938.5, "chain_tokens_p90": 6558.900000000002, "chain_tokens_max": 32667, "parse_rate": 0.9911228406909789, "looped": 37, "minutes_to_deadline": 80.0} | |
| (MainProcess pid=35900) INFO 10-31 04:21:50 [loggers.py:335] Engine 000: Avg prompt throughput: 495.7 tokens/s, Avg generation throughput: 3645.9 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 95.0%, Prefix cache hit rate: 40.6% | |
| progress {"time": "04:22:50", "elapsed_min": 63.0, "submitted": 1317, "finished": 1061, "in_flight": 256, "gen_tokens": 13432876, "tok_per_s": 3553, "chain_tokens_median": 1954.0, "chain_tokens_p90": 6628.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9903393025447691, "looped": 41, "minutes_to_deadline": 79.0} | |
| (MainProcess pid=35900) INFO 10-31 04:22:50 [loggers.py:335] Engine 000: Avg prompt throughput: 418.0 tokens/s, Avg generation throughput: 3765.6 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 42.1% | |
| progress {"time": "04:23:50", "elapsed_min": 64.0, "submitted": 1333, "finished": 1077, "in_flight": 256, "gen_tokens": 13646967, "tok_per_s": 3554, "chain_tokens_median": 1942.0, "chain_tokens_p90": 6651.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9902506963788301, "looped": 42, "minutes_to_deadline": 78.0} | |
| (MainProcess pid=35900) INFO 10-31 04:23:50 [loggers.py:335] Engine 000: Avg prompt throughput: 278.1 tokens/s, Avg generation throughput: 3847.1 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 92.2%, Prefix cache hit rate: 41.5% | |
| progress {"time": "04:24:50", "elapsed_min": 65.0, "submitted": 1345, "finished": 1089, "in_flight": 256, "gen_tokens": 13820340, "tok_per_s": 3543, "chain_tokens_median": 1948.0, "chain_tokens_p90": 6652.0, "chain_tokens_max": 32667, "parse_rate": 0.9901285583103765, "looped": 43, "minutes_to_deadline": 77.0} | |
| (MainProcess pid=35900) INFO 10-31 04:24:50 [loggers.py:335] Engine 000: Avg prompt throughput: 445.3 tokens/s, Avg generation throughput: 3722.9 tokens/s, Running: 255 reqs, Waiting: 621 reqs, GPU KV cache usage: 93.9%, Prefix cache hit rate: 43.2% | |
| progress {"time": "04:25:50", "elapsed_min": 66.0, "submitted": 1362, "finished": 1106, "in_flight": 256, "gen_tokens": 14013295, "tok_per_s": 3538, "chain_tokens_median": 1933.0, "chain_tokens_p90": 6636.300000000003, "chain_tokens_max": 32667, "parse_rate": 0.9898282097649186, "looped": 45, "minutes_to_deadline": 76.0} | |
| (MainProcess pid=35900) INFO 10-31 04:25:50 [loggers.py:335] Engine 000: Avg prompt throughput: 414.3 tokens/s, Avg generation throughput: 3718.6 tokens/s, Running: 255 reqs, Waiting: 619 reqs, GPU KV cache usage: 93.2%, Prefix cache hit rate: 44.2% | |
| progress {"time": "04:26:50", "elapsed_min": 67.0, "submitted": 1386, "finished": 1130, "in_flight": 256, "gen_tokens": 14291902, "tok_per_s": 3555, "chain_tokens_median": 1937.0, "chain_tokens_p90": 6627.299999999999, "chain_tokens_max": 32667, "parse_rate": 0.9900442477876106, "looped": 45, "minutes_to_deadline": 75.0} | |
| (MainProcess pid=35900) INFO 10-31 04:26:50 [loggers.py:335] Engine 000: Avg prompt throughput: 397.6 tokens/s, Avg generation throughput: 3763.7 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 44.8% | |
| progress {"time": "04:27:50", "elapsed_min": 68.0, "submitted": 1401, "finished": 1145, "in_flight": 256, "gen_tokens": 14558872, "tok_per_s": 3568, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6654.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.990174672489083, "looped": 45, "minutes_to_deadline": 74.0} | |
| (MainProcess pid=35900) INFO 10-31 04:27:50 [loggers.py:335] Engine 000: Avg prompt throughput: 279.7 tokens/s, Avg generation throughput: 3821.4 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 92.6%, Prefix cache hit rate: 44.9% | |
| progress {"time": "04:28:50", "elapsed_min": 69.0, "submitted": 1412, "finished": 1156, "in_flight": 256, "gen_tokens": 14703234, "tok_per_s": 3551, "chain_tokens_median": 1948.0, "chain_tokens_p90": 6656.7, "chain_tokens_max": 32667, "parse_rate": 0.9900519031141869, "looped": 46, "minutes_to_deadline": 73.0} | |
| (MainProcess pid=35900) INFO 10-31 04:28:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.5 tokens/s, Avg generation throughput: 3833.6 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 95.5%, Prefix cache hit rate: 45.4% | |
| progress {"time": "04:29:50", "elapsed_min": 70.0, "submitted": 1429, "finished": 1173, "in_flight": 256, "gen_tokens": 14938583, "tok_per_s": 3556, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6649.300000000004, "chain_tokens_max": 32667, "parse_rate": 0.9899829497016198, "looped": 47, "minutes_to_deadline": 72.0} | |
| (MainProcess pid=35900) INFO 10-31 04:29:50 [loggers.py:335] Engine 000: Avg prompt throughput: 368.4 tokens/s, Avg generation throughput: 3748.4 tokens/s, Running: 255 reqs, Waiting: 645 reqs, GPU KV cache usage: 96.5%, Prefix cache hit rate: 45.8% | |
| progress {"time": "04:30:50", "elapsed_min": 71.0, "submitted": 1441, "finished": 1185, "in_flight": 256, "gen_tokens": 15037877, "tok_per_s": 3530, "chain_tokens_median": 1960.0, "chain_tokens_p90": 6626.1, "chain_tokens_max": 32667, "parse_rate": 0.990084388185654, "looped": 47, "minutes_to_deadline": 71.0} | |
| (MainProcess pid=35900) INFO 10-31 04:30:50 [loggers.py:335] Engine 000: Avg prompt throughput: 371.3 tokens/s, Avg generation throughput: 3628.8 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 44.8% | |
| progress {"time": "04:31:50", "elapsed_min": 72.0, "submitted": 1456, "finished": 1200, "in_flight": 256, "gen_tokens": 15179402, "tok_per_s": 3513, "chain_tokens_median": 1949.5, "chain_tokens_p90": 6618.1, "chain_tokens_max": 32667, "parse_rate": 0.9902083333333334, "looped": 47, "minutes_to_deadline": 70.0} | |
| (MainProcess pid=35900) INFO 10-31 04:31:50 [loggers.py:335] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 3543.6 tokens/s, Running: 250 reqs, Waiting: 628 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 45.7% | |
| progress {"time": "04:32:50", "elapsed_min": 73.0, "submitted": 1468, "finished": 1212, "in_flight": 256, "gen_tokens": 15480680, "tok_per_s": 3534, "chain_tokens_median": 1974.5, "chain_tokens_p90": 6651.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.989480198019802, "looped": 51, "minutes_to_deadline": 69.0} | |
| (MainProcess pid=35900) INFO 10-31 04:32:50 [loggers.py:335] Engine 000: Avg prompt throughput: 197.4 tokens/s, Avg generation throughput: 3039.5 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 94.9%, Prefix cache hit rate: 45.4% | |
| progress {"time": "04:33:50", "elapsed_min": 74.0, "submitted": 1487, "finished": 1231, "in_flight": 256, "gen_tokens": 15743030, "tok_per_s": 3545, "chain_tokens_median": 1985.0, "chain_tokens_p90": 6623.899999999999, "chain_tokens_max": 32667, "parse_rate": 0.9892363931762794, "looped": 53, "minutes_to_deadline": 68.0} | |
| (MainProcess pid=35900) INFO 10-31 04:33:50 [loggers.py:335] Engine 000: Avg prompt throughput: 515.6 tokens/s, Avg generation throughput: 3616.0 tokens/s, Running: 256 reqs, Waiting: 615 reqs, GPU KV cache usage: 90.0%, Prefix cache hit rate: 45.3% | |
| progress {"time": "04:34:50", "elapsed_min": 75.0, "submitted": 1504, "finished": 1248, "in_flight": 256, "gen_tokens": 16016777, "tok_per_s": 3559, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6638.100000000005, "chain_tokens_max": 32667, "parse_rate": 0.9891826923076923, "looped": 54, "minutes_to_deadline": 67.0} | |
| (MainProcess pid=35900) INFO 10-31 04:34:50 [loggers.py:335] Engine 000: Avg prompt throughput: 279.1 tokens/s, Avg generation throughput: 3808.2 tokens/s, Running: 256 reqs, Waiting: 608 reqs, GPU KV cache usage: 90.0%, Prefix cache hit rate: 45.7% | |
| progress {"time": "04:35:50", "elapsed_min": 76.0, "submitted": 1527, "finished": 1271, "in_flight": 256, "gen_tokens": 16324688, "tok_per_s": 3580, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6655.4, "chain_tokens_max": 32667, "parse_rate": 0.9891817466561762, "looped": 55, "minutes_to_deadline": 66.0} | |
| (MainProcess pid=35900) INFO 10-31 04:35:50 [loggers.py:335] Engine 000: Avg prompt throughput: 346.3 tokens/s, Avg generation throughput: 3868.3 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 90.3%, Prefix cache hit rate: 45.4% | |
| progress {"time": "04:36:50", "elapsed_min": 77.0, "submitted": 1544, "finished": 1288, "in_flight": 256, "gen_tokens": 16533380, "tok_per_s": 3578, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6653.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9891304347826086, "looped": 56, "minutes_to_deadline": 65.0} | |
| (MainProcess pid=35900) INFO 10-31 04:36:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.0 tokens/s, Avg generation throughput: 3816.0 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 44.9% | |
| progress {"time": "04:37:50", "elapsed_min": 78.0, "submitted": 1564, "finished": 1308, "in_flight": 256, "gen_tokens": 16856942, "tok_per_s": 3602, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6658.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.988914373088685, "looped": 58, "minutes_to_deadline": 64.0} | |
| (MainProcess pid=35900) INFO 10-31 04:37:50 [loggers.py:335] Engine 000: Avg prompt throughput: 427.3 tokens/s, Avg generation throughput: 3778.5 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 43.4% | |
| progress {"time": "04:38:50", "elapsed_min": 79.0, "submitted": 1580, "finished": 1324, "in_flight": 256, "gen_tokens": 17004682, "tok_per_s": 3587, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6641.0, "chain_tokens_max": 32667, "parse_rate": 0.9890483383685801, "looped": 58, "minutes_to_deadline": 63.0} | |
| (MainProcess pid=35900) INFO 10-31 04:38:50 [loggers.py:335] Engine 000: Avg prompt throughput: 350.7 tokens/s, Avg generation throughput: 3880.9 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 89.6%, Prefix cache hit rate: 43.2% | |
| progress {"time": "04:39:50", "elapsed_min": 80.0, "submitted": 1596, "finished": 1340, "in_flight": 256, "gen_tokens": 17225887, "tok_per_s": 3588, "chain_tokens_median": 1997.0, "chain_tokens_p90": 6643.700000000003, "chain_tokens_max": 32667, "parse_rate": 0.9888059701492538, "looped": 60, "minutes_to_deadline": 62.0} | |
| (MainProcess pid=35900) INFO 10-31 04:39:50 [loggers.py:335] Engine 000: Avg prompt throughput: 388.3 tokens/s, Avg generation throughput: 3832.3 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 41.5% | |
| progress {"time": "04:40:50", "elapsed_min": 81.0, "submitted": 1616, "finished": 1360, "in_flight": 256, "gen_tokens": 17502977, "tok_per_s": 3601, "chain_tokens_median": 1995.5, "chain_tokens_p90": 6650.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9887867647058823, "looped": 61, "minutes_to_deadline": 61.0} | |
| (MainProcess pid=35900) INFO 10-31 04:40:50 [loggers.py:335] Engine 000: Avg prompt throughput: 402.5 tokens/s, Avg generation throughput: 3824.8 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 39.1% | |
| progress {"time": "04:41:50", "elapsed_min": 82.0, "submitted": 1640, "finished": 1384, "in_flight": 256, "gen_tokens": 17780909, "tok_per_s": 3614, "chain_tokens_median": 1997.0, "chain_tokens_p90": 6628.5, "chain_tokens_max": 32667, "parse_rate": 0.9888005780346821, "looped": 62, "minutes_to_deadline": 60.0} | |
| (MainProcess pid=35900) INFO 10-31 04:41:50 [loggers.py:335] Engine 000: Avg prompt throughput: 425.1 tokens/s, Avg generation throughput: 3876.4 tokens/s, Running: 255 reqs, Waiting: 638 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 37.7% | |
| progress {"time": "04:42:50", "elapsed_min": 83.0, "submitted": 1652, "finished": 1396, "in_flight": 256, "gen_tokens": 17890485, "tok_per_s": 3592, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.9888968481375359, "looped": 62, "minutes_to_deadline": 59.0} | |
| (MainProcess pid=35900) INFO 10-31 04:42:50 [loggers.py:335] Engine 000: Avg prompt throughput: 319.0 tokens/s, Avg generation throughput: 3851.3 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 37.5% | |
| progress {"time": "04:43:50", "elapsed_min": 84.0, "submitted": 1673, "finished": 1417, "in_flight": 256, "gen_tokens": 18237221, "tok_per_s": 3618, "chain_tokens_median": 1985.0, "chain_tokens_p90": 6640.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885321100917431, "looped": 65, "minutes_to_deadline": 58.0} | |
| (MainProcess pid=35900) INFO 10-31 04:43:50 [loggers.py:335] Engine 000: Avg prompt throughput: 349.9 tokens/s, Avg generation throughput: 3872.2 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 35.4% | |
| progress {"time": "04:44:50", "elapsed_min": 85.0, "submitted": 1697, "finished": 1441, "in_flight": 256, "gen_tokens": 18480870, "tok_per_s": 3623, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6599.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9887231089521166, "looped": 65, "minutes_to_deadline": 57.0} | |
| (MainProcess pid=35900) INFO 10-31 04:44:50 [loggers.py:335] Engine 000: Avg prompt throughput: 569.4 tokens/s, Avg generation throughput: 3876.4 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 82.8%, Prefix cache hit rate: 35.8% | |
| progress {"time": "04:45:50", "elapsed_min": 86.0, "submitted": 1717, "finished": 1461, "in_flight": 256, "gen_tokens": 18658120, "tok_per_s": 3615, "chain_tokens_median": 1961.5, "chain_tokens_p90": 6583.199999999999, "chain_tokens_max": 32667, "parse_rate": 0.9888774811772758, "looped": 65, "minutes_to_deadline": 56.0} | |
| (MainProcess pid=35900) INFO 10-31 04:45:50 [loggers.py:335] Engine 000: Avg prompt throughput: 580.3 tokens/s, Avg generation throughput: 3842.5 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 39.2% | |
| progress {"time": "04:46:50", "elapsed_min": 87.0, "submitted": 1739, "finished": 1483, "in_flight": 256, "gen_tokens": 19007559, "tok_per_s": 3641, "chain_tokens_median": 1973.5, "chain_tokens_p90": 6592.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9887053270397842, "looped": 67, "minutes_to_deadline": 55.0} | |
| (MainProcess pid=35900) INFO 10-31 04:46:50 [loggers.py:335] Engine 000: Avg prompt throughput: 324.8 tokens/s, Avg generation throughput: 3863.6 tokens/s, Running: 256 reqs, Waiting: 656 reqs, GPU KV cache usage: 85.0%, Prefix cache hit rate: 39.5% | |
| progress {"time": "04:47:50", "elapsed_min": 88.0, "submitted": 1758, "finished": 1502, "in_flight": 256, "gen_tokens": 19187177, "tok_per_s": 3634, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6569.3, "chain_tokens_max": 32667, "parse_rate": 0.9888482023968043, "looped": 67, "minutes_to_deadline": 54.0} | |
| (MainProcess pid=35900) INFO 10-31 04:47:50 [loggers.py:335] Engine 000: Avg prompt throughput: 342.4 tokens/s, Avg generation throughput: 3906.4 tokens/s, Running: 256 reqs, Waiting: 652 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 39.2% | |
| progress {"time": "04:48:50", "elapsed_min": 89.0, "submitted": 1777, "finished": 1521, "in_flight": 256, "gen_tokens": 19478374, "tok_per_s": 3647, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6599.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9886587771203156, "looped": 69, "minutes_to_deadline": 53.0} | |
| (MainProcess pid=35900) INFO 10-31 04:48:50 [loggers.py:335] Engine 000: Avg prompt throughput: 432.5 tokens/s, Avg generation throughput: 3944.2 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 82.9%, Prefix cache hit rate: 39.8% | |
| progress {"time": "04:49:50", "elapsed_min": 90.0, "submitted": 1793, "finished": 1537, "in_flight": 256, "gen_tokens": 19708115, "tok_per_s": 3649, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6603.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886141834743006, "looped": 70, "minutes_to_deadline": 52.0} | |
| (MainProcess pid=35900) INFO 10-31 04:49:50 [loggers.py:335] Engine 000: Avg prompt throughput: 544.4 tokens/s, Avg generation throughput: 3923.4 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 40.7% | |
| progress {"time": "04:50:50", "elapsed_min": 91.0, "submitted": 1813, "finished": 1557, "in_flight": 256, "gen_tokens": 19881265, "tok_per_s": 3641, "chain_tokens_median": 1947.0, "chain_tokens_p90": 6585.0, "chain_tokens_max": 32667, "parse_rate": 0.9887604367373154, "looped": 70, "minutes_to_deadline": 51.0} | |
| (MainProcess pid=35900) INFO 10-31 04:50:50 [loggers.py:335] Engine 000: Avg prompt throughput: 480.5 tokens/s, Avg generation throughput: 3897.5 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 83.4%, Prefix cache hit rate: 41.0% | |
| progress {"time": "04:51:50", "elapsed_min": 92.0, "submitted": 1832, "finished": 1576, "in_flight": 256, "gen_tokens": 20101026, "tok_per_s": 3641, "chain_tokens_median": 1946.5, "chain_tokens_p90": 6578.4, "chain_tokens_max": 32667, "parse_rate": 0.9888959390862944, "looped": 70, "minutes_to_deadline": 50.0} | |
| (MainProcess pid=35900) INFO 10-31 04:51:50 [loggers.py:335] Engine 000: Avg prompt throughput: 281.9 tokens/s, Avg generation throughput: 4081.4 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 41.6% | |
| progress {"time": "04:52:50", "elapsed_min": 93.0, "submitted": 1850, "finished": 1594, "in_flight": 256, "gen_tokens": 20312937, "tok_per_s": 3640, "chain_tokens_median": 1950.5, "chain_tokens_p90": 6584.0, "chain_tokens_max": 32667, "parse_rate": 0.989021329987453, "looped": 70, "minutes_to_deadline": 49.0} | |
| (MainProcess pid=35900) INFO 10-31 04:52:50 [loggers.py:335] Engine 000: Avg prompt throughput: 270.4 tokens/s, Avg generation throughput: 3953.5 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 87.4%, Prefix cache hit rate: 42.3% | |
| progress {"time": "04:53:50", "elapsed_min": 94.0, "submitted": 1870, "finished": 1614, "in_flight": 256, "gen_tokens": 20642847, "tok_per_s": 3660, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6597.5, "chain_tokens_max": 32667, "parse_rate": 0.989002478314746, "looped": 71, "minutes_to_deadline": 48.0} | |
| (MainProcess pid=35900) INFO 10-31 04:53:50 [loggers.py:335] Engine 000: Avg prompt throughput: 438.0 tokens/s, Avg generation throughput: 3765.4 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 43.5% | |
| progress {"time": "04:54:50", "elapsed_min": 95.0, "submitted": 1888, "finished": 1632, "in_flight": 256, "gen_tokens": 20868168, "tok_per_s": 3661, "chain_tokens_median": 1952.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32667, "parse_rate": 0.9891237745098039, "looped": 71, "minutes_to_deadline": 47.0} | |
| (MainProcess pid=35900) INFO 10-31 04:54:50 [loggers.py:335] Engine 000: Avg prompt throughput: 429.3 tokens/s, Avg generation throughput: 3867.5 tokens/s, Running: 256 reqs, Waiting: 654 reqs, GPU KV cache usage: 87.8%, Prefix cache hit rate: 44.0% | |
| progress {"time": "04:55:50", "elapsed_min": 96.0, "submitted": 1904, "finished": 1648, "in_flight": 256, "gen_tokens": 21111739, "tok_per_s": 3665, "chain_tokens_median": 1951.5, "chain_tokens_p90": 6601.100000000005, "chain_tokens_max": 32667, "parse_rate": 0.9887742718446602, "looped": 74, "minutes_to_deadline": 46.0} | |
| (MainProcess pid=35900) INFO 10-31 04:55:50 [loggers.py:335] Engine 000: Avg prompt throughput: 464.3 tokens/s, Avg generation throughput: 3799.8 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 44.5% | |
| progress {"time": "04:56:50", "elapsed_min": 97.0, "submitted": 1922, "finished": 1666, "in_flight": 256, "gen_tokens": 21387563, "tok_per_s": 3674, "chain_tokens_median": 1967.0, "chain_tokens_p90": 6592.4, "chain_tokens_max": 32667, "parse_rate": 0.9885954381752701, "looped": 76, "minutes_to_deadline": 45.0} | |
| (MainProcess pid=35900) INFO 10-31 04:56:50 [loggers.py:335] Engine 000: Avg prompt throughput: 278.0 tokens/s, Avg generation throughput: 4004.7 tokens/s, Running: 256 reqs, Waiting: 659 reqs, GPU KV cache usage: 84.1%, Prefix cache hit rate: 44.4% | |
| progress {"time": "04:57:50", "elapsed_min": 98.0, "submitted": 1936, "finished": 1680, "in_flight": 256, "gen_tokens": 21572174, "tok_per_s": 3668, "chain_tokens_median": 1983.0, "chain_tokens_p90": 6585.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9886904761904762, "looped": 76, "minutes_to_deadline": 44.0} | |
| (MainProcess pid=35900) INFO 10-31 04:57:50 [loggers.py:335] Engine 000: Avg prompt throughput: 291.6 tokens/s, Avg generation throughput: 4017.5 tokens/s, Running: 256 reqs, Waiting: 655 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 45.1% | |
| progress {"time": "04:58:50", "elapsed_min": 99.0, "submitted": 1946, "finished": 1690, "in_flight": 256, "gen_tokens": 21687067, "tok_per_s": 3651, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6602.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9887573964497042, "looped": 76, "minutes_to_deadline": 43.0} | |
| (MainProcess pid=35900) INFO 10-31 04:58:50 [loggers.py:335] Engine 000: Avg prompt throughput: 361.5 tokens/s, Avg generation throughput: 3940.7 tokens/s, Running: 256 reqs, Waiting: 633 reqs, GPU KV cache usage: 88.5%, Prefix cache hit rate: 44.3% | |
| progress {"time": "04:59:50", "elapsed_min": 100.0, "submitted": 1964, "finished": 1708, "in_flight": 256, "gen_tokens": 22015887, "tok_per_s": 3669, "chain_tokens_median": 1990.5, "chain_tokens_p90": 6626.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885831381733021, "looped": 78, "minutes_to_deadline": 42.0} | |
| (MainProcess pid=35900) INFO 10-31 04:59:50 [loggers.py:335] Engine 000: Avg prompt throughput: 371.3 tokens/s, Avg generation throughput: 3944.5 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 41.3% | |
| progress {"time": "05:00:50", "elapsed_min": 101.0, "submitted": 1978, "finished": 1722, "in_flight": 256, "gen_tokens": 22123145, "tok_per_s": 3650, "chain_tokens_median": 1970.5, "chain_tokens_p90": 6621.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886759581881533, "looped": 78, "minutes_to_deadline": 41.0} | |
| (MainProcess pid=35900) INFO 10-31 05:00:50 [loggers.py:335] Engine 000: Avg prompt throughput: 522.1 tokens/s, Avg generation throughput: 3794.8 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 41.7% | |
| progress {"time": "05:01:50", "elapsed_min": 102.0, "submitted": 2001, "finished": 1745, "in_flight": 256, "gen_tokens": 22451966, "tok_per_s": 3668, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9888252148997134, "looped": 78, "minutes_to_deadline": 40.0} | |
| (MainProcess pid=35900) INFO 10-31 05:01:50 [loggers.py:335] Engine 000: Avg prompt throughput: 386.5 tokens/s, Avg generation throughput: 3829.2 tokens/s, Running: 255 reqs, Waiting: 628 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 41.9% | |
| progress {"time": "05:02:50", "elapsed_min": 103.0, "submitted": 2022, "finished": 1766, "in_flight": 256, "gen_tokens": 22830935, "tok_per_s": 3694, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6660.4, "chain_tokens_max": 32667, "parse_rate": 0.9888165345413363, "looped": 79, "minutes_to_deadline": 39.0} | |
| (MainProcess pid=35900) INFO 10-31 05:02:50 [loggers.py:335] Engine 000: Avg prompt throughput: 410.6 tokens/s, Avg generation throughput: 3902.0 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 41.6% | |
| progress {"time": "05:03:50", "elapsed_min": 104.0, "submitted": 2044, "finished": 1788, "in_flight": 256, "gen_tokens": 23073636, "tok_per_s": 3697, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6656.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889541387024608, "looped": 79, "minutes_to_deadline": 38.0} | |
| (MainProcess pid=35900) INFO 10-31 05:03:50 [loggers.py:335] Engine 000: Avg prompt throughput: 479.3 tokens/s, Avg generation throughput: 4033.7 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 42.4% | |
| progress {"time": "05:04:50", "elapsed_min": 105.0, "submitted": 2060, "finished": 1804, "in_flight": 256, "gen_tokens": 23228172, "tok_per_s": 3687, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6652.0, "chain_tokens_max": 32667, "parse_rate": 0.9890521064301552, "looped": 79, "minutes_to_deadline": 37.0} | |
| (MainProcess pid=35900) INFO 10-31 05:04:50 [loggers.py:335] Engine 000: Avg prompt throughput: 331.4 tokens/s, Avg generation throughput: 4077.2 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 85.5%, Prefix cache hit rate: 41.6% | |
| progress {"time": "05:05:50", "elapsed_min": 106.0, "submitted": 2077, "finished": 1821, "in_flight": 256, "gen_tokens": 23447235, "tok_per_s": 3686, "chain_tokens_median": 1995.5, "chain_tokens_p90": 6655.4, "chain_tokens_max": 32667, "parse_rate": 0.9891543108182317, "looped": 79, "minutes_to_deadline": 36.0} | |
| (MainProcess pid=35900) INFO 10-31 05:05:50 [loggers.py:335] Engine 000: Avg prompt throughput: 282.0 tokens/s, Avg generation throughput: 4038.8 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 41.3% | |
| progress {"time": "05:06:50", "elapsed_min": 107.0, "submitted": 2096, "finished": 1840, "in_flight": 256, "gen_tokens": 23612614, "tok_per_s": 3678, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6627.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9892663043478261, "looped": 79, "minutes_to_deadline": 35.0} | |
| (MainProcess pid=35900) INFO 10-31 05:06:50 [loggers.py:335] Engine 000: Avg prompt throughput: 410.3 tokens/s, Avg generation throughput: 3902.3 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 40.4% | |
| progress {"time": "05:07:50", "elapsed_min": 108.0, "submitted": 2110, "finished": 1854, "in_flight": 256, "gen_tokens": 23883206, "tok_per_s": 3685, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6648.5, "chain_tokens_max": 32667, "parse_rate": 0.9893473570658037, "looped": 79, "minutes_to_deadline": 34.0} | |
| (MainProcess pid=35900) INFO 10-31 05:07:50 [loggers.py:335] Engine 000: Avg prompt throughput: 224.7 tokens/s, Avg generation throughput: 3949.6 tokens/s, Running: 256 reqs, Waiting: 652 reqs, GPU KV cache usage: 91.7%, Prefix cache hit rate: 40.6% | |
| progress {"time": "05:08:50", "elapsed_min": 109.0, "submitted": 2127, "finished": 1871, "in_flight": 256, "gen_tokens": 24138428, "tok_per_s": 3690, "chain_tokens_median": 2022.5, "chain_tokens_p90": 6636.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9891769107429182, "looped": 81, "minutes_to_deadline": 33.0} | |
| (MainProcess pid=35900) INFO 10-31 05:08:50 [loggers.py:335] Engine 000: Avg prompt throughput: 377.3 tokens/s, Avg generation throughput: 3829.7 tokens/s, Running: 256 reqs, Waiting: 651 reqs, GPU KV cache usage: 89.8%, Prefix cache hit rate: 40.5% | |
| progress {"time": "05:09:50", "elapsed_min": 110.0, "submitted": 2140, "finished": 1884, "in_flight": 256, "gen_tokens": 24231831, "tok_per_s": 3671, "chain_tokens_median": 2009.0, "chain_tokens_p90": 6623.0, "chain_tokens_max": 32667, "parse_rate": 0.9892515923566879, "looped": 81, "minutes_to_deadline": 32.0} | |
| (MainProcess pid=35900) INFO 10-31 05:09:50 [loggers.py:335] Engine 000: Avg prompt throughput: 260.0 tokens/s, Avg generation throughput: 3799.5 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 94.0%, Prefix cache hit rate: 40.2% | |
| progress {"time": "05:10:50", "elapsed_min": 111.0, "submitted": 2162, "finished": 1906, "in_flight": 256, "gen_tokens": 24572038, "tok_per_s": 3689, "chain_tokens_median": 2009.0, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.9889821615949633, "looped": 84, "minutes_to_deadline": 31.0} | |
| (MainProcess pid=35900) INFO 10-31 05:10:50 [loggers.py:335] Engine 000: Avg prompt throughput: 496.6 tokens/s, Avg generation throughput: 3718.2 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 40.4% | |
| progress {"time": "05:11:50", "elapsed_min": 112.0, "submitted": 2178, "finished": 1922, "in_flight": 256, "gen_tokens": 24874085, "tok_per_s": 3701, "chain_tokens_median": 2020.0, "chain_tokens_p90": 6659.6, "chain_tokens_max": 32667, "parse_rate": 0.9888137356919875, "looped": 86, "minutes_to_deadline": 30.0} | |
| (MainProcess pid=35900) INFO 10-31 05:11:50 [loggers.py:335] Engine 000: Avg prompt throughput: 374.9 tokens/s, Avg generation throughput: 3867.5 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 83.7%, Prefix cache hit rate: 40.5% | |
| progress {"time": "05:12:50", "elapsed_min": 113.0, "submitted": 2199, "finished": 1943, "in_flight": 256, "gen_tokens": 25114720, "tok_per_s": 3704, "chain_tokens_median": 2025.0, "chain_tokens_p90": 6655.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889346371590324, "looped": 86, "minutes_to_deadline": 29.0} | |
| (MainProcess pid=35900) INFO 10-31 05:12:50 [loggers.py:335] Engine 000: Avg prompt throughput: 457.7 tokens/s, Avg generation throughput: 3914.7 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 40.6% | |
| progress {"time": "05:13:50", "elapsed_min": 114.0, "submitted": 2221, "finished": 1965, "in_flight": 256, "gen_tokens": 25356254, "tok_per_s": 3707, "chain_tokens_median": 2013.5, "chain_tokens_p90": 6654.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889312977099237, "looped": 87, "minutes_to_deadline": 28.0} | |
| (MainProcess pid=35900) INFO 10-31 05:13:50 [loggers.py:335] Engine 000: Avg prompt throughput: 387.8 tokens/s, Avg generation throughput: 3919.0 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 40.4% | |
| progress {"time": "05:14:50", "elapsed_min": 115.0, "submitted": 2239, "finished": 1983, "in_flight": 256, "gen_tokens": 25570370, "tok_per_s": 3705, "chain_tokens_median": 1999.5, "chain_tokens_p90": 6655.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.988905698436712, "looped": 88, "minutes_to_deadline": 27.0} | |
| (MainProcess pid=35900) INFO 10-31 05:14:50 [loggers.py:335] Engine 000: Avg prompt throughput: 367.5 tokens/s, Avg generation throughput: 3871.7 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 86.6%, Prefix cache hit rate: 41.3% | |
| progress {"time": "05:15:50", "elapsed_min": 116.0, "submitted": 2255, "finished": 1999, "in_flight": 256, "gen_tokens": 25836882, "tok_per_s": 3712, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6658.0, "chain_tokens_max": 32667, "parse_rate": 0.9886193096548274, "looped": 91, "minutes_to_deadline": 26.0} | |
| (MainProcess pid=35900) INFO 10-31 05:15:50 [loggers.py:335] Engine 000: Avg prompt throughput: 307.6 tokens/s, Avg generation throughput: 3923.5 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 85.3%, Prefix cache hit rate: 40.8% | |
| progress {"time": "05:16:50", "elapsed_min": 117.0, "submitted": 2274, "finished": 2018, "in_flight": 256, "gen_tokens": 26057641, "tok_per_s": 3711, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6662.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9887264618434093, "looped": 91, "minutes_to_deadline": 25.0} | |
| (MainProcess pid=35900) INFO 10-31 05:16:50 [loggers.py:335] Engine 000: Avg prompt throughput: 315.4 tokens/s, Avg generation throughput: 3962.0 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 41.4% | |
| progress {"time": "05:17:50", "elapsed_min": 118.0, "submitted": 2287, "finished": 2031, "in_flight": 256, "gen_tokens": 26254328, "tok_per_s": 3708, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6665.099999999999, "chain_tokens_max": 32667, "parse_rate": 0.9885524372230429, "looped": 93, "minutes_to_deadline": 24.0} | |
| (MainProcess pid=35900) INFO 10-31 05:17:50 [loggers.py:335] Engine 000: Avg prompt throughput: 344.3 tokens/s, Avg generation throughput: 3953.5 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 39.9% | |
| progress {"time": "05:18:50", "elapsed_min": 119.0, "submitted": 2301, "finished": 2045, "in_flight": 256, "gen_tokens": 26368518, "tok_per_s": 3693, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6657.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886308068459657, "looped": 93, "minutes_to_deadline": 23.0} | |
| (MainProcess pid=35900) INFO 10-31 05:18:50 [loggers.py:335] Engine 000: Avg prompt throughput: 347.2 tokens/s, Avg generation throughput: 3880.8 tokens/s, Running: 256 reqs, Waiting: 607 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 40.2% | |
| progress {"time": "05:19:50", "elapsed_min": 120.0, "submitted": 2324, "finished": 2068, "in_flight": 256, "gen_tokens": 26790397, "tok_per_s": 3720, "chain_tokens_median": 2010.0, "chain_tokens_p90": 6668.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885154738878144, "looped": 95, "minutes_to_deadline": 22.0} | |
| (MainProcess pid=35900) INFO 10-31 05:19:50 [loggers.py:335] Engine 000: Avg prompt throughput: 438.8 tokens/s, Avg generation throughput: 3892.9 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 40.5% | |
| progress {"time": "05:20:50", "elapsed_min": 121.0, "submitted": 2348, "finished": 2092, "in_flight": 256, "gen_tokens": 27047980, "tok_per_s": 3725, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6663.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886472275334608, "looped": 95, "minutes_to_deadline": 21.0} | |
| (MainProcess pid=35900) INFO 10-31 05:20:50 [loggers.py:335] Engine 000: Avg prompt throughput: 572.7 tokens/s, Avg generation throughput: 3999.0 tokens/s, Running: 255 reqs, Waiting: 619 reqs, GPU KV cache usage: 77.1%, Prefix cache hit rate: 41.4% | |
| progress {"time": "05:21:50", "elapsed_min": 122.0, "submitted": 2373, "finished": 2117, "in_flight": 256, "gen_tokens": 27341202, "tok_per_s": 3735, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6661.0, "chain_tokens_max": 32667, "parse_rate": 0.9887812942843647, "looped": 95, "minutes_to_deadline": 20.0} | |
| (MainProcess pid=35900) INFO 10-31 05:21:50 [loggers.py:335] Engine 000: Avg prompt throughput: 541.2 tokens/s, Avg generation throughput: 4040.6 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 76.7%, Prefix cache hit rate: 41.0% | |
| progress {"time": "05:22:50", "elapsed_min": 123.0, "submitted": 2393, "finished": 2137, "in_flight": 256, "gen_tokens": 27547550, "tok_per_s": 3732, "chain_tokens_median": 2002.5, "chain_tokens_p90": 6659.6, "chain_tokens_max": 32667, "parse_rate": 0.988886289190454, "looped": 95, "minutes_to_deadline": 19.0} | |
| (MainProcess pid=35900) INFO 10-31 05:22:50 [loggers.py:335] Engine 000: Avg prompt throughput: 374.7 tokens/s, Avg generation throughput: 4123.5 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 40.8% | |
| progress {"time": "05:23:50", "elapsed_min": 124.0, "submitted": 2414, "finished": 2158, "in_flight": 256, "gen_tokens": 27769738, "tok_per_s": 3732, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6656.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889944392956441, "looped": 95, "minutes_to_deadline": 18.0} | |
| (MainProcess pid=35900) INFO 10-31 05:23:50 [loggers.py:335] Engine 000: Avg prompt throughput: 457.2 tokens/s, Avg generation throughput: 3940.0 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 80.1%, Prefix cache hit rate: 41.0% | |
| progress {"time": "05:24:50", "elapsed_min": 125.0, "submitted": 2439, "finished": 2183, "in_flight": 256, "gen_tokens": 27992542, "tok_per_s": 3732, "chain_tokens_median": 1974.0, "chain_tokens_p90": 6629.700000000002, "chain_tokens_max": 32667, "parse_rate": 0.98900595510765, "looped": 96, "minutes_to_deadline": 17.0} | |
| (MainProcess pid=35900) INFO 10-31 05:24:50 [loggers.py:335] Engine 000: Avg prompt throughput: 302.6 tokens/s, Avg generation throughput: 4033.4 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 40.7% | |
| progress {"time": "05:25:50", "elapsed_min": 126.0, "submitted": 2457, "finished": 2201, "in_flight": 256, "gen_tokens": 28182219, "tok_per_s": 3727, "chain_tokens_median": 1962.5, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.988982280781463, "looped": 97, "minutes_to_deadline": 16.0} | |
| (MainProcess pid=35900) INFO 10-31 05:25:50 [loggers.py:335] Engine 000: Avg prompt throughput: 450.7 tokens/s, Avg generation throughput: 3986.7 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 82.5%, Prefix cache hit rate: 40.6% | |
| progress {"time": "05:26:50", "elapsed_min": 127.0, "submitted": 2476, "finished": 2220, "in_flight": 256, "gen_tokens": 28408410, "tok_per_s": 3728, "chain_tokens_median": 1957.5, "chain_tokens_p90": 6626.0, "chain_tokens_max": 32667, "parse_rate": 0.9889639639639639, "looped": 98, "minutes_to_deadline": 15.0} | |
| (MainProcess pid=35900) INFO 10-31 05:26:50 [loggers.py:335] Engine 000: Avg prompt throughput: 426.8 tokens/s, Avg generation throughput: 3948.6 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 82.6%, Prefix cache hit rate: 41.0% | |
| progress {"time": "05:27:50", "elapsed_min": 128.0, "submitted": 2495, "finished": 2239, "in_flight": 256, "gen_tokens": 28643720, "tok_per_s": 3729, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6626.5, "chain_tokens_max": 32667, "parse_rate": 0.9890576150066994, "looped": 98, "minutes_to_deadline": 14.0} | |
| (MainProcess pid=35900) INFO 10-31 05:27:50 [loggers.py:335] Engine 000: Avg prompt throughput: 384.8 tokens/s, Avg generation throughput: 3897.1 tokens/s, Running: 256 reqs, Waiting: 628 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 39.8% | |
| progress {"time": "05:28:50", "elapsed_min": 129.0, "submitted": 2516, "finished": 2260, "in_flight": 256, "gen_tokens": 28989670, "tok_per_s": 3745, "chain_tokens_median": 1959.0, "chain_tokens_p90": 6626.1, "chain_tokens_max": 32667, "parse_rate": 0.9889380530973452, "looped": 100, "minutes_to_deadline": 13.0} | |
| (MainProcess pid=35900) INFO 10-31 05:28:50 [loggers.py:335] Engine 000: Avg prompt throughput: 338.9 tokens/s, Avg generation throughput: 4068.6 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 39.8% | |
| progress {"time": "05:29:50", "elapsed_min": 130.0, "submitted": 2537, "finished": 2281, "in_flight": 256, "gen_tokens": 29280613, "tok_per_s": 3753, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9889302937308199, "looped": 101, "minutes_to_deadline": 12.0} | |
| (MainProcess pid=35900) INFO 10-31 05:29:50 [loggers.py:335] Engine 000: Avg prompt throughput: 461.6 tokens/s, Avg generation throughput: 3987.0 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 75.9%, Prefix cache hit rate: 39.9% | |
| progress {"time": "05:30:50", "elapsed_min": 131.0, "submitted": 2561, "finished": 2305, "in_flight": 256, "gen_tokens": 29568331, "tok_per_s": 3761, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9890455531453363, "looped": 101, "minutes_to_deadline": 11.0} | |
| (MainProcess pid=35900) INFO 10-31 05:30:50 [loggers.py:335] Engine 000: Avg prompt throughput: 546.2 tokens/s, Avg generation throughput: 4038.2 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 40.8% | |
| progress {"time": "05:31:50", "elapsed_min": 132.0, "submitted": 2587, "finished": 2331, "in_flight": 256, "gen_tokens": 29895411, "tok_per_s": 3774, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6619.700000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889532389532389, "looped": 103, "minutes_to_deadline": 10.0} | |
| (MainProcess pid=35900) INFO 10-31 05:31:50 [loggers.py:335] Engine 000: Avg prompt throughput: 515.7 tokens/s, Avg generation throughput: 4007.8 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 40.7% | |
| progress {"time": "05:32:50", "elapsed_min": 133.0, "submitted": 2606, "finished": 2350, "in_flight": 256, "gen_tokens": 30078218, "tok_per_s": 3769, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6602.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9889361702127659, "looped": 104, "minutes_to_deadline": 9.0} | |
| (MainProcess pid=35900) INFO 10-31 05:32:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.3 tokens/s, Avg generation throughput: 4367.1 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 39.5% | |
| progress {"time": "05:33:50", "elapsed_min": 134.0, "submitted": 2627, "finished": 2371, "in_flight": 256, "gen_tokens": 30315090, "tok_per_s": 3770, "chain_tokens_median": 1961.5, "chain_tokens_p90": 6584.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9890341628005062, "looped": 104, "minutes_to_deadline": 8.0} | |
| (MainProcess pid=35900) INFO 10-31 05:33:50 [loggers.py:335] Engine 000: Avg prompt throughput: 324.7 tokens/s, Avg generation throughput: 4375.8 tokens/s, Running: 256 reqs, Waiting: 663 reqs, GPU KV cache usage: 77.1%, Prefix cache hit rate: 40.4% | |
| progress {"time": "05:34:50", "elapsed_min": 135.0, "submitted": 2637, "finished": 2381, "in_flight": 256, "gen_tokens": 30456764, "tok_per_s": 3760, "chain_tokens_median": 1970.5, "chain_tokens_p90": 6584.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9890802183956321, "looped": 104, "minutes_to_deadline": 7.0} | |
| (MainProcess pid=35900) INFO 10-31 05:34:50 [loggers.py:335] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 4265.1 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 81.6%, Prefix cache hit rate: 39.4% | |
| progress {"time": "05:35:50", "elapsed_min": 136.0, "submitted": 2656, "finished": 2400, "in_flight": 256, "gen_tokens": 30709556, "tok_per_s": 3763, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6585.0, "chain_tokens_max": 32667, "parse_rate": 0.9891666666666666, "looped": 104, "minutes_to_deadline": 6.0} | |
| (MainProcess pid=35900) INFO 10-31 05:35:50 [loggers.py:335] Engine 000: Avg prompt throughput: 292.4 tokens/s, Avg generation throughput: 4149.7 tokens/s, Running: 256 reqs, Waiting: 662 reqs, GPU KV cache usage: 83.7%, Prefix cache hit rate: 39.8% | |
| progress {"time": "05:36:50", "elapsed_min": 137.0, "submitted": 2669, "finished": 2413, "in_flight": 256, "gen_tokens": 30854327, "tok_per_s": 3753, "chain_tokens_median": 1981.5, "chain_tokens_p90": 6569.9, "chain_tokens_max": 32667, "parse_rate": 0.9892250310816411, "looped": 104, "minutes_to_deadline": 5.0} | |
| (MainProcess pid=35900) INFO 10-31 05:36:50 [loggers.py:335] Engine 000: Avg prompt throughput: 382.3 tokens/s, Avg generation throughput: 4013.1 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 38.7% | |
| progress {"time": "05:37:50", "elapsed_min": 138.0, "submitted": 2686, "finished": 2430, "in_flight": 256, "gen_tokens": 31070233, "tok_per_s": 3752, "chain_tokens_median": 1989.0, "chain_tokens_p90": 6568.1, "chain_tokens_max": 32667, "parse_rate": 0.9893004115226337, "looped": 104, "minutes_to_deadline": 4.0} | |
| (MainProcess pid=35900) INFO 10-31 05:37:50 [loggers.py:335] Engine 000: Avg prompt throughput: 404.9 tokens/s, Avg generation throughput: 3885.1 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 40.2% | |
| progress {"time": "05:38:50", "elapsed_min": 139.0, "submitted": 2702, "finished": 2446, "in_flight": 256, "gen_tokens": 31238513, "tok_per_s": 3745, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6551.700000000001, "chain_tokens_max": 32667, "parse_rate": 0.9893704006541292, "looped": 104, "minutes_to_deadline": 3.0} | |
| (MainProcess pid=35900) INFO 10-31 05:38:50 [loggers.py:335] Engine 000: Avg prompt throughput: 397.4 tokens/s, Avg generation throughput: 3927.2 tokens/s, Running: 256 reqs, Waiting: 634 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 41.3% | |
| progress {"time": "05:39:50", "elapsed_min": 140.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 31464710, "tok_per_s": 3745, "chain_tokens_median": 1993.5, "chain_tokens_p90": 6554.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.9894351889475823, "looped": 104, "minutes_to_deadline": 2.0} | |
| (MainProcess pid=35900) INFO 10-31 05:39:50 [loggers.py:335] Engine 000: Avg prompt throughput: 408.7 tokens/s, Avg generation throughput: 3884.8 tokens/s, Running: 256 reqs, Waiting: 616 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 40.6% | |
| progress {"time": "05:40:50", "elapsed_min": 141.0, "submitted": 2741, "finished": 2485, "in_flight": 256, "gen_tokens": 31833728, "tok_per_s": 3762, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6570.700000000003, "chain_tokens_max": 32667, "parse_rate": 0.9892354124748491, "looped": 107, "minutes_to_deadline": 1.0} | |
| (MainProcess pid=35900) INFO 10-31 05:40:50 [loggers.py:335] Engine 000: Avg prompt throughput: 385.7 tokens/s, Avg generation throughput: 3961.8 tokens/s, Running: 255 reqs, Waiting: 641 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 40.8% | |
| progress {"time": "05:41:50", "elapsed_min": 142.0, "submitted": 2757, "finished": 2501, "in_flight": 256, "gen_tokens": 32068955, "tok_per_s": 3763, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6574.900000000005, "chain_tokens_max": 32667, "parse_rate": 0.989204318272691, "looped": 108, "minutes_to_deadline": -0.0} | |
| (MainProcess pid=35900) INFO 10-31 05:41:50 [loggers.py:335] Engine 000: Avg prompt throughput: 352.1 tokens/s, Avg generation throughput: 4021.7 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 82.5%, Prefix cache hit rate: 41.2% | |
| progress {"time": "05:42:50", "elapsed_min": 143.0, "submitted": 2757, "finished": 2517, "in_flight": 240, "gen_tokens": 32229386, "tok_per_s": 3756, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6569.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9892729439809297, "looped": 108, "minutes_to_deadline": -1.0} | |
| (MainProcess pid=35900) INFO 10-31 05:42:50 [loggers.py:335] Engine 000: Avg prompt throughput: 323.1 tokens/s, Avg generation throughput: 3953.2 tokens/s, Running: 256 reqs, Waiting: 559 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 41.2% | |
| progress {"time": "05:43:50", "elapsed_min": 144.0, "submitted": 2757, "finished": 2548, "in_flight": 209, "gen_tokens": 32584742, "tok_per_s": 3771, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6568.9, "chain_tokens_max": 32667, "parse_rate": 0.9892072213500785, "looped": 110, "minutes_to_deadline": -2.0} | |
| (MainProcess pid=35900) INFO 10-31 05:43:50 [loggers.py:335] Engine 000: Avg prompt throughput: 409.0 tokens/s, Avg generation throughput: 3901.6 tokens/s, Running: 256 reqs, Waiting: 467 reqs, GPU KV cache usage: 80.1%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:44:50", "elapsed_min": 145.0, "submitted": 2757, "finished": 2564, "in_flight": 193, "gen_tokens": 32749380, "tok_per_s": 3764, "chain_tokens_median": 1973.0, "chain_tokens_p90": 6547.0, "chain_tokens_max": 32667, "parse_rate": 0.9891770670826833, "looped": 111, "minutes_to_deadline": -3.0} | |
| (MainProcess pid=35900) INFO 10-31 05:44:50 [loggers.py:335] Engine 000: Avg prompt throughput: 429.9 tokens/s, Avg generation throughput: 3944.5 tokens/s, Running: 256 reqs, Waiting: 387 reqs, GPU KV cache usage: 81.7%, Prefix cache hit rate: 41.2% | |
| progress {"time": "05:45:51", "elapsed_min": 146.0, "submitted": 2757, "finished": 2581, "in_flight": 176, "gen_tokens": 32954949, "tok_per_s": 3761, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6554.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.9891514916698954, "looped": 112, "minutes_to_deadline": -4.0} | |
| (MainProcess pid=35900) INFO 10-31 05:45:51 [loggers.py:335] Engine 000: Avg prompt throughput: 336.4 tokens/s, Avg generation throughput: 4034.4 tokens/s, Running: 256 reqs, Waiting: 328 reqs, GPU KV cache usage: 83.9%, Prefix cache hit rate: 42.1% | |
| progress {"time": "05:46:51", "elapsed_min": 147.0, "submitted": 2757, "finished": 2594, "in_flight": 163, "gen_tokens": 33088724, "tok_per_s": 3751, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6551.5, "chain_tokens_max": 32667, "parse_rate": 0.9891094834232845, "looped": 113, "minutes_to_deadline": -5.0} | |
| (MainProcess pid=35900) INFO 10-31 05:46:51 [loggers.py:335] Engine 000: Avg prompt throughput: 277.8 tokens/s, Avg generation throughput: 4064.4 tokens/s, Running: 256 reqs, Waiting: 268 reqs, GPU KV cache usage: 87.5%, Prefix cache hit rate: 41.8% | |
| progress {"time": "05:47:51", "elapsed_min": 148.0, "submitted": 2757, "finished": 2608, "in_flight": 149, "gen_tokens": 33216830, "tok_per_s": 3740, "chain_tokens_median": 1954.5, "chain_tokens_p90": 6538.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.989167944785276, "looped": 113, "minutes_to_deadline": -6.0} | |
| (MainProcess pid=35900) INFO 10-31 05:47:51 [loggers.py:335] Engine 000: Avg prompt throughput: 263.2 tokens/s, Avg generation throughput: 3970.5 tokens/s, Running: 256 reqs, Waiting: 213 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 41.9% | |
| progress {"time": "05:48:51", "elapsed_min": 149.0, "submitted": 2757, "finished": 2623, "in_flight": 134, "gen_tokens": 33414248, "tok_per_s": 3737, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6541.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.989229889439573, "looped": 113, "minutes_to_deadline": -7.0} | |
| (MainProcess pid=35900) INFO 10-31 05:48:51 [loggers.py:335] Engine 000: Avg prompt throughput: 218.9 tokens/s, Avg generation throughput: 3932.4 tokens/s, Running: 256 reqs, Waiting: 161 reqs, GPU KV cache usage: 94.1%, Prefix cache hit rate: 41.4% | |
| progress {"time": "05:49:51", "elapsed_min": 150.0, "submitted": 2757, "finished": 2636, "in_flight": 121, "gen_tokens": 33635716, "tok_per_s": 3737, "chain_tokens_median": 1962.5, "chain_tokens_p90": 6565.000000000007, "chain_tokens_max": 32667, "parse_rate": 0.9892830045523521, "looped": 113, "minutes_to_deadline": -8.0} | |
| (MainProcess pid=35900) INFO 10-31 05:49:51 [loggers.py:335] Engine 000: Avg prompt throughput: 325.9 tokens/s, Avg generation throughput: 3774.2 tokens/s, Running: 255 reqs, Waiting: 94 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 41.4% | |
| progress {"time": "05:50:51", "elapsed_min": 151.0, "submitted": 2757, "finished": 2652, "in_flight": 105, "gen_tokens": 33814899, "tok_per_s": 3732, "chain_tokens_median": 1967.0, "chain_tokens_p90": 6552.900000000003, "chain_tokens_max": 32667, "parse_rate": 0.9893476621417798, "looped": 113, "minutes_to_deadline": -9.0} | |
| (MainProcess pid=35900) INFO 10-31 05:50:51 [loggers.py:335] Engine 000: Avg prompt throughput: 375.4 tokens/s, Avg generation throughput: 3710.2 tokens/s, Running: 256 reqs, Waiting: 15 reqs, GPU KV cache usage: 90.8%, Prefix cache hit rate: 41.7% | |
| progress {"time": "05:51:51", "elapsed_min": 152.0, "submitted": 2757, "finished": 2675, "in_flight": 82, "gen_tokens": 34201661, "tok_per_s": 3750, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6593.1, "chain_tokens_max": 32667, "parse_rate": 0.9893457943925233, "looped": 114, "minutes_to_deadline": -10.0} | |
| (MainProcess pid=35900) INFO 10-31 05:51:51 [loggers.py:335] Engine 000: Avg prompt throughput: 72.4 tokens/s, Avg generation throughput: 3792.0 tokens/s, Running: 179 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:52:51", "elapsed_min": 153.0, "submitted": 2757, "finished": 2691, "in_flight": 66, "gen_tokens": 34439537, "tok_per_s": 3751, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6604.800000000003, "chain_tokens_max": 32667, "parse_rate": 0.9894091415830546, "looped": 114, "minutes_to_deadline": -11.0} | |
| (MainProcess pid=35900) INFO 10-31 05:52:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3143.7 tokens/s, Running: 133 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.7%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:53:51", "elapsed_min": 154.0, "submitted": 2757, "finished": 2706, "in_flight": 51, "gen_tokens": 34801910, "tok_per_s": 3766, "chain_tokens_median": 1996.5, "chain_tokens_p90": 6629.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.989190687361419, "looped": 117, "minutes_to_deadline": -12.0} | |
| (MainProcess pid=35900) INFO 10-31 05:53:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2680.2 tokens/s, Running: 94 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.4%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:54:51", "elapsed_min": 155.0, "submitted": 2757, "finished": 2718, "in_flight": 39, "gen_tokens": 35051862, "tok_per_s": 3769, "chain_tokens_median": 2011.5, "chain_tokens_p90": 6649.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.9891464311994114, "looped": 118, "minutes_to_deadline": -13.0} | |
| (MainProcess pid=35900) INFO 10-31 05:54:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2183.3 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.6%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:55:51", "elapsed_min": 156.0, "submitted": 2757, "finished": 2724, "in_flight": 33, "gen_tokens": 35211479, "tok_per_s": 3761, "chain_tokens_median": 2018.5, "chain_tokens_p90": 6660.0, "chain_tokens_max": 32667, "parse_rate": 0.9890785609397944, "looped": 119, "minutes_to_deadline": -14.0} | |
| (MainProcess pid=35900) INFO 10-31 05:55:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1754.1 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.0%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:56:51", "elapsed_min": 157.0, "submitted": 2757, "finished": 2732, "in_flight": 25, "gen_tokens": 35544005, "tok_per_s": 3773, "chain_tokens_median": 2027.0, "chain_tokens_p90": 6684.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889275256222547, "looped": 121, "minutes_to_deadline": -15.0} | |
| (MainProcess pid=35900) INFO 10-31 05:56:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1510.1 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.0%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:57:51", "elapsed_min": 158.0, "submitted": 2757, "finished": 2737, "in_flight": 20, "gen_tokens": 35783010, "tok_per_s": 3774, "chain_tokens_median": 2033.5, "chain_tokens_p90": 6719.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9887650712458896, "looped": 123, "minutes_to_deadline": -16.0} | |
| (MainProcess pid=35900) INFO 10-31 05:57:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1310.6 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.2%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:58:51", "elapsed_min": 159.0, "submitted": 2757, "finished": 2738, "in_flight": 19, "gen_tokens": 35836233, "tok_per_s": 3756, "chain_tokens_median": 2035.0, "chain_tokens_p90": 6722.9, "chain_tokens_max": 32667, "parse_rate": 0.9886778670562454, "looped": 124, "minutes_to_deadline": -17.0} | |
| (MainProcess pid=35900) INFO 10-31 05:58:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1234.4 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.2%, Prefix cache hit rate: 41.1% | |
| progress {"time": "05:59:51", "elapsed_min": 160.0, "submitted": 2757, "finished": 2746, "in_flight": 11, "gen_tokens": 36307083, "tok_per_s": 3781, "chain_tokens_median": 2049.5, "chain_tokens_p90": 6773.0, "chain_tokens_max": 32667, "parse_rate": 0.98761835396941, "looped": 136, "minutes_to_deadline": -18.0} | |
| (MainProcess pid=35900) INFO 10-31 05:59:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1096.8 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 41.1% | |
| progress {"time": "06:00:51", "elapsed_min": 161.0, "submitted": 2757, "finished": 2747, "in_flight": 10, "gen_tokens": 36364540, "tok_per_s": 3764, "chain_tokens_median": 2051.5, "chain_tokens_p90": 6777.70000000001, "chain_tokens_max": 32667, "parse_rate": 0.9875318529304696, "looped": 137, "minutes_to_deadline": -19.0} | |
| (MainProcess pid=35900) INFO 10-31 06:00:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 892.5 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 41.1% | |
| progress {"time": "06:01:51", "elapsed_min": 162.0, "submitted": 2757, "finished": 2750, "in_flight": 7, "gen_tokens": 36515259, "tok_per_s": 3756, "chain_tokens_median": 2056.5, "chain_tokens_p90": 6791.1, "chain_tokens_max": 32667, "parse_rate": 0.9872727272727273, "looped": 140, "minutes_to_deadline": -20.0} | |
| (MainProcess pid=35900) INFO 10-31 06:01:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 716.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 41.1% | |
| progress {"time": "06:02:51", "elapsed_min": 163.0, "submitted": 2757, "finished": 2754, "in_flight": 3, "gen_tokens": 36697448, "tok_per_s": 3752, "chain_tokens_median": 2058.5, "chain_tokens_p90": 6802.5, "chain_tokens_max": 32667, "parse_rate": 0.9869281045751634, "looped": 144, "minutes_to_deadline": -21.0} | |
| (MainProcess pid=35900) INFO 10-31 06:02:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 622.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 41.1% | |
| (MainProcess pid=35900) INFO 10-31 06:03:04 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=35900) INFO 10-31 06:03:04 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=35900) INFO 10-31 06:03:04 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=35900) WARNING 10-31 06:03:09 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=35900) WARNING 10-31 06:03:09 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 36321 | |
| (MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:777] [shutdown] MPClient: complete | |
| { | |
| "items": 3169, | |
| "samples": 12676, | |
| "tokens_total": 40974033, | |
| "tokens_median": 1945.0, | |
| "tokens_p90": 6661.0, | |
| "tokens_max": 32667, | |
| "parse_rate": 0.9878510571158094, | |
| "looped": 154, | |
| "items_with_correct_chain": 2818, | |
| "majority_accuracy": 0.8520037866834964, | |
| "this_run": { | |
| "submitted": 2757, | |
| "finished": 2757, | |
| "gen_tokens": 36922408, | |
| "t0": 1793416789.8045087, | |
| "seconds": 9795.067398786545, | |
| "tok_per_s": 3769.489937820551 | |
| }, | |
| "seconds_this_run": 10013.39470243454 | |
| } | |
Xet Storage Details
- Size:
- 179 kB
- Xet hash:
- 63cdd9d77e0a7cec4fd38ec92e363948e904204006008690caf4aace5242d7b6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.