Buckets:

Mercity
/

Desicion-Model-Custom

Files
xet
Mercity/Desicion-Model-Custom / run1 /logs /eval_joint.log
Pranav2748
8 days ago
download
raw
108 kB
(MainProcess pid=50120) INFO 10-31 07:23:55 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=50120) INFO 10-31 07:23:55 [model.py:2152] Using max model len 32768
(MainProcess pid=50120) INFO 10-31 07:23:55 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=50120) INFO 10-31 07:23:55 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=50120) INFO 10-31 07:23:55 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=50398) INFO 10-31 07:24:01 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=50398) INFO 10-31 07:24:02 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2dee883a0b7446c7828a1f57565dc0d8 backend=nccl
(EngineCore pid=50398) INFO 10-31 07:24:02 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=50398) INFO 10-31 07:24:02 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=50398) INFO 10-31 07:24:04 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=50398) INFO 10-31 07:24:04 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=50398) INFO 10-31 07:24:05 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=50398) INFO 10-31 07:24:05 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=50398) INFO 10-31 07:24:05 [base_loader.py:82] Initializing model took 1.592674 seconds
(EngineCore pid=50398) INFO 10-31 07:24:05 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 169.09 GiB.
(EngineCore pid=50398) INFO 10-31 07:24:05 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.

Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.23it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.23it/s]

(EngineCore pid=50398) INFO 10-31 07:24:06 [default_loader.py:501] Loading weights took 0.84 seconds
(EngineCore pid=50398) INFO 10-31 07:24:08 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.422364 seconds
(EngineCore pid=50398) INFO 10-31 07:24:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=50398) INFO 10-31 07:24:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=50398) INFO 10-31 07:24:09 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=50398) INFO 10-31 07:24:09 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=50398) INFO 10-31 07:24:09 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=50398) INFO 10-31 07:24:10 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=50398) INFO 10-31 07:24:15 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/c228b0ef18/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=50398) INFO 10-31 07:24:15 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.45 s
(EngineCore pid=50398) INFO 10-31 07:24:31 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 15.85 s
(EngineCore pid=50398) INFO 10-31 07:24:34 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20434244 bytes total
(EngineCore pid=50398) INFO 10-31 07:24:34 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a0756fe669bc0fbaca33b91d233bcfa2f00d9afbd58c548177a6ef2f7d1ad0b0/rank_0_0/model
(EngineCore pid=50398) INFO 10-31 07:24:34 [monitor.py:62] [backbone] torch.compile took 24.27 s in total
(EngineCore pid=50398) INFO 10-31 07:26:00 [monitor.py:91] [backbone] Initial profiling/warmup run took 86.39 s

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   2%|▏         | 1/51 [00:01<00:51,  1.03s/it]
Capturing CUDA graphs (PIECEWISE):   4%|▍         | 2/51 [00:04<01:51,  2.28s/it]
Capturing CUDA graphs (PIECEWISE):   6%|▌         | 3/51 [00:04<01:09,  1.45s/it]
Capturing CUDA graphs (PIECEWISE):   8%|▊         | 4/51 [00:04<00:47,  1.01s/it]
Capturing CUDA graphs (PIECEWISE):  10%|▉         | 5/51 [00:05<00:35,  1.28it/s]
Capturing CUDA graphs (PIECEWISE):  12%|█▏        | 6/51 [00:05<00:29,  1.54it/s]
Capturing CUDA graphs (PIECEWISE):  14%|█▎        | 7/51 [00:06<00:27,  1.63it/s]
Capturing CUDA graphs (PIECEWISE):  16%|█▌        | 8/51 [00:06<00:24,  1.73it/s]
Capturing CUDA graphs (PIECEWISE):  18%|█▊        | 9/51 [00:09<00:57,  1.36s/it]
Capturing CUDA graphs (PIECEWISE):  20%|█▉        | 10/51 [00:10<00:42,  1.03s/it]
Capturing CUDA graphs (PIECEWISE):  22%|██▏       | 11/51 [00:10<00:32,  1.25it/s]
Capturing CUDA graphs (PIECEWISE):  24%|██▎       | 12/51 [00:10<00:25,  1.50it/s]
Capturing CUDA graphs (PIECEWISE):  25%|██▌       | 13/51 [00:11<00:21,  1.79it/s]
Capturing CUDA graphs (PIECEWISE):  27%|██▋       | 14/51 [00:11<00:17,  2.08it/s]
Capturing CUDA graphs (PIECEWISE):  29%|██▉       | 15/51 [00:11<00:14,  2.51it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 16/51 [00:11<00:12,  2.91it/s]
Capturing CUDA graphs (PIECEWISE):  33%|███▎      | 17/51 [00:12<00:18,  1.80it/s]
Capturing CUDA graphs (PIECEWISE):  35%|███▌      | 18/51 [00:13<00:15,  2.14it/s]
Capturing CUDA graphs (PIECEWISE):  37%|███▋      | 19/51 [00:13<00:11,  2.74it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 22/51 [00:13<00:05,  5.35it/s]
Capturing CUDA graphs (PIECEWISE):  49%|████▉     | 25/51 [00:13<00:03,  8.01it/s]
Capturing CUDA graphs (PIECEWISE):  53%|█████▎    | 27/51 [00:13<00:02,  8.74it/s]
Capturing CUDA graphs (PIECEWISE):  57%|█████▋    | 29/51 [00:13<00:02, 10.46it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 32/51 [00:13<00:01, 12.99it/s]
Capturing CUDA graphs (PIECEWISE):  67%|██████▋   | 34/51 [00:14<00:01, 14.12it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████   | 36/51 [00:14<00:01, 12.48it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▍  | 38/51 [00:14<00:00, 13.95it/s]
Capturing CUDA graphs (PIECEWISE):  80%|████████  | 41/51 [00:14<00:00, 15.84it/s]
Capturing CUDA graphs (PIECEWISE):  84%|████████▍ | 43/51 [00:14<00:00, 13.91it/s]
Capturing CUDA graphs (PIECEWISE):  88%|████████▊ | 45/51 [00:14<00:00, 14.89it/s]
Capturing CUDA graphs (PIECEWISE):  94%|█████████▍| 48/51 [00:14<00:00, 16.73it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:15<00:00, 13.48it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:15<00:00,  3.33it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/2 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.42it/s]
(EngineCore pid=50398) INFO 10-31 07:26:19 [model_runner.py:1127] Graph capturing finished in 16 secs, took 0.40 GiB
(EngineCore pid=50398) INFO 10-31 07:26:19 [gpu_worker.py:759] Available KV cache memory: 59.1 GiB
(EngineCore pid=50398) INFO 10-31 07:26:19 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8940 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9060. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:2327] GPU KV cache size: 1,741,452 tokens, Maximum concurrency for 32,768 tokens per request: 53.14x
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=50398) INFO 10-31 07:26:19 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=50398) INFO 10-31 07:26:19 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=50398) INFO 10-31 07:26:19 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=50398) INFO 10-31 07:26:21 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   2%|▏         | 1/51 [00:00<00:41,  1.22it/s]
Capturing CUDA graphs (PIECEWISE):   4%|▍         | 2/51 [00:01<00:40,  1.22it/s]
Capturing CUDA graphs (PIECEWISE):   6%|▌         | 3/51 [00:02<00:38,  1.24it/s]
Capturing CUDA graphs (PIECEWISE):   8%|▊         | 4/51 [00:03<00:36,  1.30it/s]
Capturing CUDA graphs (PIECEWISE):  10%|▉         | 5/51 [00:03<00:33,  1.38it/s]
Capturing CUDA graphs (PIECEWISE):  12%|█▏        | 6/51 [00:04<00:30,  1.46it/s]
Capturing CUDA graphs (PIECEWISE):  14%|█▎        | 7/51 [00:04<00:28,  1.56it/s]
Capturing CUDA graphs (PIECEWISE):  16%|█▌        | 8/51 [00:05<00:25,  1.68it/s]
Capturing CUDA graphs (PIECEWISE):  18%|█▊        | 9/51 [00:05<00:22,  1.83it/s]
Capturing CUDA graphs (PIECEWISE):  20%|█▉        | 10/51 [00:06<00:20,  2.00it/s]
Capturing CUDA graphs (PIECEWISE):  22%|██▏       | 11/51 [00:06<00:18,  2.21it/s]
Capturing CUDA graphs (PIECEWISE):  24%|██▎       | 12/51 [00:06<00:15,  2.49it/s]
Capturing CUDA graphs (PIECEWISE):  25%|██▌       | 13/51 [00:07<00:15,  2.48it/s]
Capturing CUDA graphs (PIECEWISE):  27%|██▋       | 14/51 [00:07<00:13,  2.78it/s]
Capturing CUDA graphs (PIECEWISE):  29%|██▉       | 15/51 [00:07<00:12,  2.93it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 16/51 [00:08<00:09,  3.54it/s]
Capturing CUDA graphs (PIECEWISE):  37%|███▋      | 19/51 [00:08<00:04,  6.69it/s]
Capturing CUDA graphs (PIECEWISE):  39%|███▉      | 20/51 [00:08<00:04,  6.91it/s]
Capturing CUDA graphs (PIECEWISE):  45%|████▌     | 23/51 [00:08<00:02, 10.29it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████     | 26/51 [00:08<00:01, 13.07it/s]
Capturing CUDA graphs (PIECEWISE):  55%|█████▍    | 28/51 [00:08<00:01, 13.16it/s]
Capturing CUDA graphs (PIECEWISE):  59%|█████▉    | 30/51 [00:08<00:01, 12.88it/s]
Capturing CUDA graphs (PIECEWISE):  65%|██████▍   | 33/51 [00:09<00:01, 15.13it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████   | 36/51 [00:09<00:00, 16.70it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▍  | 38/51 [00:09<00:00, 15.02it/s]
Capturing CUDA graphs (PIECEWISE):  80%|████████  | 41/51 [00:09<00:00, 16.86it/s]
Capturing CUDA graphs (PIECEWISE):  86%|████████▋ | 44/51 [00:09<00:00, 18.18it/s]
Capturing CUDA graphs (PIECEWISE):  90%|█████████ | 46/51 [00:09<00:00, 15.01it/s]
Capturing CUDA graphs (PIECEWISE):  96%|█████████▌| 49/51 [00:09<00:00, 16.88it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00,  5.08it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL):   9%|▊         | 3/35 [00:00<00:01, 28.10it/s]
Capturing CUDA graphs (FULL):  17%|█▋        | 6/35 [00:00<00:01, 21.27it/s]
Capturing CUDA graphs (FULL):  26%|██▌       | 9/35 [00:00<00:01, 23.77it/s]
Capturing CUDA graphs (FULL):  34%|███▍      | 12/35 [00:00<00:00, 25.13it/s]
Capturing CUDA graphs (FULL):  43%|████▎     | 15/35 [00:00<00:00, 25.78it/s]
Capturing CUDA graphs (FULL):  51%|█████▏    | 18/35 [00:00<00:00, 22.98it/s]
Capturing CUDA graphs (FULL):  60%|██████    | 21/35 [00:00<00:00, 24.42it/s]
Capturing CUDA graphs (FULL):  69%|██████▊   | 24/35 [00:00<00:00, 25.30it/s]
Capturing CUDA graphs (FULL):  77%|███████▋  | 27/35 [00:01<00:00, 26.23it/s]
Capturing CUDA graphs (FULL):  86%|████████▌ | 30/35 [00:01<00:00, 22.60it/s]
Capturing CUDA graphs (FULL):  94%|█████████▍| 33/35 [00:01<00:00, 23.28it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 24.13it/s]
(EngineCore pid=50398) INFO 10-31 07:26:52 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.24 GiB
(EngineCore pid=50398) INFO 10-31 07:26:52 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%).
(EngineCore pid=50398) INFO 10-31 07:26:52 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 1.71 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63044161434` (58.71 GiB) to fit into requested memory, or `--kv-cache-memory=71110227456` (66.23 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.1 GiB.
(EngineCore pid=50398) INFO 10-31 07:26:52 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=50398) INFO 10-31 07:26:53 [core.py:395] init engine (profile, create kv cache, warmup model) took 164.06 s (compilation: 24.27 s)
(EngineCore pid=50398) INFO 10-31 07:26:53 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=50120) WARNING 10-31 07:26:54 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=50398) WARNING 10-31 07:26:54 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout joint: 5957 items in 113s
(EngineCore pid=50398) WARNING 10-31 07:28:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=50398) WARNING 10-31 07:28:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=50398) WARNING 10-31 07:28:49 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
[rank0]:[W1031 07:29:03.201210575 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 245366784 bytes (free: 205389824, total: 85094825984).
progress {"time": "07:29:46", "elapsed_min": 1.0, "submitted": 323, "finished": 67, "in_flight": 256, "gen_tokens": 46544, "tok_per_s": 776, "chain_tokens_median": 708.0, "chain_tokens_p90": 869.4, "chain_tokens_max": 951, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:29:46 [loggers.py:335] Engine 000: Avg prompt throughput: 17008.8 tokens/s, Avg generation throughput: 1397.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.1%, Prefix cache hit rate: 2.0%
progress {"time": "07:30:46", "elapsed_min": 2.0, "submitted": 411, "finished": 155, "in_flight": 256, "gen_tokens": 152883, "tok_per_s": 1274, "chain_tokens_median": 870.0, "chain_tokens_p90": 1637.3999999999999, "chain_tokens_max": 1944, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:30:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3354.1 tokens/s, Avg generation throughput: 4326.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.1%, Prefix cache hit rate: 5.1%
progress {"time": "07:31:46", "elapsed_min": 3.0, "submitted": 489, "finished": 233, "in_flight": 256, "gen_tokens": 279612, "tok_per_s": 1553, "chain_tokens_median": 954.0, "chain_tokens_p90": 2206.2000000000007, "chain_tokens_max": 2819, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:31:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3902.0 tokens/s, Avg generation throughput: 3763.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.8%, Prefix cache hit rate: 6.7%
progress {"time": "07:32:46", "elapsed_min": 4.0, "submitted": 589, "finished": 333, "in_flight": 256, "gen_tokens": 417903, "tok_per_s": 1741, "chain_tokens_median": 914.0, "chain_tokens_p90": 2622.2000000000003, "chain_tokens_max": 3626, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:32:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3775.4 tokens/s, Avg generation throughput: 3535.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 7.6%
progress {"time": "07:33:46", "elapsed_min": 5.0, "submitted": 679, "finished": 423, "in_flight": 256, "gen_tokens": 570512, "tok_per_s": 1902, "chain_tokens_median": 911.0, "chain_tokens_p90": 2939.0, "chain_tokens_max": 4448, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:33:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3862.0 tokens/s, Avg generation throughput: 3374.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.8%, Prefix cache hit rate: 7.0%
progress {"time": "07:34:46", "elapsed_min": 6.0, "submitted": 758, "finished": 502, "in_flight": 256, "gen_tokens": 744444, "tok_per_s": 2068, "chain_tokens_median": 931.5, "chain_tokens_p90": 3435.2000000000007, "chain_tokens_max": 5176, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:34:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3989.0 tokens/s, Avg generation throughput: 3302.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.3%, Prefix cache hit rate: 7.2%
progress {"time": "07:35:46", "elapsed_min": 7.0, "submitted": 833, "finished": 577, "in_flight": 256, "gen_tokens": 872988, "tok_per_s": 2078, "chain_tokens_median": 930.0, "chain_tokens_p90": 3548.2, "chain_tokens_max": 5974, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:35:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2934.6 tokens/s, Avg generation throughput: 3631.9 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.8%, Prefix cache hit rate: 6.1%
progress {"time": "07:36:46", "elapsed_min": 8.0, "submitted": 911, "finished": 655, "in_flight": 256, "gen_tokens": 1041235, "tok_per_s": 2169, "chain_tokens_median": 951.0, "chain_tokens_p90": 3695.8, "chain_tokens_max": 6791, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:36:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3802.0 tokens/s, Avg generation throughput: 3208.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.6%, Prefix cache hit rate: 4.3%
progress {"time": "07:37:46", "elapsed_min": 9.0, "submitted": 987, "finished": 731, "in_flight": 256, "gen_tokens": 1236325, "tok_per_s": 2289, "chain_tokens_median": 974.0, "chain_tokens_p90": 3969.0, "chain_tokens_max": 7508, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:37:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4333.7 tokens/s, Avg generation throughput: 2995.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.9%, Prefix cache hit rate: 3.0%
progress {"time": "07:38:46", "elapsed_min": 10.0, "submitted": 1060, "finished": 804, "in_flight": 256, "gen_tokens": 1432867, "tok_per_s": 2388, "chain_tokens_median": 1014.0, "chain_tokens_p90": 4231.600000000002, "chain_tokens_max": 7982, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:38:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4310.3 tokens/s, Avg generation throughput: 3084.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.9%, Prefix cache hit rate: 1.8%
progress {"time": "07:39:46", "elapsed_min": 11.0, "submitted": 1132, "finished": 876, "in_flight": 256, "gen_tokens": 1601814, "tok_per_s": 2427, "chain_tokens_median": 1005.5, "chain_tokens_p90": 4388.5, "chain_tokens_max": 8959, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:39:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3864.1 tokens/s, Avg generation throughput: 3055.0 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.3%, Prefix cache hit rate: 1.2%
progress {"time": "07:40:46", "elapsed_min": 12.0, "submitted": 1212, "finished": 956, "in_flight": 256, "gen_tokens": 1782507, "tok_per_s": 2476, "chain_tokens_median": 996.0, "chain_tokens_p90": 4443.0, "chain_tokens_max": 9298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:40:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4023.8 tokens/s, Avg generation throughput: 3053.9 tokens/s, Running: 253 reqs, Waiting: 2 reqs, GPU KV cache usage: 67.0%, Prefix cache hit rate: 1.1%
progress {"time": "07:41:46", "elapsed_min": 13.0, "submitted": 1313, "finished": 1057, "in_flight": 256, "gen_tokens": 1951521, "tok_per_s": 2502, "chain_tokens_median": 949.0, "chain_tokens_p90": 4460.6, "chain_tokens_max": 9845, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:41:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3915.1 tokens/s, Avg generation throughput: 2993.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.5%
progress {"time": "07:42:46", "elapsed_min": 14.0, "submitted": 1418, "finished": 1162, "in_flight": 256, "gen_tokens": 2124699, "tok_per_s": 2529, "chain_tokens_median": 912.5, "chain_tokens_p90": 4455.200000000001, "chain_tokens_max": 10721, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:42:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4457.4 tokens/s, Avg generation throughput: 2746.6 tokens/s, Running: 255 reqs, Waiting: 1 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.4%
progress {"time": "07:43:46", "elapsed_min": 15.0, "submitted": 1496, "finished": 1240, "in_flight": 256, "gen_tokens": 2244199, "tok_per_s": 2493, "chain_tokens_median": 901.0, "chain_tokens_p90": 4448.800000000001, "chain_tokens_max": 10721, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:43:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3324.6 tokens/s, Avg generation throughput: 3208.5 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.6%, Prefix cache hit rate: 0.6%
progress {"time": "07:44:46", "elapsed_min": 16.0, "submitted": 1569, "finished": 1313, "in_flight": 256, "gen_tokens": 2405334, "tok_per_s": 2505, "chain_tokens_median": 918.0, "chain_tokens_p90": 4492.5999999999985, "chain_tokens_max": 11957, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:44:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3674.2 tokens/s, Avg generation throughput: 3166.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.6%, Prefix cache hit rate: 0.8%
progress {"time": "07:45:46", "elapsed_min": 17.0, "submitted": 1670, "finished": 1414, "in_flight": 256, "gen_tokens": 2593630, "tok_per_s": 2543, "chain_tokens_median": 917.5, "chain_tokens_p90": 4573.4, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:45:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4271.7 tokens/s, Avg generation throughput: 2753.6 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.8%
progress {"time": "07:46:46", "elapsed_min": 18.0, "submitted": 1760, "finished": 1504, "in_flight": 256, "gen_tokens": 2757289, "tok_per_s": 2553, "chain_tokens_median": 915.0, "chain_tokens_p90": 4573.4, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:46:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3646.6 tokens/s, Avg generation throughput: 3079.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.8%, Prefix cache hit rate: 0.8%
progress {"time": "07:47:46", "elapsed_min": 19.0, "submitted": 1828, "finished": 1572, "in_flight": 256, "gen_tokens": 2914911, "tok_per_s": 2557, "chain_tokens_median": 936.5, "chain_tokens_p90": 4575.8, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:47:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3826.4 tokens/s, Avg generation throughput: 2974.2 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.9%
progress {"time": "07:48:46", "elapsed_min": 20.0, "submitted": 1895, "finished": 1639, "in_flight": 256, "gen_tokens": 3053029, "tok_per_s": 2544, "chain_tokens_median": 939.0, "chain_tokens_p90": 4585.800000000001, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:48:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3096.4 tokens/s, Avg generation throughput: 3205.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.9%, Prefix cache hit rate: 1.2%
progress {"time": "07:49:46", "elapsed_min": 21.0, "submitted": 1973, "finished": 1717, "in_flight": 256, "gen_tokens": 3159228, "tok_per_s": 2507, "chain_tokens_median": 923.0, "chain_tokens_p90": 4572.8, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:49:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2989.7 tokens/s, Avg generation throughput: 2867.7 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.4%, Prefix cache hit rate: 1.3%
progress {"time": "07:50:46", "elapsed_min": 22.0, "submitted": 2047, "finished": 1791, "in_flight": 256, "gen_tokens": 3311368, "tok_per_s": 2509, "chain_tokens_median": 917.0, "chain_tokens_p90": 4576.0, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:50:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4242.5 tokens/s, Avg generation throughput: 2534.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.8%, Prefix cache hit rate: 0.8%
progress {"time": "07:51:46", "elapsed_min": 23.0, "submitted": 2098, "finished": 1842, "in_flight": 256, "gen_tokens": 3480011, "tok_per_s": 2522, "chain_tokens_median": 941.5, "chain_tokens_p90": 4612.300000000001, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:51:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3543.7 tokens/s, Avg generation throughput: 2958.7 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.2%, Prefix cache hit rate: 0.6%
progress {"time": "07:52:46", "elapsed_min": 24.0, "submitted": 2160, "finished": 1904, "in_flight": 256, "gen_tokens": 3607304, "tok_per_s": 2505, "chain_tokens_median": 939.5, "chain_tokens_p90": 4608.900000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:52:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2999.3 tokens/s, Avg generation throughput: 2779.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.3%, Prefix cache hit rate: 0.7%
progress {"time": "07:53:46", "elapsed_min": 25.0, "submitted": 2223, "finished": 1967, "in_flight": 256, "gen_tokens": 3742421, "tok_per_s": 2495, "chain_tokens_median": 941.0, "chain_tokens_p90": 4620.400000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:53:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3278.4 tokens/s, Avg generation throughput: 2804.8 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 0.6%
progress {"time": "07:54:46", "elapsed_min": 26.0, "submitted": 2289, "finished": 2033, "in_flight": 256, "gen_tokens": 3895368, "tok_per_s": 2497, "chain_tokens_median": 952.0, "chain_tokens_p90": 4633.0, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:54:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3650.6 tokens/s, Avg generation throughput: 2552.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 0.6%
progress {"time": "07:55:46", "elapsed_min": 27.0, "submitted": 2365, "finished": 2109, "in_flight": 256, "gen_tokens": 4016987, "tok_per_s": 2480, "chain_tokens_median": 942.0, "chain_tokens_p90": 4625.8, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:55:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3179.1 tokens/s, Avg generation throughput: 2658.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 0.2%
progress {"time": "07:56:46", "elapsed_min": 28.0, "submitted": 2435, "finished": 2179, "in_flight": 256, "gen_tokens": 4131815, "tok_per_s": 2459, "chain_tokens_median": 941.0, "chain_tokens_p90": 4619.200000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:56:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2816.6 tokens/s, Avg generation throughput: 2821.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 0.2%
progress {"time": "07:57:46", "elapsed_min": 29.0, "submitted": 2512, "finished": 2256, "in_flight": 256, "gen_tokens": 4240220, "tok_per_s": 2437, "chain_tokens_median": 936.5, "chain_tokens_p90": 4579.5, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:57:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2949.3 tokens/s, Avg generation throughput: 2650.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.0%, Prefix cache hit rate: 0.0%
progress {"time": "07:58:46", "elapsed_min": 30.0, "submitted": 2569, "finished": 2313, "in_flight": 256, "gen_tokens": 4341840, "tok_per_s": 2412, "chain_tokens_median": 939.0, "chain_tokens_p90": 4617.200000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:58:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2860.4 tokens/s, Avg generation throughput: 2557.7 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.0%, Prefix cache hit rate: 0.0%
progress {"time": "07:59:46", "elapsed_min": 31.0, "submitted": 2611, "finished": 2355, "in_flight": 256, "gen_tokens": 4415281, "tok_per_s": 2374, "chain_tokens_median": 941.0, "chain_tokens_p90": 4591.399999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:59:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1945.8 tokens/s, Avg generation throughput: 2746.8 tokens/s, Running: 250 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:00:46", "elapsed_min": 32.0, "submitted": 2616, "finished": 2360, "in_flight": 256, "gen_tokens": 4436745, "tok_per_s": 2311, "chain_tokens_median": 941.0, "chain_tokens_p90": 4618.599999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:00:46 [loggers.py:335] Engine 000: Avg prompt throughput: 442.3 tokens/s, Avg generation throughput: 2340.3 tokens/s, Running: 220 reqs, Waiting: 35 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:01:46", "elapsed_min": 33.0, "submitted": 2656, "finished": 2400, "in_flight": 256, "gen_tokens": 4517481, "tok_per_s": 2281, "chain_tokens_median": 950.5, "chain_tokens_p90": 4618.599999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:01:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1925.8 tokens/s, Avg generation throughput: 2034.0 tokens/s, Running: 205 reqs, Waiting: 51 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:02:46", "elapsed_min": 34.0, "submitted": 2683, "finished": 2427, "in_flight": 256, "gen_tokens": 4610168, "tok_per_s": 2260, "chain_tokens_median": 956.0, "chain_tokens_p90": 4644.6, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:02:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1923.7 tokens/s, Avg generation throughput: 1885.7 tokens/s, Running: 200 reqs, Waiting: 54 reqs, GPU KV cache usage: 98.0%, Prefix cache hit rate: 0.0%
progress {"time": "08:03:46", "elapsed_min": 35.0, "submitted": 2713, "finished": 2457, "in_flight": 256, "gen_tokens": 4704725, "tok_per_s": 2240, "chain_tokens_median": 970.0, "chain_tokens_p90": 4684.800000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:03:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2226.1 tokens/s, Avg generation throughput: 1887.7 tokens/s, Running: 200 reqs, Waiting: 56 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:04:46", "elapsed_min": 36.0, "submitted": 2715, "finished": 2459, "in_flight": 256, "gen_tokens": 4717543, "tok_per_s": 2184, "chain_tokens_median": 973.0, "chain_tokens_p90": 4728.600000000005, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:04:46 [loggers.py:335] Engine 000: Avg prompt throughput: 350.0 tokens/s, Avg generation throughput: 2303.6 tokens/s, Running: 183 reqs, Waiting: 73 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0%
progress {"time": "08:05:46", "elapsed_min": 37.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 4720605, "tok_per_s": 2126, "chain_tokens_median": 973.0, "chain_tokens_p90": 4725.0, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:05:46 [loggers.py:335] Engine 000: Avg prompt throughput: 87.2 tokens/s, Avg generation throughput: 2105.9 tokens/s, Running: 170 reqs, Waiting: 86 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:06:46", "elapsed_min": 38.0, "submitted": 2736, "finished": 2480, "in_flight": 256, "gen_tokens": 4779993, "tok_per_s": 2096, "chain_tokens_median": 976.5, "chain_tokens_p90": 4755.5, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:06:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1208.2 tokens/s, Avg generation throughput: 1816.0 tokens/s, Running: 153 reqs, Waiting: 103 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 0.0%
progress {"time": "08:07:46", "elapsed_min": 39.0, "submitted": 2819, "finished": 2563, "in_flight": 256, "gen_tokens": 5068653, "tok_per_s": 2166, "chain_tokens_median": 1004.0, "chain_tokens_p90": 4854.800000000003, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:07:46 [loggers.py:335] Engine 000: Avg prompt throughput: 5749.3 tokens/s, Avg generation throughput: 1155.6 tokens/s, Running: 171 reqs, Waiting: 85 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:08:46", "elapsed_min": 40.0, "submitted": 2819, "finished": 2563, "in_flight": 256, "gen_tokens": 5068653, "tok_per_s": 2112, "chain_tokens_median": 1004.0, "chain_tokens_p90": 4854.800000000003, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:08:46 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1909.1 tokens/s, Running: 140 reqs, Waiting: 116 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0%
(MainProcess pid=52652) INFO 10-31 08:10:04 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=52652) INFO 10-31 08:10:04 [model.py:2152] Using max model len 32768
(MainProcess pid=52652) INFO 10-31 08:10:04 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=52652) INFO 10-31 08:10:04 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=52652) INFO 10-31 08:10:04 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=52921) INFO 10-31 08:10:10 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=52921) INFO 10-31 08:10:11 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_26d27b6bcf31426282aa10a426927f05 backend=nccl
(EngineCore pid=52921) INFO 10-31 08:10:11 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=52921) INFO 10-31 08:10:11 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=52921) INFO 10-31 08:10:13 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=52921) INFO 10-31 08:10:13 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=52921) INFO 10-31 08:10:14 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=52921) INFO 10-31 08:10:14 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=52921) INFO 10-31 08:10:17 [base_loader.py:82] Initializing model took 4.067586 seconds
(EngineCore pid=52921) INFO 10-31 08:10:17 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 163.58 GiB.
(EngineCore pid=52921) INFO 10-31 08:10:17 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.

Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.05it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.05it/s]
(EngineCore pid=52921) INFO 10-31 08:10:18 [default_loader.py:501] Loading weights took 0.98 seconds
(EngineCore pid=52921) INFO 10-31 08:10:19 [model_runner.py:424] Model loading took 7.94 GiB memory and 6.792969 seconds
(EngineCore pid=52921) INFO 10-31 08:10:20 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=52921) INFO 10-31 08:10:20 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=52921) INFO 10-31 08:10:20 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=52921) INFO 10-31 08:10:20 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=52921) INFO 10-31 08:10:20 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=52921) INFO 10-31 08:10:21 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=52921) INFO 10-31 08:10:27 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/764a5d377e/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=52921) INFO 10-31 08:10:27 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.29 s
(EngineCore pid=52921) INFO 10-31 08:10:43 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 16.27 s
(EngineCore pid=52921) INFO 10-31 08:10:46 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 26071923 bytes total
(EngineCore pid=52921) INFO 10-31 08:10:46 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/104e4206a2f7547583b1741445864a147db2dd2fea66f017ae75eae0a52768df/rank_0_0/model
(EngineCore pid=52921) INFO 10-31 08:10:46 [monitor.py:62] [backbone] torch.compile took 24.57 s in total
(EngineCore pid=52921) INFO 10-31 08:12:13 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.29 s

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   3%|▎         | 1/35 [00:00<00:32,  1.06it/s]
Capturing CUDA graphs (PIECEWISE):   6%|▌         | 2/35 [00:01<00:25,  1.29it/s]
Capturing CUDA graphs (PIECEWISE):   9%|▊         | 3/35 [00:01<00:18,  1.72it/s]
Capturing CUDA graphs (PIECEWISE):  11%|█▏        | 4/35 [00:02<00:13,  2.22it/s]
Capturing CUDA graphs (PIECEWISE):  14%|█▍        | 5/35 [00:02<00:12,  2.42it/s]
Capturing CUDA graphs (PIECEWISE):  17%|█▋        | 6/35 [00:02<00:10,  2.87it/s]
Capturing CUDA graphs (PIECEWISE):  20%|██        | 7/35 [00:03<00:09,  2.93it/s]
Capturing CUDA graphs (PIECEWISE):  23%|██▎       | 8/35 [00:03<00:08,  3.25it/s]
Capturing CUDA graphs (PIECEWISE):  26%|██▌       | 9/35 [00:03<00:08,  3.03it/s]
Capturing CUDA graphs (PIECEWISE):  29%|██▊       | 10/35 [00:04<00:08,  2.88it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 11/35 [00:04<00:07,  3.20it/s]
Capturing CUDA graphs (PIECEWISE):  34%|███▍      | 12/35 [00:04<00:07,  3.24it/s]
Capturing CUDA graphs (PIECEWISE):  37%|███▋      | 13/35 [00:04<00:05,  3.72it/s]
Capturing CUDA graphs (PIECEWISE):  40%|████      | 14/35 [00:05<00:05,  3.72it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 15/35 [00:05<00:04,  4.40it/s]
Capturing CUDA graphs (PIECEWISE):  46%|████▌     | 16/35 [00:05<00:03,  5.09it/s]
Capturing CUDA graphs (PIECEWISE):  49%|████▊     | 17/35 [00:06<00:08,  2.17it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████▏    | 18/35 [00:06<00:06,  2.47it/s]
Capturing CUDA graphs (PIECEWISE):  57%|█████▋    | 20/35 [00:09<00:12,  1.23it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 22/35 [00:09<00:06,  1.95it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████▏  | 25/35 [00:09<00:03,  3.23it/s]
Capturing CUDA graphs (PIECEWISE):  74%|███████▍  | 26/35 [00:12<00:06,  1.40it/s]
Capturing CUDA graphs (PIECEWISE):  80%|████████  | 28/35 [00:12<00:03,  2.03it/s]
Capturing CUDA graphs (PIECEWISE):  89%|████████▊ | 31/35 [00:12<00:01,  3.28it/s]
Capturing CUDA graphs (PIECEWISE):  94%|█████████▍| 33/35 [00:12<00:00,  4.30it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:12<00:00,  4.91it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:12<00:00,  2.74it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/2 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.79it/s]
(EngineCore pid=52921) INFO 10-31 08:12:29 [model_runner.py:1127] Graph capturing finished in 14 secs, took 0.30 GiB
(EngineCore pid=52921) INFO 10-31 08:12:30 [gpu_worker.py:759] Available KV cache memory: 59.37 GiB
(EngineCore pid=52921) INFO 10-31 08:12:30 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:2327] GPU KV cache size: 1,749,526 tokens, Maximum concurrency for 32,768 tokens per request: 53.39x
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=52921) INFO 10-31 08:12:30 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=52921) INFO 10-31 08:12:30 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=52921) INFO 10-31 08:12:30 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=52921) INFO 10-31 08:12:31 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   3%|▎         | 1/35 [00:00<00:15,  2.15it/s]
Capturing CUDA graphs (PIECEWISE):   6%|▌         | 2/35 [00:00<00:11,  2.77it/s]
Capturing CUDA graphs (PIECEWISE):   9%|▊         | 3/35 [00:01<00:12,  2.64it/s]
Capturing CUDA graphs (PIECEWISE):  11%|█▏        | 4/35 [00:01<00:11,  2.63it/s]
Capturing CUDA graphs (PIECEWISE):  14%|█▍        | 5/35 [00:01<00:10,  2.88it/s]
Capturing CUDA graphs (PIECEWISE):  17%|█▋        | 6/35 [00:02<00:10,  2.68it/s]
Capturing CUDA graphs (PIECEWISE):  20%|██        | 7/35 [00:02<00:09,  3.10it/s]
Capturing CUDA graphs (PIECEWISE):  23%|██▎       | 8/35 [00:02<00:09,  2.84it/s]
Capturing CUDA graphs (PIECEWISE):  26%|██▌       | 9/35 [00:03<00:08,  3.05it/s]
Capturing CUDA graphs (PIECEWISE):  29%|██▊       | 10/35 [00:03<00:08,  2.79it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 11/35 [00:03<00:07,  3.11it/s]
Capturing CUDA graphs (PIECEWISE):  34%|███▍      | 12/35 [00:04<00:07,  2.96it/s]
Capturing CUDA graphs (PIECEWISE):  37%|███▋      | 13/35 [00:04<00:06,  3.45it/s]
Capturing CUDA graphs (PIECEWISE):  40%|████      | 14/35 [00:04<00:06,  3.19it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 15/35 [00:04<00:05,  3.85it/s]
Capturing CUDA graphs (PIECEWISE):  46%|████▌     | 16/35 [00:04<00:04,  4.59it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████▏    | 18/35 [00:05<00:02,  5.89it/s]
Capturing CUDA graphs (PIECEWISE):  57%|█████▋    | 20/35 [00:05<00:01,  7.54it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 22/35 [00:05<00:01,  9.38it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████▏  | 25/35 [00:05<00:00, 12.52it/s]
Capturing CUDA graphs (PIECEWISE):  77%|███████▋  | 27/35 [00:05<00:00, 10.92it/s]
Capturing CUDA graphs (PIECEWISE):  83%|████████▎ | 29/35 [00:06<00:00, 11.58it/s]
Capturing CUDA graphs (PIECEWISE):  91%|█████████▏| 32/35 [00:06<00:00, 14.27it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 16.33it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00,  5.56it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL):  11%|█         | 2/19 [00:00<00:01, 13.02it/s]
Capturing CUDA graphs (FULL):  26%|██▋       | 5/19 [00:00<00:00, 18.88it/s]
Capturing CUDA graphs (FULL):  42%|████▏     | 8/19 [00:00<00:00, 22.31it/s]
Capturing CUDA graphs (FULL):  58%|█████▊    | 11/19 [00:00<00:00, 24.28it/s]
Capturing CUDA graphs (FULL):  74%|███████▎  | 14/19 [00:00<00:00, 25.43it/s]
Capturing CUDA graphs (FULL):  89%|████████▉ | 17/19 [00:00<00:00, 21.59it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.78it/s]
(EngineCore pid=52921) INFO 10-31 08:12:58 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.16 GiB
(EngineCore pid=52921) INFO 10-31 08:12:58 [gpu_worker.py:971] CUDA graph pool memory: 0.16 GiB (actual), 0.32 GiB (estimated), difference: 0.16 GiB (105.0%).
(EngineCore pid=52921) INFO 10-31 08:12:58 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.39 GiB for consumed memory (weights + non-torch), 1.56 GiB for peak activation, and 0.16 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63427942810` (59.07 GiB) to fit into requested memory, or `--kv-cache-memory=71494008832` (66.58 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.37 GiB.
(EngineCore pid=52921) INFO 10-31 08:12:58 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=52921) INFO 10-31 08:12:59 [core.py:395] init engine (profile, create kv cache, warmup model) took 158.78 s (compilation: 24.57 s)
(EngineCore pid=52921) INFO 10-31 08:12:59 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
readout joint: exists
(MainProcess pid=52652) WARNING 10-31 08:13:00 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=52921) WARNING 10-31 08:13:01 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=52921) WARNING 10-31 08:13:02 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=52921) WARNING 10-31 08:13:03 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "08:13:59", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:13:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1678.8 tokens/s, Avg generation throughput: 4172.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:14:59", "elapsed_min": 2.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:14:59 [loggers.py:335] Engine 000: Avg prompt throughput: 248.1 tokens/s, Avg generation throughput: 4617.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 38.0%, Prefix cache hit rate: 0.0%
(EngineCore pid=52921) WARNING 10-31 08:15:44 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "08:15:59", "elapsed_min": 3.0, "submitted": 343, "finished": 87, "in_flight": 256, "gen_tokens": 165975, "tok_per_s": 922, "chain_tokens_median": 1378.0, "chain_tokens_p90": 4092.0, "chain_tokens_max": 4697, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:15:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3332.2 tokens/s, Avg generation throughput: 3318.0 tokens/s, Running: 121 reqs, Waiting: 132 reqs, GPU KV cache usage: 40.1%, Prefix cache hit rate: 4.4%
progress {"time": "08:16:59", "elapsed_min": 4.0, "submitted": 394, "finished": 138, "in_flight": 256, "gen_tokens": 263082, "tok_per_s": 1096, "chain_tokens_median": 1068.5, "chain_tokens_p90": 4408.0, "chain_tokens_max": 5727, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:16:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2247.1 tokens/s, Avg generation throughput: 3514.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 41.4%, Prefix cache hit rate: 4.3%
progress {"time": "08:17:59", "elapsed_min": 5.0, "submitted": 538, "finished": 282, "in_flight": 256, "gen_tokens": 468930, "tok_per_s": 1563, "chain_tokens_median": 805.0, "chain_tokens_p90": 4581.500000000001, "chain_tokens_max": 7348, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:17:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4654.2 tokens/s, Avg generation throughput: 2732.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 39.3%, Prefix cache hit rate: 3.1%
progress {"time": "08:18:59", "elapsed_min": 6.0, "submitted": 668, "finished": 412, "in_flight": 256, "gen_tokens": 688748, "tok_per_s": 1913, "chain_tokens_median": 706.0, "chain_tokens_p90": 5116.300000000001, "chain_tokens_max": 8800, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:18:59 [loggers.py:335] Engine 000: Avg prompt throughput: 5184.1 tokens/s, Avg generation throughput: 2707.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 38.5%, Prefix cache hit rate: 2.5%
progress {"time": "08:19:59", "elapsed_min": 7.0, "submitted": 786, "finished": 530, "in_flight": 256, "gen_tokens": 894546, "tok_per_s": 2130, "chain_tokens_median": 727.5, "chain_tokens_p90": 5277.500000000001, "chain_tokens_max": 10487, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:19:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4759.2 tokens/s, Avg generation throughput: 2671.5 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 41.0%, Prefix cache hit rate: 2.4%
progress {"time": "08:20:59", "elapsed_min": 8.0, "submitted": 804, "finished": 548, "in_flight": 256, "gen_tokens": 927713, "tok_per_s": 1933, "chain_tokens_median": 738.5, "chain_tokens_p90": 5166.500000000002, "chain_tokens_max": 10849, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:20:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1168.9 tokens/s, Avg generation throughput: 3686.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 40.8%, Prefix cache hit rate: 1.6%
progress {"time": "08:21:59", "elapsed_min": 9.0, "submitted": 913, "finished": 657, "in_flight": 256, "gen_tokens": 1186905, "tok_per_s": 2198, "chain_tokens_median": 827.0, "chain_tokens_p90": 4835.999999999998, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:21:59 [loggers.py:335] Engine 000: Avg prompt throughput: 5622.2 tokens/s, Avg generation throughput: 2262.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 43.1%, Prefix cache hit rate: 1.2%
progress {"time": "08:22:59", "elapsed_min": 10.0, "submitted": 921, "finished": 665, "in_flight": 256, "gen_tokens": 1197698, "tok_per_s": 1996, "chain_tokens_median": 834.0, "chain_tokens_p90": 4762.400000000002, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:22:59 [loggers.py:335] Engine 000: Avg prompt throughput: 477.5 tokens/s, Avg generation throughput: 3661.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 48.1%, Prefix cache hit rate: 1.1%
progress {"time": "08:23:59", "elapsed_min": 11.0, "submitted": 932, "finished": 676, "in_flight": 256, "gen_tokens": 1223701, "tok_per_s": 1854, "chain_tokens_median": 849.0, "chain_tokens_p90": 4796.5, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:23:59 [loggers.py:335] Engine 000: Avg prompt throughput: 717.7 tokens/s, Avg generation throughput: 3409.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 0.9%
progress {"time": "08:24:59", "elapsed_min": 12.0, "submitted": 961, "finished": 705, "in_flight": 256, "gen_tokens": 1306087, "tok_per_s": 1814, "chain_tokens_median": 891.0, "chain_tokens_p90": 4751.000000000002, "chain_tokens_max": 14338, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:24:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1795.3 tokens/s, Avg generation throughput: 2950.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.1%, Prefix cache hit rate: 0.9%
progress {"time": "08:25:59", "elapsed_min": 13.0, "submitted": 1042, "finished": 786, "in_flight": 256, "gen_tokens": 1652094, "tok_per_s": 2118, "chain_tokens_median": 989.0, "chain_tokens_p90": 5533.5, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:25:59 [loggers.py:335] Engine 000: Avg prompt throughput: 6602.1 tokens/s, Avg generation throughput: 1718.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.6%, Prefix cache hit rate: 0.7%
progress {"time": "08:26:59", "elapsed_min": 14.0, "submitted": 1051, "finished": 795, "in_flight": 256, "gen_tokens": 1699022, "tok_per_s": 2023, "chain_tokens_median": 1003.0, "chain_tokens_p90": 5605.600000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:26:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1079.1 tokens/s, Avg generation throughput: 3151.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.8%, Prefix cache hit rate: 0.7%
progress {"time": "08:27:59", "elapsed_min": 15.0, "submitted": 1062, "finished": 806, "in_flight": 256, "gen_tokens": 1734071, "tok_per_s": 1927, "chain_tokens_median": 1026.5, "chain_tokens_p90": 5570.0, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:27:59 [loggers.py:335] Engine 000: Avg prompt throughput: 780.9 tokens/s, Avg generation throughput: 3200.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 55.8%, Prefix cache hit rate: 0.6%
progress {"time": "08:28:59", "elapsed_min": 16.0, "submitted": 1117, "finished": 861, "in_flight": 256, "gen_tokens": 1941549, "tok_per_s": 2022, "chain_tokens_median": 1098.0, "chain_tokens_p90": 5793.0, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:28:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3922.4 tokens/s, Avg generation throughput: 2171.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.4%, Prefix cache hit rate: 0.6%
progress {"time": "08:29:59", "elapsed_min": 17.0, "submitted": 1171, "finished": 915, "in_flight": 256, "gen_tokens": 2137071, "tok_per_s": 2095, "chain_tokens_median": 1141.0, "chain_tokens_p90": 5915.200000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:29:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3689.8 tokens/s, Avg generation throughput: 2212.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.7%, Prefix cache hit rate: 0.5%
progress {"time": "08:30:59", "elapsed_min": 18.0, "submitted": 1180, "finished": 924, "in_flight": 256, "gen_tokens": 2171831, "tok_per_s": 2011, "chain_tokens_median": 1153.0, "chain_tokens_p90": 5916.900000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:30:59 [loggers.py:335] Engine 000: Avg prompt throughput: 846.9 tokens/s, Avg generation throughput: 2884.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.5%, Prefix cache hit rate: 0.5%
progress {"time": "08:31:59", "elapsed_min": 19.0, "submitted": 1192, "finished": 936, "in_flight": 256, "gen_tokens": 2198587, "tok_per_s": 1929, "chain_tokens_median": 1156.0, "chain_tokens_p90": 5913.5, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:31:59 [loggers.py:335] Engine 000: Avg prompt throughput: 658.8 tokens/s, Avg generation throughput: 2936.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 62.6%, Prefix cache hit rate: 0.5%
progress {"time": "08:32:59", "elapsed_min": 20.0, "submitted": 1233, "finished": 977, "in_flight": 256, "gen_tokens": 2342154, "tok_per_s": 1952, "chain_tokens_median": 1192.0, "chain_tokens_p90": 5911.799999999999, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:32:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2665.8 tokens/s, Avg generation throughput: 2277.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 65.8%, Prefix cache hit rate: 0.4%
progress {"time": "08:33:59", "elapsed_min": 21.0, "submitted": 1300, "finished": 1044, "in_flight": 256, "gen_tokens": 2601133, "tok_per_s": 2064, "chain_tokens_median": 1291.5, "chain_tokens_p90": 6045.7, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:33:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4735.4 tokens/s, Avg generation throughput: 1738.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.3%
progress {"time": "08:34:59", "elapsed_min": 22.0, "submitted": 1311, "finished": 1055, "in_flight": 256, "gen_tokens": 2638613, "tok_per_s": 1999, "chain_tokens_median": 1308.0, "chain_tokens_p90": 6045.6, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:34:59 [loggers.py:335] Engine 000: Avg prompt throughput: 838.6 tokens/s, Avg generation throughput: 2635.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 68.9%, Prefix cache hit rate: 0.2%
progress {"time": "08:35:59", "elapsed_min": 23.0, "submitted": 1325, "finished": 1069, "in_flight": 256, "gen_tokens": 2851339, "tok_per_s": 2066, "chain_tokens_median": 1340.0, "chain_tokens_p90": 6204.6, "chain_tokens_max": 31461, "parse_rate": 0.9943872778297475, "looped": 6, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:35:59 [loggers.py:335] Engine 000: Avg prompt throughput: 628.1 tokens/s, Avg generation throughput: 2738.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 62.5%, Prefix cache hit rate: 0.2%
progress {"time": "08:36:59", "elapsed_min": 24.0, "submitted": 1409, "finished": 1153, "in_flight": 256, "gen_tokens": 3526194, "tok_per_s": 2449, "chain_tokens_median": 1465.0, "chain_tokens_p90": 6371.799999999999, "chain_tokens_max": 32532, "parse_rate": 0.9826539462272333, "looped": 20, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:36:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4263.2 tokens/s, Avg generation throughput: 2219.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 40.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:37:59", "elapsed_min": 25.0, "submitted": 1440, "finished": 1184, "in_flight": 256, "gen_tokens": 3626150, "tok_per_s": 2417, "chain_tokens_median": 1475.5, "chain_tokens_p90": 6357.5, "chain_tokens_max": 32532, "parse_rate": 0.9831081081081081, "looped": 20, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:37:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1989.0 tokens/s, Avg generation throughput: 2971.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 43.5%, Prefix cache hit rate: 0.0%
progress {"time": "08:38:59", "elapsed_min": 26.0, "submitted": 1453, "finished": 1197, "in_flight": 256, "gen_tokens": 3697685, "tok_per_s": 2370, "chain_tokens_median": 1490.0, "chain_tokens_p90": 6356.0, "chain_tokens_max": 32532, "parse_rate": 0.9824561403508771, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:38:59 [loggers.py:335] Engine 000: Avg prompt throughput: 932.8 tokens/s, Avg generation throughput: 3320.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 45.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:39:59", "elapsed_min": 27.0, "submitted": 1526, "finished": 1270, "in_flight": 256, "gen_tokens": 3899922, "tok_per_s": 2407, "chain_tokens_median": 1523.5, "chain_tokens_p90": 6286.4000000000015, "chain_tokens_max": 32532, "parse_rate": 0.9834645669291339, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:39:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3934.1 tokens/s, Avg generation throughput: 2396.6 tokens/s, Running: 127 reqs, Waiting: 129 reqs, GPU KV cache usage: 47.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:40:59", "elapsed_min": 28.0, "submitted": 1569, "finished": 1313, "in_flight": 256, "gen_tokens": 4028728, "tok_per_s": 2398, "chain_tokens_median": 1565.0, "chain_tokens_p90": 6236.599999999999, "chain_tokens_max": 32532, "parse_rate": 0.984006092916984, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:40:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2687.9 tokens/s, Avg generation throughput: 2658.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 49.6%, Prefix cache hit rate: 0.0%
progress {"time": "08:41:59", "elapsed_min": 29.0, "submitted": 1588, "finished": 1332, "in_flight": 256, "gen_tokens": 4082479, "tok_per_s": 2346, "chain_tokens_median": 1581.5, "chain_tokens_p90": 6213.700000000001, "chain_tokens_max": 32532, "parse_rate": 0.9842342342342343, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:41:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1343.6 tokens/s, Avg generation throughput: 2901.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.4%, Prefix cache hit rate: 0.4%
progress {"time": "08:42:59", "elapsed_min": 30.0, "submitted": 1677, "finished": 1421, "in_flight": 256, "gen_tokens": 4297616, "tok_per_s": 2387, "chain_tokens_median": 1586.0, "chain_tokens_p90": 6046.0, "chain_tokens_max": 32532, "parse_rate": 0.9852216748768473, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:42:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4315.3 tokens/s, Avg generation throughput: 2051.3 tokens/s, Running: 126 reqs, Waiting: 128 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 0.6%
progress {"time": "08:43:59", "elapsed_min": 31.0, "submitted": 1705, "finished": 1449, "in_flight": 256, "gen_tokens": 4431351, "tok_per_s": 2382, "chain_tokens_median": 1596.0, "chain_tokens_p90": 6065.200000000001, "chain_tokens_max": 32532, "parse_rate": 0.9841269841269841, "looped": 23, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:43:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1684.6 tokens/s, Avg generation throughput: 2676.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.3%, Prefix cache hit rate: 0.9%
progress {"time": "08:44:59", "elapsed_min": 32.0, "submitted": 1740, "finished": 1484, "in_flight": 256, "gen_tokens": 4502360, "tok_per_s": 2345, "chain_tokens_median": 1575.5, "chain_tokens_p90": 6044.1, "chain_tokens_max": 32532, "parse_rate": 0.9845013477088949, "looped": 23, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:44:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1733.2 tokens/s, Avg generation throughput: 2661.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 54.8%, Prefix cache hit rate: 1.3%
progress {"time": "08:45:59", "elapsed_min": 33.0, "submitted": 1821, "finished": 1565, "in_flight": 256, "gen_tokens": 4724627, "tok_per_s": 2386, "chain_tokens_median": 1576.0, "chain_tokens_p90": 5990.6, "chain_tokens_max": 32532, "parse_rate": 0.984664536741214, "looped": 24, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:45:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3980.6 tokens/s, Avg generation throughput: 1996.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 54.1%, Prefix cache hit rate: 1.8%
progress {"time": "08:46:59", "elapsed_min": 34.0, "submitted": 1839, "finished": 1583, "in_flight": 256, "gen_tokens": 4787783, "tok_per_s": 2347, "chain_tokens_median": 1586.0, "chain_tokens_p90": 6044.4, "chain_tokens_max": 32532, "parse_rate": 0.9848389134554643, "looped": 24, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:46:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1468.1 tokens/s, Avg generation throughput: 2728.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.2%, Prefix cache hit rate: 1.9%
progress {"time": "08:47:59", "elapsed_min": 35.0, "submitted": 1880, "finished": 1624, "in_flight": 256, "gen_tokens": 4897258, "tok_per_s": 2332, "chain_tokens_median": 1576.5, "chain_tokens_p90": 5990.7, "chain_tokens_max": 32532, "parse_rate": 0.9846059113300493, "looped": 25, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:47:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1896.3 tokens/s, Avg generation throughput: 2524.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 58.3%, Prefix cache hit rate: 2.3%
progress {"time": "08:48:59", "elapsed_min": 36.0, "submitted": 1950, "finished": 1694, "in_flight": 256, "gen_tokens": 5050366, "tok_per_s": 2338, "chain_tokens_median": 1575.5, "chain_tokens_p90": 5961.4, "chain_tokens_max": 32532, "parse_rate": 0.9852420306965761, "looped": 25, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:48:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3216.4 tokens/s, Avg generation throughput: 2126.8 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 2.5%
progress {"time": "08:49:59", "elapsed_min": 37.0, "submitted": 1971, "finished": 1715, "in_flight": 256, "gen_tokens": 5128978, "tok_per_s": 2310, "chain_tokens_median": 1577.0, "chain_tokens_p90": 5955.200000000002, "chain_tokens_max": 32594, "parse_rate": 0.9848396501457726, "looped": 26, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:49:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1338.6 tokens/s, Avg generation throughput: 2598.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 61.8%, Prefix cache hit rate: 2.5%
progress {"time": "08:50:59", "elapsed_min": 38.0, "submitted": 2016, "finished": 1760, "in_flight": 256, "gen_tokens": 5314684, "tok_per_s": 2331, "chain_tokens_median": 1593.0, "chain_tokens_p90": 5924.600000000004, "chain_tokens_max": 32601, "parse_rate": 0.9835227272727273, "looped": 29, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:50:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2103.3 tokens/s, Avg generation throughput: 2403.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.3%, Prefix cache hit rate: 2.6%
progress {"time": "08:51:59", "elapsed_min": 39.0, "submitted": 2079, "finished": 1823, "in_flight": 256, "gen_tokens": 5527220, "tok_per_s": 2362, "chain_tokens_median": 1575.0, "chain_tokens_p90": 5888.799999999999, "chain_tokens_max": 32609, "parse_rate": 0.9824465167306637, "looped": 32, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:51:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2570.5 tokens/s, Avg generation throughput: 2290.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.3%, Prefix cache hit rate: 2.8%
progress {"time": "08:52:59", "elapsed_min": 40.0, "submitted": 2099, "finished": 1843, "in_flight": 256, "gen_tokens": 5609243, "tok_per_s": 2337, "chain_tokens_median": 1582.0, "chain_tokens_p90": 5918.599999999999, "chain_tokens_max": 32609, "parse_rate": 0.9820944112859469, "looped": 33, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:52:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1284.9 tokens/s, Avg generation throughput: 2675.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.2%, Prefix cache hit rate: 2.5%
progress {"time": "08:53:59", "elapsed_min": 41.0, "submitted": 2114, "finished": 1858, "in_flight": 256, "gen_tokens": 5695518, "tok_per_s": 2315, "chain_tokens_median": 1572.0, "chain_tokens_p90": 5951.599999999999, "chain_tokens_max": 32609, "parse_rate": 0.9811625403659849, "looped": 35, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:53:59 [loggers.py:335] Engine 000: Avg prompt throughput: 768.6 tokens/s, Avg generation throughput: 2793.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.4%, Prefix cache hit rate: 2.5%
progress {"time": "08:54:59", "elapsed_min": 42.0, "submitted": 2185, "finished": 1929, "in_flight": 256, "gen_tokens": 5907026, "tok_per_s": 2344, "chain_tokens_median": 1577.0, "chain_tokens_p90": 5908.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9813374805598756, "looped": 36, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:54:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3748.3 tokens/s, Avg generation throughput: 1960.4 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 59.0%, Prefix cache hit rate: 2.1%
progress {"time": "08:55:59", "elapsed_min": 43.0, "submitted": 2225, "finished": 1969, "in_flight": 256, "gen_tokens": 6039758, "tok_per_s": 2341, "chain_tokens_median": 1599.0, "chain_tokens_p90": 5908.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9812087353986796, "looped": 36, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:55:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2863.8 tokens/s, Avg generation throughput: 2279.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 61.4%, Prefix cache hit rate: 1.6%
progress {"time": "08:56:59", "elapsed_min": 44.0, "submitted": 2237, "finished": 1981, "in_flight": 256, "gen_tokens": 6132150, "tok_per_s": 2323, "chain_tokens_median": 1606.0, "chain_tokens_p90": 5922.0, "chain_tokens_max": 32609, "parse_rate": 0.9803129732458354, "looped": 38, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:56:59 [loggers.py:335] Engine 000: Avg prompt throughput: 825.5 tokens/s, Avg generation throughput: 2782.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 1.4%
progress {"time": "08:57:59", "elapsed_min": 45.0, "submitted": 2277, "finished": 2021, "in_flight": 256, "gen_tokens": 6248411, "tok_per_s": 2314, "chain_tokens_median": 1602.0, "chain_tokens_p90": 5884.0, "chain_tokens_max": 32609, "parse_rate": 0.9802078179119248, "looped": 39, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:57:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1960.2 tokens/s, Avg generation throughput: 2397.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 63.4%, Prefix cache hit rate: 1.3%
progress {"time": "08:58:59", "elapsed_min": 46.0, "submitted": 2340, "finished": 2084, "in_flight": 256, "gen_tokens": 6434665, "tok_per_s": 2331, "chain_tokens_median": 1609.5, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9803262955854126, "looped": 40, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:58:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3265.1 tokens/s, Avg generation throughput: 1975.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 64.7%, Prefix cache hit rate: 0.8%
progress {"time": "08:59:59", "elapsed_min": 47.0, "submitted": 2360, "finished": 2104, "in_flight": 256, "gen_tokens": 6538259, "tok_per_s": 2318, "chain_tokens_median": 1615.5, "chain_tokens_p90": 5874.1, "chain_tokens_max": 32609, "parse_rate": 0.9795627376425855, "looped": 42, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:59:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1057.3 tokens/s, Avg generation throughput: 2620.2 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 63.7%, Prefix cache hit rate: 0.8%
progress {"time": "09:00:59", "elapsed_min": 48.0, "submitted": 2379, "finished": 2123, "in_flight": 256, "gen_tokens": 6641160, "tok_per_s": 2306, "chain_tokens_median": 1621.0, "chain_tokens_p90": 5882.2, "chain_tokens_max": 32609, "parse_rate": 0.9788035798398492, "looped": 44, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:00:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1096.5 tokens/s, Avg generation throughput: 2624.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 0.5%
progress {"time": "09:01:59", "elapsed_min": 49.0, "submitted": 2433, "finished": 2177, "in_flight": 256, "gen_tokens": 6801584, "tok_per_s": 2313, "chain_tokens_median": 1625.0, "chain_tokens_p90": 5878.6, "chain_tokens_max": 32609, "parse_rate": 0.979329352319706, "looped": 44, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:01:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3335.1 tokens/s, Avg generation throughput: 2103.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 63.6%, Prefix cache hit rate: 0.3%
progress {"time": "09:02:59", "elapsed_min": 50.0, "submitted": 2483, "finished": 2227, "in_flight": 256, "gen_tokens": 7056645, "tok_per_s": 2352, "chain_tokens_median": 1632.0, "chain_tokens_p90": 5944.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9784463403682083, "looped": 47, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:02:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3284.0 tokens/s, Avg generation throughput: 2125.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.7%, Prefix cache hit rate: 0.2%
progress {"time": "09:03:59", "elapsed_min": 51.0, "submitted": 2502, "finished": 2246, "in_flight": 256, "gen_tokens": 7185483, "tok_per_s": 2348, "chain_tokens_median": 1628.0, "chain_tokens_p90": 5954.0, "chain_tokens_max": 32609, "parse_rate": 0.9772929652715939, "looped": 50, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:03:59 [loggers.py:335] Engine 000: Avg prompt throughput: 839.8 tokens/s, Avg generation throughput: 2825.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 0.2%
progress {"time": "09:04:59", "elapsed_min": 52.0, "submitted": 2542, "finished": 2286, "in_flight": 256, "gen_tokens": 7353109, "tok_per_s": 2357, "chain_tokens_median": 1631.5, "chain_tokens_p90": 5954.0, "chain_tokens_max": 32609, "parse_rate": 0.9768153980752406, "looped": 52, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:04:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2328.6 tokens/s, Avg generation throughput: 2463.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 0.2%
progress {"time": "09:05:59", "elapsed_min": 53.0, "submitted": 2599, "finished": 2343, "in_flight": 256, "gen_tokens": 7506482, "tok_per_s": 2360, "chain_tokens_median": 1657.0, "chain_tokens_p90": 5902.000000000003, "chain_tokens_max": 32609, "parse_rate": 0.9773794280836534, "looped": 52, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:05:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3190.8 tokens/s, Avg generation throughput: 2229.6 tokens/s, Running: 127 reqs, Waiting: 128 reqs, GPU KV cache usage: 58.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:06:59", "elapsed_min": 54.0, "submitted": 2630, "finished": 2374, "in_flight": 256, "gen_tokens": 7690019, "tok_per_s": 2373, "chain_tokens_median": 1682.5, "chain_tokens_p90": 5916.900000000005, "chain_tokens_max": 32609, "parse_rate": 0.9764111204717776, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:06:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1926.2 tokens/s, Avg generation throughput: 2595.8 tokens/s, Running: 127 reqs, Waiting: 128 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 0.0%
progress {"time": "09:07:59", "elapsed_min": 55.0, "submitted": 2659, "finished": 2403, "in_flight": 256, "gen_tokens": 7773368, "tok_per_s": 2355, "chain_tokens_median": 1720.0, "chain_tokens_p90": 5889.000000000001, "chain_tokens_max": 32609, "parse_rate": 0.976695796920516, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:07:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1874.4 tokens/s, Avg generation throughput: 2580.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 0.0%
progress {"time": "09:08:59", "elapsed_min": 56.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 7896972, "tok_per_s": 2350, "chain_tokens_median": 1686.0, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9772450223486387, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:08:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2715.0 tokens/s, Avg generation throughput: 2276.9 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 60.0%, Prefix cache hit rate: 0.0%
progress {"time": "09:09:59", "elapsed_min": 57.0, "submitted": 2760, "finished": 2504, "in_flight": 256, "gen_tokens": 8109477, "tok_per_s": 2371, "chain_tokens_median": 1723.5, "chain_tokens_p90": 5881.300000000003, "chain_tokens_max": 32609, "parse_rate": 0.9764376996805112, "looped": 58, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:09:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2452.9 tokens/s, Avg generation throughput: 2431.0 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 57.1%, Prefix cache hit rate: 0.0%
progress {"time": "09:10:59", "elapsed_min": 58.0, "submitted": 2783, "finished": 2527, "in_flight": 256, "gen_tokens": 8237496, "tok_per_s": 2367, "chain_tokens_median": 1745.0, "chain_tokens_p90": 5884.4, "chain_tokens_max": 32609, "parse_rate": 0.9758607043925603, "looped": 60, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:10:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1453.5 tokens/s, Avg generation throughput: 2728.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.3%, Prefix cache hit rate: 0.0%
progress {"time": "09:11:59", "elapsed_min": 59.0, "submitted": 2845, "finished": 2589, "in_flight": 256, "gen_tokens": 8485585, "tok_per_s": 2397, "chain_tokens_median": 1749.0, "chain_tokens_p90": 5885.0, "chain_tokens_max": 32609, "parse_rate": 0.9752800308999614, "looped": 63, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:11:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3196.2 tokens/s, Avg generation throughput: 2303.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 51.8%, Prefix cache hit rate: 0.0%
progress {"time": "09:12:59", "elapsed_min": 60.0, "submitted": 2893, "finished": 2637, "in_flight": 256, "gen_tokens": 8674516, "tok_per_s": 2409, "chain_tokens_median": 1790.0, "chain_tokens_p90": 5887.0, "chain_tokens_max": 32609, "parse_rate": 0.975350777398559, "looped": 64, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:12:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3230.1 tokens/s, Avg generation throughput: 2458.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 51.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:13:59", "elapsed_min": 61.0, "submitted": 2926, "finished": 2670, "in_flight": 256, "gen_tokens": 8825417, "tok_per_s": 2411, "chain_tokens_median": 1816.5, "chain_tokens_p90": 5885.0, "chain_tokens_max": 32609, "parse_rate": 0.9749063670411985, "looped": 66, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:13:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1973.7 tokens/s, Avg generation throughput: 2781.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.9%, Prefix cache hit rate: 0.0%
progress {"time": "09:14:59", "elapsed_min": 62.0, "submitted": 3003, "finished": 2747, "in_flight": 256, "gen_tokens": 9134214, "tok_per_s": 2455, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9741536221332363, "looped": 70, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:14:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3829.7 tokens/s, Avg generation throughput: 2343.8 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 46.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:15:59", "elapsed_min": 63.0, "submitted": 3034, "finished": 2778, "in_flight": 256, "gen_tokens": 9239868, "tok_per_s": 2444, "chain_tokens_median": 1850.0, "chain_tokens_p90": 5853.800000000005, "chain_tokens_max": 32609, "parse_rate": 0.9740820734341252, "looped": 71, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:15:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2049.3 tokens/s, Avg generation throughput: 2772.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.1%, Prefix cache hit rate: 0.0%
progress {"time": "09:16:59", "elapsed_min": 64.0, "submitted": 3094, "finished": 2838, "in_flight": 256, "gen_tokens": 9417973, "tok_per_s": 2452, "chain_tokens_median": 1850.0, "chain_tokens_p90": 5835.200000000001, "chain_tokens_max": 32609, "parse_rate": 0.974277660324172, "looped": 72, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:16:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3508.2 tokens/s, Avg generation throughput: 2427.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.5%, Prefix cache hit rate: 0.1%
progress {"time": "09:17:59", "elapsed_min": 65.0, "submitted": 3145, "finished": 2889, "in_flight": 256, "gen_tokens": 9526840, "tok_per_s": 2443, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5806.400000000007, "chain_tokens_max": 32609, "parse_rate": 0.9747317410868813, "looped": 72, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:17:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2765.2 tokens/s, Avg generation throughput: 2586.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.9%, Prefix cache hit rate: 0.4%
progress {"time": "09:18:59", "elapsed_min": 66.0, "submitted": 3193, "finished": 2937, "in_flight": 256, "gen_tokens": 9704069, "tok_per_s": 2450, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5796.200000000001, "chain_tokens_max": 32609, "parse_rate": 0.9744637385086823, "looped": 74, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:18:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2828.7 tokens/s, Avg generation throughput: 2432.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 49.3%, Prefix cache hit rate: 1.0%
progress {"time": "09:19:59", "elapsed_min": 67.0, "submitted": 3268, "finished": 3012, "in_flight": 256, "gen_tokens": 9891835, "tok_per_s": 2461, "chain_tokens_median": 1814.0, "chain_tokens_p90": 5773.1, "chain_tokens_max": 32609, "parse_rate": 0.9744355909694555, "looped": 76, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:19:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2870.7 tokens/s, Avg generation throughput: 2446.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 48.0%, Prefix cache hit rate: 2.0%
progress {"time": "09:20:59", "elapsed_min": 68.0, "submitted": 3303, "finished": 3047, "in_flight": 256, "gen_tokens": 10060251, "tok_per_s": 2466, "chain_tokens_median": 1810.0, "chain_tokens_p90": 5782.6, "chain_tokens_max": 32609, "parse_rate": 0.9737446668854611, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:20:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1999.5 tokens/s, Avg generation throughput: 2791.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.0%, Prefix cache hit rate: 2.1%
progress {"time": "09:21:59", "elapsed_min": 69.0, "submitted": 3375, "finished": 3119, "in_flight": 256, "gen_tokens": 10205474, "tok_per_s": 2465, "chain_tokens_median": 1777.0, "chain_tokens_p90": 5761.800000000001, "chain_tokens_max": 32609, "parse_rate": 0.9743507534466175, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:21:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3165.8 tokens/s, Avg generation throughput: 2510.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 44.6%, Prefix cache hit rate: 2.3%
progress {"time": "09:22:59", "elapsed_min": 70.0, "submitted": 3393, "finished": 3169, "in_flight": 224, "gen_tokens": 10346752, "tok_per_s": 2463, "chain_tokens_median": 1781.0, "chain_tokens_p90": 5761.800000000001, "chain_tokens_max": 32609, "parse_rate": 0.9747554433575261, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:22:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3240.5 tokens/s, Avg generation throughput: 2577.3 tokens/s, Running: 128 reqs, Waiting: 95 reqs, GPU KV cache usage: 47.1%, Prefix cache hit rate: 3.1%
progress {"time": "09:23:59", "elapsed_min": 71.0, "submitted": 3393, "finished": 3258, "in_flight": 135, "gen_tokens": 10553734, "tok_per_s": 2477, "chain_tokens_median": 1752.0, "chain_tokens_p90": 5756.100000000001, "chain_tokens_max": 32609, "parse_rate": 0.9754450583179864, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:23:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4118.8 tokens/s, Avg generation throughput: 2313.4 tokens/s, Running: 126 reqs, Waiting: 9 reqs, GPU KV cache usage: 48.4%, Prefix cache hit rate: 3.2%
progress {"time": "09:24:59", "elapsed_min": 72.0, "submitted": 3393, "finished": 3310, "in_flight": 83, "gen_tokens": 10687462, "tok_per_s": 2474, "chain_tokens_median": 1761.5, "chain_tokens_p90": 5745.2, "chain_tokens_max": 32609, "parse_rate": 0.9758308157099698, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:24:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2549.9 tokens/s, Avg generation throughput: 2266.4 tokens/s, Running: 83 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.6%, Prefix cache hit rate: 3.4%
progress {"time": "09:25:59", "elapsed_min": 73.0, "submitted": 3393, "finished": 3342, "in_flight": 51, "gen_tokens": 10936272, "tok_per_s": 2497, "chain_tokens_median": 1809.5, "chain_tokens_p90": 5778.5, "chain_tokens_max": 32609, "parse_rate": 0.9751645721125075, "looped": 82, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:25:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2774.6 tokens/s, Avg generation throughput: 1771.5 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.3%, Prefix cache hit rate: 3.1%
progress {"time": "09:26:59", "elapsed_min": 74.0, "submitted": 3393, "finished": 3362, "in_flight": 31, "gen_tokens": 11140853, "tok_per_s": 2509, "chain_tokens_median": 1825.5, "chain_tokens_p90": 5845.300000000001, "chain_tokens_max": 32609, "parse_rate": 0.97441998810232, "looped": 85, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:26:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1923.5 tokens/s, Avg generation throughput: 1561.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 3.0%
progress {"time": "09:27:59", "elapsed_min": 75.0, "submitted": 3393, "finished": 3367, "in_flight": 26, "gen_tokens": 11209436, "tok_per_s": 2491, "chain_tokens_median": 1836.0, "chain_tokens_p90": 5878.6, "chain_tokens_max": 32609, "parse_rate": 0.9741609741609741, "looped": 86, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:27:59 [loggers.py:335] Engine 000: Avg prompt throughput: 619.2 tokens/s, Avg generation throughput: 1482.4 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 2.9%
progress {"time": "09:28:59", "elapsed_min": 76.0, "submitted": 3393, "finished": 3371, "in_flight": 22, "gen_tokens": 11300106, "tok_per_s": 2478, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5890.0, "chain_tokens_max": 32609, "parse_rate": 0.9735983387718777, "looped": 88, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:28:59 [loggers.py:335] Engine 000: Avg prompt throughput: 447.3 tokens/s, Avg generation throughput: 1288.2 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 2.9%
progress {"time": "09:29:59", "elapsed_min": 77.0, "submitted": 3393, "finished": 3373, "in_flight": 20, "gen_tokens": 11364874, "tok_per_s": 2460, "chain_tokens_median": 1849.0, "chain_tokens_p90": 5891.6, "chain_tokens_max": 32609, "parse_rate": 0.9730210495108212, "looped": 90, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:29:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1163.2 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 2.9%
progress {"time": "09:30:59", "elapsed_min": 78.0, "submitted": 3393, "finished": 3374, "in_flight": 19, "gen_tokens": 11397252, "tok_per_s": 2435, "chain_tokens_median": 1849.5, "chain_tokens_p90": 5901.100000000004, "chain_tokens_max": 32609, "parse_rate": 0.972732661529342, "looped": 91, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:30:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1058.4 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 2.9%
progress {"time": "09:31:59", "elapsed_min": 79.0, "submitted": 3393, "finished": 3380, "in_flight": 13, "gen_tokens": 11589503, "tok_per_s": 2445, "chain_tokens_median": 1857.5, "chain_tokens_p90": 5960.2, "chain_tokens_max": 32609, "parse_rate": 0.9710059171597634, "looped": 97, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:31:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 952.6 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 2.9%
progress {"time": "09:33:00", "elapsed_min": 80.0, "submitted": 3393, "finished": 3381, "in_flight": 12, "gen_tokens": 11621729, "tok_per_s": 2421, "chain_tokens_median": 1860.0, "chain_tokens_p90": 5962.0, "chain_tokens_max": 32609, "parse_rate": 0.9707187222715173, "looped": 98, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:33:00 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 789.7 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 2.9%
progress {"time": "09:34:00", "elapsed_min": 81.0, "submitted": 3393, "finished": 3387, "in_flight": 6, "gen_tokens": 11815954, "tok_per_s": 2431, "chain_tokens_median": 1868.0, "chain_tokens_p90": 5985.200000000001, "chain_tokens_max": 32646, "parse_rate": 0.9689991142604074, "looped": 104, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:34:00 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 2.9%
reason joint: {"submitted": 3393, "finished": 3393, "gen_tokens": 12010760, "t0": 1793434379.7662153, "seconds": 4885.523628473282, "tok_per_s": 2458.438626721644}
reason joint: post-chain readout 5847 taken at generation, 0 by prefill
reason joint: 5957 items, post-readout in 4886s total
latency joint: {"readout_ms": {"median": 40.53856432437897, "p90": 74.21627789735797, "n": 100}, "reasoning_ms": {"median": 11583.74154753983, "p90": 55556.02454841137, "n": 20}}
(MainProcess pid=52652) INFO 10-31 09:44:46 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=52652) INFO 10-31 09:44:46 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=52652) INFO 10-31 09:44:46 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:777] [shutdown] MPClient: complete
Xet Storage Details

Size:: 108 kB
Xet hash:: f7c9b407a9f71c6492e475e8898a31d178277c1c8b62a637dc649c9c46ba2706
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.