Buckets:
| (MainProcess pid=160904) INFO 11-01 04:08:06 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=160904) INFO 11-01 04:08:06 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=160904) INFO 11-01 04:08:06 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=160904) INFO 11-01 04:08:06 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| Fetching 10 files: 0%| | 0/10 [00:00<?, ?it/s] Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 942.71it/s] | |
| (EngineCore pid=161179) INFO 11-01 04:08:09 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='openbmb/MiniCPM5-2B', speculative_config=None, tokenizer='openbmb/MiniCPM5-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=openbmb/MiniCPM5-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 192, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=161179) INFO 11-01 04:08:10 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_5d481ffa19714397b5ee3bce53e5b060 backend=nccl | |
| (EngineCore pid=161179) INFO 11-01 04:08:10 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=161179) INFO 11-01 04:08:10 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=161179) INFO 11-01 04:08:12 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=161179) INFO 11-01 04:08:13 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=161179) INFO 11-01 04:08:13 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=161179) INFO 11-01 04:08:13 [base_loader.py:82] Initializing model took 1.439778 seconds | |
| (EngineCore pid=161179) INFO 11-01 04:08:13 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 128.93 GiB. | |
| (EngineCore pid=161179) INFO 11-01 04:08:13 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.02it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.02it/s] | |
| (EngineCore pid=161179) INFO 11-01 04:08:14 [default_loader.py:501] Loading weights took 0.50 seconds | |
| (EngineCore pid=161179) INFO 11-01 04:08:15 [model_runner.py:424] Model loading took 4.75 GiB memory and 3.900131 seconds | |
| (EngineCore pid=161179) INFO 11-01 04:08:15 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=161179) INFO 11-01 04:08:15 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=161179) INFO 11-01 04:08:15 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=161179) INFO 11-01 04:08:16 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=161179) INFO 11-01 04:08:26 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/b8a233aec8/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=161179) INFO 11-01 04:08:26 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.83 s | |
| (EngineCore pid=161179) INFO 11-01 04:08:35 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 9.21 s | |
| (EngineCore pid=161179) INFO 11-01 04:08:41 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889897 bytes total | |
| (EngineCore pid=161179) INFO 11-01 04:08:41 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/079f9792e3d16f500ebd185e4eec3d55f120146e9213b0e5f52379456efe4c60/rank_0_0/model | |
| (EngineCore pid=161179) INFO 11-01 04:08:41 [monitor.py:62] [backbone] torch.compile took 24.96 s in total | |
| (EngineCore pid=161179) INFO 11-01 04:08:41 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.21 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 16.86it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 20.44it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 21.53it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 21.98it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 21.35it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 21.03it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 20.73it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 20.63it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 19.56it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.12it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.39it/s] | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [model_runner.py:1127] Graph capturing finished in 2 secs, took 0.24 GiB | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [gpu_worker.py:759] Available KV cache memory: 67.09 GiB | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9164 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9236. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [kv_cache_utils.py:2327] GPU KV cache size: 1,675,072 tokens, Maximum concurrency for 32,768 tokens per request: 51.12x | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=161179) INFO 11-01 04:08:46 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 14.15it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 19.28it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 21.02it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 21.82it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 21.79it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 21.48it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 21.36it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 21.13it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 20.84it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.86it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/15 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 20%|██ | 3/15 [00:00<00:00, 24.02it/s] Capturing CUDA graphs (FULL): 40%|████ | 6/15 [00:00<00:00, 24.59it/s] Capturing CUDA graphs (FULL): 60%|██████ | 9/15 [00:00<00:00, 24.82it/s] Capturing CUDA graphs (FULL): 80%|████████ | 12/15 [00:00<00:00, 25.07it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.29it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.03it/s] | |
| (EngineCore pid=161179) INFO 11-01 04:09:07 [model_runner.py:1127] Graph capturing finished in 4 secs, took 0.12 GiB | |
| (EngineCore pid=161179) INFO 11-01 04:09:07 [gpu_worker.py:971] CUDA graph pool memory: 0.12 GiB (actual), 0.29 GiB (estimated), difference: 0.17 GiB (146.7%). | |
| (EngineCore pid=161179) INFO 11-01 04:09:07 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.02 GiB for consumed memory (weights + non-torch), 0.79 GiB for peak activation, and 0.12 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=71758921954` (66.83 GiB) to fit into requested memory, or `--kv-cache-memory=78123091456` (72.76 GiB) to fully utilize gpu memory. Current kv cache memory in use is 67.09 GiB. | |
| (EngineCore pid=161179) INFO 11-01 04:09:07 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=161179) INFO 11-01 04:09:08 [core.py:395] init engine (profile, create kv cache, warmup model) took 52.65 s (compilation: 24.96 s) | |
| (EngineCore pid=161179) INFO 11-01 04:09:08 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=160904) WARNING 11-01 04:09:09 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=161179) WARNING 11-01 04:09:09 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=161179) WARNING 11-01 04:09:10 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=161179) WARNING 11-01 04:09:11 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=161179) WARNING 11-01 04:09:48 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "04:10:08", "elapsed_min": 1.0, "submitted": 421, "finished": 165, "in_flight": 256, "gen_tokens": 113482, "tok_per_s": 1891, "chain_tokens_median": 532.0, "chain_tokens_p90": 1384.8, "chain_tokens_max": 2158, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:10:08 [loggers.py:335] Engine 000: Avg prompt throughput: 3589.8 tokens/s, Avg generation throughput: 4933.3 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 51.6% | |
| progress {"time": "04:11:08", "elapsed_min": 2.0, "submitted": 776, "finished": 520, "in_flight": 256, "gen_tokens": 414951, "tok_per_s": 3458, "chain_tokens_median": 504.0, "chain_tokens_p90": 1879.6000000000004, "chain_tokens_max": 4891, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:11:08 [loggers.py:335] Engine 000: Avg prompt throughput: 2318.4 tokens/s, Avg generation throughput: 4250.8 tokens/s, Running: 96 reqs, Waiting: 156 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 79.6% | |
| progress {"time": "04:12:08", "elapsed_min": 3.0, "submitted": 1240, "finished": 984, "in_flight": 256, "gen_tokens": 677204, "tok_per_s": 3762, "chain_tokens_median": 404.5, "chain_tokens_p90": 1588.8000000000002, "chain_tokens_max": 5787, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:12:08 [loggers.py:335] Engine 000: Avg prompt throughput: 2027.9 tokens/s, Avg generation throughput: 4193.5 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 75.1% | |
| progress {"time": "04:13:08", "elapsed_min": 4.0, "submitted": 1571, "finished": 1315, "in_flight": 256, "gen_tokens": 880111, "tok_per_s": 3667, "chain_tokens_median": 407.0, "chain_tokens_p90": 1554.6000000000001, "chain_tokens_max": 5787, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:13:08 [loggers.py:335] Engine 000: Avg prompt throughput: 1538.9 tokens/s, Avg generation throughput: 3710.2 tokens/s, Running: 96 reqs, Waiting: 159 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 80.5% | |
| progress {"time": "04:14:08", "elapsed_min": 5.0, "submitted": 1739, "finished": 1483, "in_flight": 256, "gen_tokens": 1011567, "tok_per_s": 3372, "chain_tokens_median": 422.0, "chain_tokens_p90": 1540.8, "chain_tokens_max": 5787, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:14:08 [loggers.py:335] Engine 000: Avg prompt throughput: 2297.1 tokens/s, Avg generation throughput: 3333.9 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 73.0% | |
| progress {"time": "04:15:08", "elapsed_min": 6.0, "submitted": 2013, "finished": 1794, "in_flight": 219, "gen_tokens": 1257500, "tok_per_s": 3493, "chain_tokens_median": 433.0, "chain_tokens_p90": 1551.7, "chain_tokens_max": 10322, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:15:08 [loggers.py:335] Engine 000: Avg prompt throughput: 1575.2 tokens/s, Avg generation throughput: 2821.0 tokens/s, Running: 96 reqs, Waiting: 122 reqs, GPU KV cache usage: 9.7%, Prefix cache hit rate: 72.9% | |
| progress {"time": "04:16:08", "elapsed_min": 7.0, "submitted": 2013, "finished": 2004, "in_flight": 9, "gen_tokens": 1464972, "tok_per_s": 3488, "chain_tokens_median": 449.0, "chain_tokens_p90": 1600.0000000000005, "chain_tokens_max": 10322, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:16:08 [loggers.py:335] Engine 000: Avg prompt throughput: 414.0 tokens/s, Avg generation throughput: 2422.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 78.4% | |
| progress {"time": "04:17:08", "elapsed_min": 8.0, "submitted": 2013, "finished": 2011, "in_flight": 2, "gen_tokens": 1516764, "tok_per_s": 3160, "chain_tokens_median": 451.0, "chain_tokens_p90": 1622.0, "chain_tokens_max": 16754, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:17:08 [loggers.py:335] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 403.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 79.3% | |
| progress {"time": "04:18:08", "elapsed_min": 9.0, "submitted": 2013, "finished": 2012, "in_flight": 1, "gen_tokens": 1547187, "tok_per_s": 2865, "chain_tokens_median": 451.5, "chain_tokens_p90": 1623.8000000000002, "chain_tokens_max": 30423, "parse_rate": 0.9995029821073559, "looped": 1, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=160904) INFO 11-01 04:18:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 223.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 79.3% | |
| reason base2b: {"submitted": 2013, "finished": 2013, "gen_tokens": 1579074, "t0": 1793506148.658043, "seconds": 559.0171127319336, "tok_per_s": 2824.732846339922} | |
| reason base2b: post-chain readout 2011 taken at generation, 0 by prefill | |
| reason base2b: 2013 items, post-readout in 559s total | |
| (MainProcess pid=160904) INFO 11-01 04:18:27 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=160904) INFO 11-01 04:18:27 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=160904) INFO 11-01 04:18:27 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=161179) INFO 11-01 04:18:27 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=161179) INFO 11-01 04:18:27 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=161179) INFO 11-01 04:18:27 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=161179) INFO 11-01 04:18:27 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=160904) INFO 11-01 04:18:30 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=160904) INFO 11-01 04:18:30 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=160904) INFO 11-01 04:18:30 [core_client.py:777] [shutdown] MPClient: complete | |
Xet Storage Details
- Size:
- 20.9 kB
- Xet hash:
- bee0e10ace48bdaa5068827cc714c0d6bbbbf07bc9b685ce2f07fd7347fdd865
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.