Hugging Face's logo

Buckets:
Mercity
/
Desicion-Model-Custom

Files
xet
Mercity/Desicion-Model-Custom / run1 /logs /baseline.log
Pranav2748's picture
Pranav2748
download
raw
52.9 kB
(MainProcess pid=27742) INFO 10-31 01:03:39 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=27742) INFO 10-31 01:03:39 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1315.38it/s]
(MainProcess pid=27742) INFO 10-31 01:03:41 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=27742) INFO 10-31 01:03:41 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.46it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 8.15it/s]
(MainProcess pid=27742) INFO 10-31 01:03:42 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=27742) WARNING 10-31 01:03:53 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=28182) INFO 10-31 01:04:03 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=28182) INFO 10-31 01:04:04 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_633fbdd626484f9c92eeae6e2b5b8bd6 backend=nccl
(EngineCore pid=28182) INFO 10-31 01:04:04 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=28182) INFO 10-31 01:04:04 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=28182) INFO 10-31 01:04:06 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1131.02it/s]
(EngineCore pid=28182) INFO 10-31 01:04:07 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=28182) INFO 10-31 01:04:07 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=28182) INFO 10-31 01:04:07 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=28182) INFO 10-31 01:04:08 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=28182) INFO 10-31 01:04:08 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=28182) INFO 10-31 01:04:08 [base_loader.py:82] Initializing model took 2.133625 seconds
(EngineCore pid=28182) INFO 10-31 01:04:08 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.61 GiB.
(EngineCore pid=28182) INFO 10-31 01:04:08 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.83it/s]
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.94it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.93it/s]
(EngineCore pid=28182) INFO 10-31 01:04:09 [default_loader.py:501] Loading weights took 1.04 seconds
(EngineCore pid=28182) INFO 10-31 01:04:10 [model_runner.py:424] Model loading took 8.56 GiB memory and 5.040389 seconds
(EngineCore pid=28182) INFO 10-31 01:04:11 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=28182) INFO 10-31 01:04:11 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=28182) INFO 10-31 01:04:11 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=28182) INFO 10-31 01:04:11 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=28182) INFO 10-31 01:04:11 [utils.py:347] Using LBNHC KV cache layout.
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=27742) INFO 10-31 01:04:15 [base.py:280] Multi-modal warmup completed in 21.668s
(EngineCore pid=28182) INFO 10-31 01:04:18 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=28182) INFO 10-31 01:04:23 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33
(EngineCore pid=28182) INFO 10-31 01:04:23 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model
(EngineCore pid=28182) INFO 10-31 01:04:23 [monitor.py:62] [backbone] torch.compile took 0.98 s in total
(EngineCore pid=28182) INFO 10-31 01:04:23 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.32 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:16, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.64it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:03<00:46, 1.46s/it] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:03<00:31, 1.00s/it] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:04<00:22, 1.35it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:04<00:16, 1.74it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:04<00:13, 2.14it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:04<00:10, 2.52it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:05<00:09, 2.69it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:05<00:08, 2.92it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:05<00:07, 3.09it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:05<00:06, 3.41it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:06<00:05, 3.79it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:06<00:05, 4.17it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:06<00:04, 4.60it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:06<00:03, 5.05it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:06<00:02, 7.27it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:06<00:01, 9.48it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:07<00:01, 11.43it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:07<00:00, 12.58it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:07<00:00, 14.14it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:07<00:00, 15.34it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:07<00:00, 16.39it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:07<00:00, 16.88it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:07<00:00, 17.40it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:07<00:00, 4.52it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 24.25it/s]
(EngineCore pid=28182) INFO 10-31 01:04:33 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.28 GiB
(EngineCore pid=28182) INFO 10-31 01:04:33 [gpu_worker.py:759] Available KV cache memory: 60.55 GiB
(EngineCore pid=28182) INFO 10-31 01:04:33 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:2327] GPU KV cache size: 1,784,193 tokens, Maximum concurrency for 32,768 tokens per request: 54.45x
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=28182) INFO 10-31 01:04:33 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=28182) INFO 10-31 01:04:33 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=28182) INFO 10-31 01:04:33 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=28182) INFO 10-31 01:04:33 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=28182) INFO 10-31 01:04:33 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:10, 3.35it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.75it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.66it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.68it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.78it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:09, 2.92it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:08, 3.14it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.36it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:02<00:07, 3.28it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.13it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:07, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.62it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 3.81it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:05, 4.16it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.55it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.97it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 7.09it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 8.88it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.37it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:00, 11.63it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 12.55it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 13.13it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 13.68it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.90it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 14.16it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 6.00it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 20.51it/s] Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 20.71it/s] Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 20.72it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 20.92it/s] Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 21.39it/s] Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 21.48it/s] Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.23it/s]
(EngineCore pid=28182) INFO 10-31 01:04:41 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.14 GiB
(EngineCore pid=28182) INFO 10-31 01:04:41 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%).
(EngineCore pid=28182) INFO 10-31 01:04:41 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 8.86 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=64709187482` (60.27 GiB) to fit into requested memory, or `--kv-cache-memory=72775253504` (67.78 GiB) to fully utilize gpu memory. Current kv cache memory in use is 60.55 GiB.
(EngineCore pid=28182) INFO 10-31 01:04:42 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=28182) INFO 10-31 01:04:42 [core.py:395] init engine (profile, create kv cache, warmup model) took 31.63 s (compilation: 0.98 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.44it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 8.25it/s]
(EngineCore pid=28182) INFO 10-31 01:04:43 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=27742) WARNING 10-31 01:04:43 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=28182) WARNING 10-31 01:04:44 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout base: 5957 items in 114s
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "01:07:37", "elapsed_min": 1.0, "submitted": 329, "finished": 73, "in_flight": 256, "gen_tokens": 84514, "tok_per_s": 1408, "chain_tokens_median": 986.0, "chain_tokens_p90": 2080.8, "chain_tokens_max": 2321, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:08:37", "elapsed_min": 2.0, "submitted": 392, "finished": 136, "in_flight": 256, "gen_tokens": 232486, "tok_per_s": 1937, "chain_tokens_median": 1451.0, "chain_tokens_p90": 3250.5, "chain_tokens_max": 4430, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:09:37", "elapsed_min": 3.0, "submitted": 475, "finished": 219, "in_flight": 256, "gen_tokens": 463478, "tok_per_s": 2575, "chain_tokens_median": 1658.0, "chain_tokens_p90": 4690.8, "chain_tokens_max": 6432, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:10:37", "elapsed_min": 4.0, "submitted": 527, "finished": 271, "in_flight": 256, "gen_tokens": 669475, "tok_per_s": 2789, "chain_tokens_median": 1993.0, "chain_tokens_p90": 5312.0, "chain_tokens_max": 8269, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:11:37", "elapsed_min": 5.0, "submitted": 604, "finished": 348, "in_flight": 256, "gen_tokens": 922303, "tok_per_s": 3074, "chain_tokens_median": 2004.5, "chain_tokens_p90": 5857.300000000001, "chain_tokens_max": 9657, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:12:37", "elapsed_min": 6.0, "submitted": 721, "finished": 465, "in_flight": 256, "gen_tokens": 1182133, "tok_per_s": 3284, "chain_tokens_median": 1734.0, "chain_tokens_p90": 5811.6, "chain_tokens_max": 11244, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:13:37", "elapsed_min": 7.0, "submitted": 805, "finished": 549, "in_flight": 256, "gen_tokens": 1393709, "tok_per_s": 3318, "chain_tokens_median": 1665.0, "chain_tokens_p90": 6047.599999999999, "chain_tokens_max": 13383, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:14:37", "elapsed_min": 8.0, "submitted": 886, "finished": 630, "in_flight": 256, "gen_tokens": 1624499, "tok_per_s": 3384, "chain_tokens_median": 1725.0, "chain_tokens_p90": 6091.800000000003, "chain_tokens_max": 14175, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:15:37", "elapsed_min": 9.0, "submitted": 982, "finished": 726, "in_flight": 256, "gen_tokens": 1874461, "tok_per_s": 3471, "chain_tokens_median": 1661.5, "chain_tokens_p90": 6075.5, "chain_tokens_max": 14175, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:16:37", "elapsed_min": 10.0, "submitted": 1090, "finished": 834, "in_flight": 256, "gen_tokens": 2149312, "tok_per_s": 3582, "chain_tokens_median": 1655.0, "chain_tokens_p90": 5980.500000000002, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:17:37", "elapsed_min": 11.0, "submitted": 1154, "finished": 898, "in_flight": 256, "gen_tokens": 2302889, "tok_per_s": 3489, "chain_tokens_median": 1633.0, "chain_tokens_p90": 5898.100000000002, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:18:37", "elapsed_min": 12.0, "submitted": 1226, "finished": 970, "in_flight": 256, "gen_tokens": 2500454, "tok_per_s": 3473, "chain_tokens_median": 1644.5, "chain_tokens_p90": 5832.5, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:19:37", "elapsed_min": 13.0, "submitted": 1333, "finished": 1077, "in_flight": 256, "gen_tokens": 2750870, "tok_per_s": 3527, "chain_tokens_median": 1616.0, "chain_tokens_p90": 5770.199999999999, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:20:37", "elapsed_min": 14.0, "submitted": 1489, "finished": 1233, "in_flight": 256, "gen_tokens": 2986596, "tok_per_s": 3555, "chain_tokens_median": 1380.0, "chain_tokens_p90": 5705.0, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:21:37", "elapsed_min": 15.0, "submitted": 1593, "finished": 1337, "in_flight": 256, "gen_tokens": 3166310, "tok_per_s": 3518, "chain_tokens_median": 1303.0, "chain_tokens_p90": 5640.800000000001, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:22:37", "elapsed_min": 16.0, "submitted": 1669, "finished": 1413, "in_flight": 256, "gen_tokens": 3331794, "tok_per_s": 3470, "chain_tokens_median": 1323.0, "chain_tokens_p90": 5626.4, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:23:37", "elapsed_min": 17.0, "submitted": 1795, "finished": 1539, "in_flight": 256, "gen_tokens": 3521226, "tok_per_s": 3452, "chain_tokens_median": 1256.0, "chain_tokens_p90": 5478.8, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:24:37", "elapsed_min": 18.0, "submitted": 1893, "finished": 1637, "in_flight": 256, "gen_tokens": 3733257, "tok_per_s": 3457, "chain_tokens_median": 1230.0, "chain_tokens_p90": 5395.400000000004, "chain_tokens_max": 31330, "parse_rate": 0.9993891264508247, "looped": 1, "minutes_to_deadline": Infinity}
progress {"time": "01:25:37", "elapsed_min": 19.0, "submitted": 1951, "finished": 1695, "in_flight": 256, "gen_tokens": 3964743, "tok_per_s": 3478, "chain_tokens_median": 1257.0, "chain_tokens_p90": 5443.2000000000035, "chain_tokens_max": 31330, "parse_rate": 0.9976401179941002, "looped": 4, "minutes_to_deadline": Infinity}
progress {"time": "01:26:37", "elapsed_min": 20.0, "submitted": 2068, "finished": 1812, "in_flight": 256, "gen_tokens": 4207918, "tok_per_s": 3506, "chain_tokens_median": 1235.5, "chain_tokens_p90": 5349.800000000001, "chain_tokens_max": 31330, "parse_rate": 0.9972406181015453, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:27:37", "elapsed_min": 21.0, "submitted": 2162, "finished": 1906, "in_flight": 256, "gen_tokens": 4368084, "tok_per_s": 3467, "chain_tokens_median": 1204.0, "chain_tokens_p90": 5249.5, "chain_tokens_max": 31330, "parse_rate": 0.9973767051416579, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:28:37", "elapsed_min": 22.0, "submitted": 2202, "finished": 1946, "in_flight": 256, "gen_tokens": 4501259, "tok_per_s": 3410, "chain_tokens_median": 1231.5, "chain_tokens_p90": 5275.0, "chain_tokens_max": 31330, "parse_rate": 0.9974306269270298, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:29:37", "elapsed_min": 23.0, "submitted": 2277, "finished": 2021, "in_flight": 256, "gen_tokens": 4648414, "tok_per_s": 3368, "chain_tokens_median": 1222.0, "chain_tokens_p90": 5258.0, "chain_tokens_max": 31330, "parse_rate": 0.9975259772389906, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:30:37", "elapsed_min": 24.0, "submitted": 2354, "finished": 2098, "in_flight": 256, "gen_tokens": 4840450, "tok_per_s": 3361, "chain_tokens_median": 1213.0, "chain_tokens_p90": 5244.6, "chain_tokens_max": 32107, "parse_rate": 0.9976167778836987, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:31:37", "elapsed_min": 25.0, "submitted": 2429, "finished": 2173, "in_flight": 256, "gen_tokens": 5030339, "tok_per_s": 3353, "chain_tokens_median": 1218.0, "chain_tokens_p90": 5329.0, "chain_tokens_max": 32107, "parse_rate": 0.9976990335941095, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:32:37", "elapsed_min": 26.0, "submitted": 2506, "finished": 2250, "in_flight": 256, "gen_tokens": 5174005, "tok_per_s": 3317, "chain_tokens_median": 1192.0, "chain_tokens_p90": 5330.9000000000015, "chain_tokens_max": 32107, "parse_rate": 0.9977777777777778, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:33:37", "elapsed_min": 27.0, "submitted": 2577, "finished": 2321, "in_flight": 256, "gen_tokens": 5404466, "tok_per_s": 3336, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5355.0, "chain_tokens_max": 32654, "parse_rate": 0.996984058595433, "looped": 7, "minutes_to_deadline": Infinity}
progress {"time": "01:34:37", "elapsed_min": 28.0, "submitted": 2680, "finished": 2424, "in_flight": 256, "gen_tokens": 5641858, "tok_per_s": 3358, "chain_tokens_median": 1182.5, "chain_tokens_p90": 5377.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9966996699669967, "looped": 8, "minutes_to_deadline": Infinity}
progress {"time": "01:35:37", "elapsed_min": 29.0, "submitted": 2736, "finished": 2480, "in_flight": 256, "gen_tokens": 5821732, "tok_per_s": 3346, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5557.1, "chain_tokens_max": 32654, "parse_rate": 0.9967741935483871, "looped": 8, "minutes_to_deadline": Infinity}
progress {"time": "01:36:37", "elapsed_min": 30.0, "submitted": 2801, "finished": 2545, "in_flight": 256, "gen_tokens": 6015310, "tok_per_s": 3342, "chain_tokens_median": 1205.0, "chain_tokens_p90": 5561.599999999999, "chain_tokens_max": 32654, "parse_rate": 0.9964636542239685, "looped": 9, "minutes_to_deadline": Infinity}
progress {"time": "01:37:37", "elapsed_min": 31.0, "submitted": 2845, "finished": 2589, "in_flight": 256, "gen_tokens": 6159155, "tok_per_s": 3311, "chain_tokens_median": 1218.0, "chain_tokens_p90": 5564.400000000001, "chain_tokens_max": 32654, "parse_rate": 0.9961375048281189, "looped": 10, "minutes_to_deadline": Infinity}
progress {"time": "01:38:37", "elapsed_min": 32.0, "submitted": 2916, "finished": 2660, "in_flight": 256, "gen_tokens": 6339467, "tok_per_s": 3302, "chain_tokens_median": 1205.5, "chain_tokens_p90": 5579.0, "chain_tokens_max": 32654, "parse_rate": 0.9958646616541353, "looped": 11, "minutes_to_deadline": Infinity}
progress {"time": "01:39:37", "elapsed_min": 33.0, "submitted": 3019, "finished": 2763, "in_flight": 256, "gen_tokens": 6580922, "tok_per_s": 3324, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5583.000000000001, "chain_tokens_max": 32654, "parse_rate": 0.995656894679696, "looped": 12, "minutes_to_deadline": Infinity}
progress {"time": "01:40:37", "elapsed_min": 34.0, "submitted": 3205, "finished": 2949, "in_flight": 256, "gen_tokens": 6837377, "tok_per_s": 3352, "chain_tokens_median": 1131.0, "chain_tokens_p90": 5557.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.9959308240081384, "looped": 12, "minutes_to_deadline": Infinity}
progress {"time": "01:41:37", "elapsed_min": 35.0, "submitted": 3340, "finished": 3084, "in_flight": 256, "gen_tokens": 7119072, "tok_per_s": 3390, "chain_tokens_median": 1097.5, "chain_tokens_p90": 5564.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9954604409857328, "looped": 14, "minutes_to_deadline": Infinity}
progress {"time": "01:42:37", "elapsed_min": 36.0, "submitted": 3431, "finished": 3175, "in_flight": 256, "gen_tokens": 7307967, "tok_per_s": 3383, "chain_tokens_median": 1098.0, "chain_tokens_p90": 5533.6, "chain_tokens_max": 32654, "parse_rate": 0.9952755905511811, "looped": 15, "minutes_to_deadline": Infinity}
progress {"time": "01:43:37", "elapsed_min": 37.0, "submitted": 3506, "finished": 3250, "in_flight": 256, "gen_tokens": 7493006, "tok_per_s": 3375, "chain_tokens_median": 1120.5, "chain_tokens_p90": 5522.299999999999, "chain_tokens_max": 32654, "parse_rate": 0.9953846153846154, "looped": 15, "minutes_to_deadline": Infinity}
progress {"time": "01:44:37", "elapsed_min": 38.0, "submitted": 3552, "finished": 3296, "in_flight": 256, "gen_tokens": 7660652, "tok_per_s": 3360, "chain_tokens_median": 1134.5, "chain_tokens_p90": 5499.5, "chain_tokens_max": 32654, "parse_rate": 0.9951456310679612, "looped": 16, "minutes_to_deadline": Infinity}
progress {"time": "01:45:37", "elapsed_min": 39.0, "submitted": 3603, "finished": 3347, "in_flight": 256, "gen_tokens": 7912924, "tok_per_s": 3381, "chain_tokens_median": 1164.0, "chain_tokens_p90": 5543.200000000003, "chain_tokens_max": 32654, "parse_rate": 0.9946220495966537, "looped": 18, "minutes_to_deadline": Infinity}
progress {"time": "01:46:37", "elapsed_min": 40.0, "submitted": 3654, "finished": 3398, "in_flight": 256, "gen_tokens": 8138857, "tok_per_s": 3391, "chain_tokens_median": 1185.0, "chain_tokens_p90": 5565.3, "chain_tokens_max": 32654, "parse_rate": 0.994408475573867, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:47:37", "elapsed_min": 41.0, "submitted": 3712, "finished": 3456, "in_flight": 256, "gen_tokens": 8343448, "tok_per_s": 3391, "chain_tokens_median": 1203.0, "chain_tokens_p90": 5565.5, "chain_tokens_max": 32654, "parse_rate": 0.9945023148148148, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:48:37", "elapsed_min": 42.0, "submitted": 3762, "finished": 3506, "in_flight": 256, "gen_tokens": 8531646, "tok_per_s": 3385, "chain_tokens_median": 1218.5, "chain_tokens_p90": 5588.0, "chain_tokens_max": 32654, "parse_rate": 0.9945807187678266, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:49:37", "elapsed_min": 43.0, "submitted": 3815, "finished": 3559, "in_flight": 256, "gen_tokens": 8708509, "tok_per_s": 3375, "chain_tokens_median": 1233.0, "chain_tokens_p90": 5606.400000000001, "chain_tokens_max": 32654, "parse_rate": 0.9946614217476819, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:50:37", "elapsed_min": 44.0, "submitted": 3867, "finished": 3611, "in_flight": 256, "gen_tokens": 9012981, "tok_per_s": 3414, "chain_tokens_median": 1255.0, "chain_tokens_p90": 5655.0, "chain_tokens_max": 32654, "parse_rate": 0.993907504846303, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:51:37", "elapsed_min": 45.0, "submitted": 3908, "finished": 3652, "in_flight": 256, "gen_tokens": 9160768, "tok_per_s": 3393, "chain_tokens_median": 1267.5, "chain_tokens_p90": 5686.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9939759036144579, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:52:37", "elapsed_min": 46.0, "submitted": 3971, "finished": 3715, "in_flight": 256, "gen_tokens": 9411451, "tok_per_s": 3410, "chain_tokens_median": 1299.0, "chain_tokens_p90": 5724.4, "chain_tokens_max": 32654, "parse_rate": 0.9940780619111709, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:53:37", "elapsed_min": 47.0, "submitted": 4028, "finished": 3772, "in_flight": 256, "gen_tokens": 9620049, "tok_per_s": 3411, "chain_tokens_median": 1324.5, "chain_tokens_p90": 5735.5, "chain_tokens_max": 32654, "parse_rate": 0.9941675503711559, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:54:37", "elapsed_min": 48.0, "submitted": 4089, "finished": 3833, "in_flight": 256, "gen_tokens": 9871397, "tok_per_s": 3427, "chain_tokens_median": 1347.0, "chain_tokens_p90": 5780.800000000002, "chain_tokens_max": 32654, "parse_rate": 0.993999478215497, "looped": 23, "minutes_to_deadline": Infinity}
progress {"time": "01:55:37", "elapsed_min": 49.0, "submitted": 4148, "finished": 3892, "in_flight": 256, "gen_tokens": 10061683, "tok_per_s": 3422, "chain_tokens_median": 1359.5, "chain_tokens_p90": 5790.8, "chain_tokens_max": 32654, "parse_rate": 0.9940904419321686, "looped": 23, "minutes_to_deadline": Infinity}
progress {"time": "01:56:37", "elapsed_min": 50.0, "submitted": 4224, "finished": 3968, "in_flight": 256, "gen_tokens": 10315378, "tok_per_s": 3438, "chain_tokens_median": 1373.0, "chain_tokens_p90": 5775.300000000002, "chain_tokens_max": 32654, "parse_rate": 0.9939516129032258, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:57:37", "elapsed_min": 51.0, "submitted": 4298, "finished": 4042, "in_flight": 256, "gen_tokens": 10507533, "tok_per_s": 3434, "chain_tokens_median": 1381.0, "chain_tokens_p90": 5758.000000000001, "chain_tokens_max": 32654, "parse_rate": 0.9940623453735774, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:58:37", "elapsed_min": 52.0, "submitted": 4367, "finished": 4111, "in_flight": 256, "gen_tokens": 10696619, "tok_per_s": 3428, "chain_tokens_median": 1395.0, "chain_tokens_p90": 5749.0, "chain_tokens_max": 32654, "parse_rate": 0.9941620043784967, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:59:37", "elapsed_min": 53.0, "submitted": 4452, "finished": 4196, "in_flight": 256, "gen_tokens": 10923622, "tok_per_s": 3435, "chain_tokens_median": 1401.0, "chain_tokens_p90": 5736.5, "chain_tokens_max": 32654, "parse_rate": 0.994280266920877, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "02:00:37", "elapsed_min": 54.0, "submitted": 4525, "finished": 4269, "in_flight": 256, "gen_tokens": 11115102, "tok_per_s": 3430, "chain_tokens_median": 1404.0, "chain_tokens_p90": 5730.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.994378074490513, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "02:01:37", "elapsed_min": 55.0, "submitted": 4588, "finished": 4332, "in_flight": 256, "gen_tokens": 11295827, "tok_per_s": 3423, "chain_tokens_median": 1410.5, "chain_tokens_p90": 5721.9, "chain_tokens_max": 32654, "parse_rate": 0.9942289935364728, "looped": 25, "minutes_to_deadline": Infinity}
progress {"time": "02:02:37", "elapsed_min": 56.0, "submitted": 4666, "finished": 4410, "in_flight": 256, "gen_tokens": 11529493, "tok_per_s": 3431, "chain_tokens_median": 1421.5, "chain_tokens_p90": 5706.799999999999, "chain_tokens_max": 32654, "parse_rate": 0.9941043083900227, "looped": 26, "minutes_to_deadline": Infinity}
progress {"time": "02:03:37", "elapsed_min": 57.0, "submitted": 4731, "finished": 4475, "in_flight": 256, "gen_tokens": 11730013, "tok_per_s": 3430, "chain_tokens_median": 1441.0, "chain_tokens_p90": 5704.0, "chain_tokens_max": 32654, "parse_rate": 0.9939664804469274, "looped": 27, "minutes_to_deadline": Infinity}
progress {"time": "02:04:37", "elapsed_min": 58.0, "submitted": 4801, "finished": 4545, "in_flight": 256, "gen_tokens": 11922399, "tok_per_s": 3426, "chain_tokens_median": 1448.0, "chain_tokens_p90": 5700.6, "chain_tokens_max": 32654, "parse_rate": 0.994059405940594, "looped": 27, "minutes_to_deadline": Infinity}
progress {"time": "02:05:37", "elapsed_min": 59.0, "submitted": 4877, "finished": 4621, "in_flight": 256, "gen_tokens": 12171920, "tok_per_s": 3438, "chain_tokens_median": 1455.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9937243020991128, "looped": 29, "minutes_to_deadline": Infinity}
progress {"time": "02:06:37", "elapsed_min": 60.0, "submitted": 4942, "finished": 4686, "in_flight": 256, "gen_tokens": 12374714, "tok_per_s": 3437, "chain_tokens_median": 1468.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9935979513444302, "looped": 30, "minutes_to_deadline": Infinity}
progress {"time": "02:07:37", "elapsed_min": 61.0, "submitted": 5007, "finished": 4751, "in_flight": 256, "gen_tokens": 12604284, "tok_per_s": 3444, "chain_tokens_median": 1483.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9932645758787624, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:08:37", "elapsed_min": 62.0, "submitted": 5081, "finished": 4825, "in_flight": 256, "gen_tokens": 12808839, "tok_per_s": 3443, "chain_tokens_median": 1499.0, "chain_tokens_p90": 5682.800000000005, "chain_tokens_max": 32654, "parse_rate": 0.9933678756476684, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:09:37", "elapsed_min": 63.0, "submitted": 5153, "finished": 4897, "in_flight": 256, "gen_tokens": 13030924, "tok_per_s": 3447, "chain_tokens_median": 1514.0, "chain_tokens_p90": 5690.000000000003, "chain_tokens_max": 32654, "parse_rate": 0.9934653869716152, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:10:37", "elapsed_min": 64.0, "submitted": 5209, "finished": 4953, "in_flight": 256, "gen_tokens": 13176545, "tok_per_s": 3431, "chain_tokens_median": 1515.0, "chain_tokens_p90": 5685.400000000002, "chain_tokens_max": 32654, "parse_rate": 0.9935392691298203, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:11:37", "elapsed_min": 65.0, "submitted": 5276, "finished": 5020, "in_flight": 256, "gen_tokens": 13376782, "tok_per_s": 3430, "chain_tokens_median": 1530.5, "chain_tokens_p90": 5676.300000000005, "chain_tokens_max": 32654, "parse_rate": 0.9936254980079682, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:12:37", "elapsed_min": 66.0, "submitted": 5338, "finished": 5082, "in_flight": 256, "gen_tokens": 13567484, "tok_per_s": 3426, "chain_tokens_median": 1538.0, "chain_tokens_p90": 5674.800000000001, "chain_tokens_max": 32654, "parse_rate": 0.9937032664305392, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:13:37", "elapsed_min": 67.0, "submitted": 5391, "finished": 5135, "in_flight": 256, "gen_tokens": 13739879, "tok_per_s": 3418, "chain_tokens_median": 1546.0, "chain_tokens_p90": 5674.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.9935735150925025, "looped": 33, "minutes_to_deadline": Infinity}
progress {"time": "02:14:37", "elapsed_min": 68.0, "submitted": 5451, "finished": 5195, "in_flight": 256, "gen_tokens": 13950313, "tok_per_s": 3419, "chain_tokens_median": 1560.0, "chain_tokens_p90": 5672.6, "chain_tokens_max": 32654, "parse_rate": 0.9934552454282964, "looped": 34, "minutes_to_deadline": Infinity}
progress {"time": "02:15:37", "elapsed_min": 69.0, "submitted": 5513, "finished": 5257, "in_flight": 256, "gen_tokens": 14156336, "tok_per_s": 3419, "chain_tokens_median": 1575.0, "chain_tokens_p90": 5680.200000000007, "chain_tokens_max": 32654, "parse_rate": 0.9935324329465475, "looped": 34, "minutes_to_deadline": Infinity}
progress {"time": "02:16:37", "elapsed_min": 70.0, "submitted": 5570, "finished": 5314, "in_flight": 256, "gen_tokens": 14364565, "tok_per_s": 3420, "chain_tokens_median": 1581.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9932254422280767, "looped": 36, "minutes_to_deadline": Infinity}
progress {"time": "02:17:37", "elapsed_min": 71.0, "submitted": 5636, "finished": 5380, "in_flight": 256, "gen_tokens": 14593072, "tok_per_s": 3425, "chain_tokens_median": 1586.0, "chain_tokens_p90": 5676.300000000005, "chain_tokens_max": 32654, "parse_rate": 0.9931226765799257, "looped": 37, "minutes_to_deadline": Infinity}
progress {"time": "02:18:37", "elapsed_min": 72.0, "submitted": 5701, "finished": 5445, "in_flight": 256, "gen_tokens": 14830622, "tok_per_s": 3433, "chain_tokens_median": 1589.0, "chain_tokens_p90": 5680.400000000003, "chain_tokens_max": 32654, "parse_rate": 0.9928374655647383, "looped": 39, "minutes_to_deadline": Infinity}
progress {"time": "02:19:37", "elapsed_min": 73.0, "submitted": 5759, "finished": 5503, "in_flight": 256, "gen_tokens": 15008970, "tok_per_s": 3427, "chain_tokens_median": 1591.0, "chain_tokens_p90": 5672.8, "chain_tokens_max": 32654, "parse_rate": 0.9927312375068145, "looped": 40, "minutes_to_deadline": Infinity}
progress {"time": "02:20:37", "elapsed_min": 74.0, "submitted": 5832, "finished": 5576, "in_flight": 256, "gen_tokens": 15200373, "tok_per_s": 3423, "chain_tokens_median": 1590.5, "chain_tokens_p90": 5668.0, "chain_tokens_max": 32654, "parse_rate": 0.9926470588235294, "looped": 41, "minutes_to_deadline": Infinity}
progress {"time": "02:21:37", "elapsed_min": 75.0, "submitted": 5904, "finished": 5648, "in_flight": 256, "gen_tokens": 15428462, "tok_per_s": 3428, "chain_tokens_median": 1592.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9925637393767706, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:22:37", "elapsed_min": 76.0, "submitted": 5957, "finished": 5712, "in_flight": 245, "gen_tokens": 15584320, "tok_per_s": 3417, "chain_tokens_median": 1591.5, "chain_tokens_p90": 5687.600000000002, "chain_tokens_max": 32654, "parse_rate": 0.9926470588235294, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:23:37", "elapsed_min": 77.0, "submitted": 5957, "finished": 5783, "in_flight": 174, "gen_tokens": 15739461, "tok_per_s": 3407, "chain_tokens_median": 1587.0, "chain_tokens_p90": 5674.6, "chain_tokens_max": 32654, "parse_rate": 0.9927373335638942, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:24:37", "elapsed_min": 78.0, "submitted": 5957, "finished": 5841, "in_flight": 116, "gen_tokens": 15895616, "tok_per_s": 3396, "chain_tokens_median": 1586.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9928094504365691, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:25:37", "elapsed_min": 79.0, "submitted": 5957, "finished": 5903, "in_flight": 54, "gen_tokens": 16243669, "tok_per_s": 3427, "chain_tokens_median": 1601.0, "chain_tokens_p90": 5721.8, "chain_tokens_max": 32654, "parse_rate": 0.9923767575808911, "looped": 45, "minutes_to_deadline": Infinity}
progress {"time": "02:26:37", "elapsed_min": 80.0, "submitted": 5957, "finished": 5938, "in_flight": 19, "gen_tokens": 16513426, "tok_per_s": 3440, "chain_tokens_median": 1615.0, "chain_tokens_p90": 5749.6, "chain_tokens_max": 32654, "parse_rate": 0.9920848770629842, "looped": 47, "minutes_to_deadline": Infinity}
progress {"time": "02:27:37", "elapsed_min": 81.0, "submitted": 5957, "finished": 5948, "in_flight": 9, "gen_tokens": 16651513, "tok_per_s": 3426, "chain_tokens_median": 1619.0, "chain_tokens_p90": 5769.900000000001, "chain_tokens_max": 32654, "parse_rate": 0.9917619367854741, "looped": 49, "minutes_to_deadline": Infinity}
progress {"time": "02:28:37", "elapsed_min": 82.0, "submitted": 5957, "finished": 5950, "in_flight": 7, "gen_tokens": 16716368, "tok_per_s": 3397, "chain_tokens_median": 1621.5, "chain_tokens_p90": 5777.500000000002, "chain_tokens_max": 32654, "parse_rate": 0.9914285714285714, "looped": 51, "minutes_to_deadline": Infinity}
progress {"time": "02:29:37", "elapsed_min": 83.0, "submitted": 5957, "finished": 5952, "in_flight": 5, "gen_tokens": 16780064, "tok_per_s": 3369, "chain_tokens_median": 1622.5, "chain_tokens_p90": 5782.900000000001, "chain_tokens_max": 32654, "parse_rate": 0.9910954301075269, "looped": 53, "minutes_to_deadline": Infinity}
progress {"time": "02:30:37", "elapsed_min": 84.0, "submitted": 5957, "finished": 5953, "in_flight": 4, "gen_tokens": 16812277, "tok_per_s": 3336, "chain_tokens_median": 1623.0, "chain_tokens_p90": 5784.6, "chain_tokens_max": 32654, "parse_rate": 0.9909289433898875, "looped": 54, "minutes_to_deadline": Infinity}
reason base: {"submitted": 5957, "finished": 5957, "gen_tokens": 16942043, "t0": 1793408797.1839204, "seconds": 5098.851123094559, "tok_per_s": 3322.717724246409}
reason base: 5957 items, post-readout in 5890s total
latency base: {"readout_ms": {"median": 42.97343268990517, "p90": 76.37803927063942, "n": 100}, "reasoning_ms": {"median": 10051.338203251362, "p90": 41781.982911750674, "n": 20}}
(MainProcess pid=27742) INFO 10-31 02:50:45 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=27742) INFO 10-31 02:50:45 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=27742) INFO 10-31 02:50:45 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:777] [shutdown] MPClient: complete

Xet Storage Details

Size:
52.9 kB
·
Xet hash:
7fbdd422a2710d290fda138f9189f522baeffb10c704a4eacec4fad6e147f20c

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.