Hugging Face's logo

Buckets:
Mercity
/
Desicion-Model-Custom

Files
xet
Mercity/Desicion-Model-Custom / run2_v2 /logs /rl_eval_driver.log
Pranav2748's picture
Pranav2748
download
raw
35.3 kB
{"time": "2026-10-31T22:47:41Z", "event": "start", "stage": "merge_rl60"}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s] Loading weights: 86%|████████▌ | 367/426 [00:00<00:00, 3667.55it/s] Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3636.52it/s]
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s] Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.98s/it] Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.98s/it]
merged checkpoints/rl_grpo_b/final into checkpoints/rl60_merged
{"time": "2026-10-31T22:48:51Z", "event": "done", "stage": "merge_rl60"}
{"time": "2026-10-31T22:49:45Z", "event": "start", "stage": "eval_rl60"}
(MainProcess pid=126953) INFO 10-31 22:49:57 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=126953) INFO 10-31 22:49:57 [model.py:2152] Using max model len 32768
(MainProcess pid=126953) INFO 10-31 22:49:57 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=126953) INFO 10-31 22:49:57 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=126953) INFO 10-31 22:49:57 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=127220) INFO 10-31 22:50:03 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/rl60_merged', speculative_config=None, tokenizer='checkpoints/rl60_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/rl60_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=127220) INFO 10-31 22:50:04 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_5a58670850414e1fbeb0ffb9b96a0a87 backend=nccl
(EngineCore pid=127220) INFO 10-31 22:50:04 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=127220) INFO 10-31 22:50:04 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=127220) INFO 10-31 22:50:06 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=127220) INFO 10-31 22:50:06 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=127220) INFO 10-31 22:50:07 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=127220) INFO 10-31 22:50:07 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=127220) INFO 10-31 22:50:07 [base_loader.py:82] Initializing model took 1.517089 seconds
(EngineCore pid=127220) INFO 10-31 22:50:07 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 135.46 GiB.
(EngineCore pid=127220) INFO 10-31 22:50:07 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.07s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.07s/it]
(EngineCore pid=127220) INFO 10-31 22:50:08 [default_loader.py:501] Loading weights took 1.10 seconds
(EngineCore pid=127220) INFO 10-31 22:50:10 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.405286 seconds
(EngineCore pid=127220) INFO 10-31 22:50:10 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=127220) INFO 10-31 22:50:10 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=127220) INFO 10-31 22:50:10 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=127220) INFO 10-31 22:50:10 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=127220) INFO 10-31 22:50:10 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=127220) INFO 10-31 22:50:11 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=127220) INFO 10-31 22:50:16 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/0719b826c0/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=127220) INFO 10-31 22:50:16 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.24 s
(EngineCore pid=127220) INFO 10-31 22:50:33 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.40 s
(EngineCore pid=127220) INFO 10-31 22:50:36 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20434246 bytes total
(EngineCore pid=127220) INFO 10-31 22:50:36 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a7572ca54f1d0b6fe8b500182cab130a44ae83ccdf4112c7dccda517142dedde/rank_0_0/model
(EngineCore pid=127220) INFO 10-31 22:50:36 [monitor.py:62] [backbone] torch.compile took 25.65 s in total
(EngineCore pid=127220) INFO 10-31 22:52:04 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.67 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:28, 1.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:01<00:24, 1.36it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:16, 1.96it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:12, 2.47it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:02<00:10, 2.88it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:09, 3.20it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:07, 3.53it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:07, 3.79it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:07, 3.57it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:06, 3.62it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.66it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:06, 3.81it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 3.96it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:04, 4.20it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.82it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 5.51it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:05<00:07, 2.26it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:05<00:06, 2.69it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:06<00:03, 4.47it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:06<00:02, 6.47it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 25/35 [00:06<00:01, 9.49it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:06<00:00, 12.11it/s] Capturing CUDA graphs (PIECEWISE): 89%|████████▊ | 31/35 [00:06<00:00, 14.34it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:06<00:00, 16.13it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 5.03it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.22it/s]
(EngineCore pid=127220) INFO 10-31 22:52:14 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.30 GiB
(EngineCore pid=127220) INFO 10-31 22:52:15 [gpu_worker.py:759] Available KV cache memory: 59.38 GiB
(EngineCore pid=127220) INFO 10-31 22:52:15 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=127220) INFO 10-31 22:52:15 [kv_cache_utils.py:2327] GPU KV cache size: 1,749,526 tokens, Maximum concurrency for 32,768 tokens per request: 53.39x
(EngineCore pid=127220) INFO 10-31 22:52:15 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=127220) INFO 10-31 22:52:15 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=127220) INFO 10-31 22:52:15 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=127220) INFO 10-31 22:52:15 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=127220) INFO 10-31 22:52:15 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=127220) INFO 10-31 22:52:16 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:10, 3.24it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.74it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:11, 2.67it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.72it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.88it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:09, 2.98it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:08, 3.21it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:07, 3.46it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:02<00:07, 3.42it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.46it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.65it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:05, 3.99it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:03<00:05, 4.28it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:03<00:04, 4.72it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:03, 5.23it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 5.88it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:01, 8.70it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 11.21it/s] Capturing CUDA graphs (PIECEWISE): 66%|██████▌ | 23/35 [00:04<00:00, 14.27it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:04<00:00, 16.28it/s] Capturing CUDA graphs (PIECEWISE): 83%|████████▎ | 29/35 [00:04<00:00, 17.64it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 18.58it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 19.11it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 6.80it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 26.68it/s] Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 27.10it/s] Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 27.48it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 26.42it/s] Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 26.88it/s] Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 26.68it/s] Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 26.80it/s]
(EngineCore pid=127220) INFO 10-31 22:52:42 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.16 GiB
(EngineCore pid=127220) INFO 10-31 22:52:42 [gpu_worker.py:971] CUDA graph pool memory: 0.16 GiB (actual), 0.32 GiB (estimated), difference: 0.16 GiB (103.8%).
(EngineCore pid=127220) INFO 10-31 22:52:42 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.39 GiB for consumed memory (weights + non-torch), 1.55 GiB for peak activation, and 0.16 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63430039962` (59.07 GiB) to fit into requested memory, or `--kv-cache-memory=71496105984` (66.59 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.38 GiB.
(EngineCore pid=127220) INFO 10-31 22:52:42 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=127220) INFO 10-31 22:52:43 [core.py:395] init engine (profile, create kv cache, warmup model) took 153.02 s (compilation: 25.65 s)
(EngineCore pid=127220) INFO 10-31 22:52:43 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=126953) WARNING 10-31 22:52:44 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=127220) WARNING 10-31 22:52:44 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout rl60: 5957 items in 114s
(EngineCore pid=127220) WARNING 10-31 22:54:38 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=127220) WARNING 10-31 22:54:38 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=127220) WARNING 10-31 22:54:39 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "22:55:37", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 22:55:37 [loggers.py:335] Engine 000: Avg prompt throughput: 16270.9 tokens/s, Avg generation throughput: 1427.6 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0%
progress {"time": "22:56:37", "elapsed_min": 2.0, "submitted": 392, "finished": 136, "in_flight": 256, "gen_tokens": 132360, "tok_per_s": 1103, "chain_tokens_median": 903.5, "chain_tokens_p90": 1504.5, "chain_tokens_max": 2206, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 22:56:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4123.3 tokens/s, Avg generation throughput: 3889.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 6.6%
progress {"time": "22:57:37", "elapsed_min": 3.0, "submitted": 526, "finished": 270, "in_flight": 256, "gen_tokens": 302900, "tok_per_s": 1683, "chain_tokens_median": 910.5, "chain_tokens_p90": 2107.7, "chain_tokens_max": 4004, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 22:57:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4057.3 tokens/s, Avg generation throughput: 3727.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 11.4%
progress {"time": "22:58:37", "elapsed_min": 4.0, "submitted": 655, "finished": 399, "in_flight": 256, "gen_tokens": 493623, "tok_per_s": 2057, "chain_tokens_median": 884.0, "chain_tokens_p90": 2601.2, "chain_tokens_max": 5745, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 22:58:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4747.4 tokens/s, Avg generation throughput: 3447.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 26.5%, Prefix cache hit rate: 10.8%
progress {"time": "22:59:37", "elapsed_min": 5.0, "submitted": 793, "finished": 537, "in_flight": 256, "gen_tokens": 741297, "tok_per_s": 2471, "chain_tokens_median": 905.0, "chain_tokens_p90": 3041.0000000000023, "chain_tokens_max": 7140, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 22:59:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5887.9 tokens/s, Avg generation throughput: 3272.9 tokens/s, Running: 127 reqs, Waiting: 127 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 10.4%
progress {"time": "23:00:37", "elapsed_min": 6.0, "submitted": 945, "finished": 689, "in_flight": 256, "gen_tokens": 949779, "tok_per_s": 2638, "chain_tokens_median": 828.0, "chain_tokens_p90": 3140.4000000000005, "chain_tokens_max": 8120, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:00:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5173.5 tokens/s, Avg generation throughput: 3341.1 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 5.3%
progress {"time": "23:01:37", "elapsed_min": 7.0, "submitted": 1170, "finished": 914, "in_flight": 256, "gen_tokens": 1263603, "tok_per_s": 3008, "chain_tokens_median": 803.5, "chain_tokens_p90": 3239.8, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:01:37 [loggers.py:335] Engine 000: Avg prompt throughput: 7190.9 tokens/s, Avg generation throughput: 2710.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 3.4%
progress {"time": "23:02:37", "elapsed_min": 8.0, "submitted": 1190, "finished": 934, "in_flight": 256, "gen_tokens": 1285595, "tok_per_s": 2678, "chain_tokens_median": 800.0, "chain_tokens_p90": 3230.400000000001, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:02:37 [loggers.py:335] Engine 000: Avg prompt throughput: 937.0 tokens/s, Avg generation throughput: 4381.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 3.5%
progress {"time": "23:03:37", "elapsed_min": 9.0, "submitted": 1314, "finished": 1058, "in_flight": 256, "gen_tokens": 1521678, "tok_per_s": 2818, "chain_tokens_median": 850.0, "chain_tokens_p90": 3287.0000000000014, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:03:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5137.2 tokens/s, Avg generation throughput: 3053.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 2.5%
progress {"time": "23:04:37", "elapsed_min": 10.0, "submitted": 1431, "finished": 1175, "in_flight": 256, "gen_tokens": 1763485, "tok_per_s": 2939, "chain_tokens_median": 902.0, "chain_tokens_p90": 3470.4000000000024, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:04:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5369.5 tokens/s, Avg generation throughput: 2950.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 2.5%
progress {"time": "23:05:37", "elapsed_min": 11.0, "submitted": 1460, "finished": 1204, "in_flight": 256, "gen_tokens": 1818397, "tok_per_s": 2755, "chain_tokens_median": 905.5, "chain_tokens_p90": 3481.8, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:05:37 [loggers.py:335] Engine 000: Avg prompt throughput: 1894.1 tokens/s, Avg generation throughput: 3817.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 2.1%
progress {"time": "23:06:37", "elapsed_min": 12.0, "submitted": 1578, "finished": 1322, "in_flight": 256, "gen_tokens": 2087332, "tok_per_s": 2899, "chain_tokens_median": 980.0, "chain_tokens_p90": 3743.7000000000003, "chain_tokens_max": 8764, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:06:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5830.6 tokens/s, Avg generation throughput: 2883.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 1.8%
progress {"time": "23:07:37", "elapsed_min": 13.0, "submitted": 1696, "finished": 1440, "in_flight": 256, "gen_tokens": 2341655, "tok_per_s": 3002, "chain_tokens_median": 1021.5, "chain_tokens_p90": 3762.4000000000005, "chain_tokens_max": 10846, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:07:37 [loggers.py:335] Engine 000: Avg prompt throughput: 5852.2 tokens/s, Avg generation throughput: 2722.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 29.5%, Prefix cache hit rate: 1.3%
progress {"time": "23:08:37", "elapsed_min": 14.0, "submitted": 1743, "finished": 1487, "in_flight": 256, "gen_tokens": 2430232, "tok_per_s": 2893, "chain_tokens_median": 1037.0, "chain_tokens_p90": 3763.6000000000004, "chain_tokens_max": 10846, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:08:37 [loggers.py:335] Engine 000: Avg prompt throughput: 3225.5 tokens/s, Avg generation throughput: 3194.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 1.4%
progress {"time": "23:09:37", "elapsed_min": 15.0, "submitted": 1846, "finished": 1590, "in_flight": 256, "gen_tokens": 2638437, "tok_per_s": 2931, "chain_tokens_median": 1053.5, "chain_tokens_p90": 3789.0, "chain_tokens_max": 10846, "parse_rate": 0.9993710691823899, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:09:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4759.3 tokens/s, Avg generation throughput: 2871.8 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 31.1%, Prefix cache hit rate: 3.2%
progress {"time": "23:10:37", "elapsed_min": 16.0, "submitted": 1969, "finished": 1713, "in_flight": 256, "gen_tokens": 2797823, "tok_per_s": 2914, "chain_tokens_median": 1029.0, "chain_tokens_p90": 3759.9999999999995, "chain_tokens_max": 10846, "parse_rate": 0.9994162288382954, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:10:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4231.7 tokens/s, Avg generation throughput: 2793.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 31.1%, Prefix cache hit rate: 4.3%
progress {"time": "23:11:37", "elapsed_min": 17.0, "submitted": 2013, "finished": 1836, "in_flight": 177, "gen_tokens": 2999944, "tok_per_s": 2941, "chain_tokens_median": 1015.5, "chain_tokens_p90": 3776.0, "chain_tokens_max": 10846, "parse_rate": 0.9989106753812637, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:11:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4488.6 tokens/s, Avg generation throughput: 2737.6 tokens/s, Running: 128 reqs, Waiting: 49 reqs, GPU KV cache usage: 30.5%, Prefix cache hit rate: 5.8%
progress {"time": "23:12:37", "elapsed_min": 18.0, "submitted": 2013, "finished": 1957, "in_flight": 56, "gen_tokens": 3241142, "tok_per_s": 3001, "chain_tokens_median": 1032.0, "chain_tokens_p90": 3797.4, "chain_tokens_max": 10846, "parse_rate": 0.9989780275932549, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:12:37 [loggers.py:335] Engine 000: Avg prompt throughput: 4889.0 tokens/s, Avg generation throughput: 2287.3 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 6.3%
progress {"time": "23:13:37", "elapsed_min": 19.0, "submitted": 2013, "finished": 2000, "in_flight": 13, "gen_tokens": 3416982, "tok_per_s": 2997, "chain_tokens_median": 1064.5, "chain_tokens_p90": 3967.2000000000044, "chain_tokens_max": 10846, "parse_rate": 0.999, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:13:37 [loggers.py:335] Engine 000: Avg prompt throughput: 3203.9 tokens/s, Avg generation throughput: 1506.4 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 6.0%
progress {"time": "23:14:37", "elapsed_min": 20.0, "submitted": 2013, "finished": 2009, "in_flight": 4, "gen_tokens": 3559923, "tok_per_s": 2966, "chain_tokens_median": 1071.0, "chain_tokens_p90": 4023.2, "chain_tokens_max": 31374, "parse_rate": 0.9975111996017919, "looped": 3, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:14:37 [loggers.py:335] Engine 000: Avg prompt throughput: 907.1 tokens/s, Avg generation throughput: 615.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 5.9%
progress {"time": "23:15:37", "elapsed_min": 21.0, "submitted": 2013, "finished": 2010, "in_flight": 3, "gen_tokens": 3577135, "tok_per_s": 2839, "chain_tokens_median": 1072.5, "chain_tokens_p90": 4025.300000000002, "chain_tokens_max": 31374, "parse_rate": 0.9975124378109452, "looped": 3, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:15:37 [loggers.py:335] Engine 000: Avg prompt throughput: 313.0 tokens/s, Avg generation throughput: 371.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 5.9%
progress {"time": "23:16:37", "elapsed_min": 22.0, "submitted": 2013, "finished": 2010, "in_flight": 3, "gen_tokens": 3577135, "tok_per_s": 2710, "chain_tokens_median": 1072.5, "chain_tokens_p90": 4025.300000000002, "chain_tokens_max": 31374, "parse_rate": 0.9975124378109452, "looped": 3, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:16:37 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 320.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 5.9%
progress {"time": "23:17:37", "elapsed_min": 23.0, "submitted": 2013, "finished": 2011, "in_flight": 2, "gen_tokens": 3608083, "tok_per_s": 2614, "chain_tokens_median": 1074.0, "chain_tokens_p90": 4037.0, "chain_tokens_max": 31374, "parse_rate": 0.9970164097463948, "looped": 4, "minutes_to_deadline": Infinity}
(MainProcess pid=126953) INFO 10-31 23:17:37 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 275.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 5.9%
reason rl60: {"submitted": 2013, "finished": 2013, "gen_tokens": 3673057, "t0": 1793487277.3236818, "seconds": 1394.017959356308, "tok_per_s": 2634.8706452075016}
reason rl60: post-chain readout 2007 taken at generation, 0 by prefill
reason rl60: 2013 items, post-readout in 1394s total
latency rl60: {"readout_ms": {"median": 42.5105094909668, "p90": 75.57755559682846, "n": 100}, "reasoning_ms": {"median": 7711.183171719313, "p90": 16987.101519852888, "n": 20}}
(MainProcess pid=126953) INFO 10-31 23:21:16 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=126953) INFO 10-31 23:21:16 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=126953) INFO 10-31 23:21:16 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=127220) INFO 10-31 23:21:16 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=127220) INFO 10-31 23:21:16 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=127220) INFO 10-31 23:21:16 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=127220) INFO 10-31 23:21:16 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=126953) INFO 10-31 23:21:19 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=126953) INFO 10-31 23:21:19 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=126953) INFO 10-31 23:21:19 [core_client.py:777] [shutdown] MPClient: complete
{"time": "2026-10-31T23:21:21Z", "event": "done", "stage": "eval_rl60"}
{"time": "2026-10-31T23:22:05Z", "event": "start", "stage": "metrics_rl60"}
readout base all 0.627 OD Cap 61.8 ECE 0.036 NLL 0.861 targets {'readout_acc_all': False, 'od_capability': False, 'agree_all': False, 'post_ece': True, 'reasoning_vs_base': True, 'loops_vs_base': True}
readout v1_gold all 0.698 OD Cap 66.0 ECE 0.019 NLL 0.731 targets {'readout_acc_all': False, 'od_capability': False}
readout v1_joint all 0.695 OD Cap 66.7 ECE 0.030 NLL 0.715 targets {'readout_acc_all': False, 'od_capability': False, 'agree_all': False, 'post_ece': True, 'reasoning_vs_base': True, 'loops_vs_base': False}
readout v2_joint all 0.727 OD Cap 67.3 ECE 0.033 NLL 0.669 targets {'readout_acc_all': True, 'od_capability': False, 'agree_all': True, 'post_ece': True, 'reasoning_vs_base': False, 'loops_vs_base': False}
readout rl60 all 0.725 OD Cap 66.8 ECE 0.030 NLL 0.672 targets {'readout_acc_all': False, 'od_capability': False, 'agree_all': False, 'post_ece': False, 'reasoning_vs_base': True, 'loops_vs_base': True}
reason base all 0.840 tokens 1578 loops 13 post-ECE 0.050 (raw 0.149)
reason v1_joint all 0.844 tokens 1323 loops 21 post-ECE 0.023 (raw 0.144)
reason v2_joint all 0.812 tokens 1340 loops 76 post-ECE 0.043 (raw 0.125)
reason rl60 all 0.831 tokens 1096 loops 3 post-ECE 0.057 (raw 0.151)
pair base agree 0.657 dis-AUROC 0.773 gap@0.3 0.50
pair v1_joint agree 0.757 dis-AUROC 0.780 gap@0.3 0.52
pair v2_joint agree 0.783 dis-AUROC 0.839 gap@0.3 0.70
pair rl60 agree 0.770 dis-AUROC 0.833 gap@0.3 0.79
{"time": "2026-10-31T23:22:12Z", "event": "done", "stage": "metrics_rl60"}
{"time": "2026-10-31T23:23:01Z", "event": "finished", "stage": "eval_rl60"}
DRIVER_EXIT 0

Xet Storage Details

Size:
35.3 kB
·
Xet hash:
aa4d3c18ea560916de802f63de7de38f1f9e14adf6fc9f1dce7803004cc007cf

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.