evals/grid_math_unhealed/glean_keep50_unhealed_chat.json.server.log

Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
WARNING 08-10 21:22:07 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299] 
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.0
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299]   █▄█▀ █     █     █     █  model   outputs/release/unhealed/winnow-olmoe-math-keep50
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:299] 
(APIServer pid=1147686) INFO 08-10 21:22:07 [utils.py:233] non-default args: {'model_tag': 'outputs/release/unhealed/winnow-olmoe-math-keep50', 'host': '127.0.0.1', 'port': 8610, 'model': 'outputs/release/unhealed/winnow-olmoe-math-keep50', 'max_model_len': 2048, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85}
(APIServer pid=1147686) INFO 08-10 21:22:15 [model.py:549] Resolved architecture: PrunedOlmoeForCausalLM
(APIServer pid=1147686) INFO 08-10 21:22:15 [model.py:1678] Using max model len 2048
(APIServer pid=1147686) INFO 08-10 21:22:15 [vllm.py:790] Asynchronous scheduling is enabled.
(APIServer pid=1147686) WARNING 08-10 21:22:15 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=1147686) WARNING 08-10 21:22:15 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1147686) INFO 08-10 21:22:15 [vllm.py:1025] Cudagraph is disabled under eager mode
(APIServer pid=1147686) INFO 08-10 21:22:15 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
(EngineCore pid=1148963) WARNING 08-10 21:22:24 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
(EngineCore pid=1148963) INFO 08-10 21:22:24 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='outputs/release/unhealed/winnow-olmoe-math-keep50', speculative_config=None, tokenizer='outputs/release/unhealed/winnow-olmoe-math-keep50', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=1148963) INFO 08-10 21:22:24 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.27:43405 backend=nccl
(EngineCore pid=1148963) INFO 08-10 21:22:24 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=1148963) INFO 08-10 21:22:25 [gpu_model_runner.py:4735] Starting to load model outputs/release/unhealed/winnow-olmoe-math-keep50...
(EngineCore pid=1148963) INFO 08-10 21:22:25 [cuda.py:334] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=1148963) INFO 08-10 21:22:25 [flash_attn.py:596] Using FlashAttention version 2
(EngineCore pid=1148963) 
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
(EngineCore pid=1148963) 
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.19it/s]
(EngineCore pid=1148963) 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.53it/s]
(EngineCore pid=1148963) 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.47it/s]
(EngineCore pid=1148963) 
(EngineCore pid=1148963) INFO 08-10 21:22:27 [default_loader.py:384] Loading weights took 1.42 seconds
(EngineCore pid=1148963) INFO 08-10 21:22:27 [gpu_model_runner.py:4820] Model loading took 6.89 GiB memory and 2.012000 seconds
(EngineCore pid=1148963) INFO 08-10 21:22:29 [gpu_worker.py:436] Available KV cache memory: 12.86 GiB
(EngineCore pid=1148963) INFO 08-10 21:22:29 [kv_cache_utils.py:1319] GPU KV cache size: 105,328 tokens
(EngineCore pid=1148963) INFO 08-10 21:22:29 [kv_cache_utils.py:1324] Maximum concurrency for 2,048 tokens per request: 51.43x
(EngineCore pid=1148963) INFO 08-10 21:22:29 [core.py:283] init engine (profile, create kv cache, warmup model) took 1.80 seconds
(EngineCore pid=1148963) INFO 08-10 21:22:30 [vllm.py:790] Asynchronous scheduling is enabled.
(EngineCore pid=1148963) WARNING 08-10 21:22:30 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(EngineCore pid=1148963) WARNING 08-10 21:22:30 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=1148963) INFO 08-10 21:22:30 [vllm.py:1025] Cudagraph is disabled under eager mode
(EngineCore pid=1148963) INFO 08-10 21:22:30 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1147686) INFO 08-10 21:22:30 [api_server.py:590] Supported tasks: ['generate']
(APIServer pid=1147686) WARNING 08-10 21:22:30 [__init__.py:14] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production!
(APIServer pid=1147686) INFO 08-10 21:22:30 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1147686) INFO 08-10 21:22:30 [api_server.py:594] Starting vLLM server on http://127.0.0.1:8610
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:37] Available routes are:
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /sleep, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /wake_up, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /is_sleeping, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /collective_rpc, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /reset_prefix_cache, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /reset_mm_cache, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /reset_encoder_cache, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /server_info, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /init_weight_transfer_engine, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /update_weights, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /get_world_size, Methods: GET
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1147686) INFO 08-10 21:22:30 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1147686) INFO:     Started server process [1147686]
(APIServer pid=1147686) INFO:     Waiting for application startup.
(APIServer pid=1147686) INFO:     Application startup complete.
(APIServer pid=1147686) INFO:     127.0.0.1:37708 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1147686) INFO 08-10 21:22:40 [loggers.py:259] Engine 000: Avg prompt throughput: 2662.8 tokens/s, Avg generation throughput: 2173.5 tokens/s, Running: 253 reqs, Waiting: 972 reqs, GPU KV cache usage: 38.1%, Prefix cache hit rate: 93.2%
(APIServer pid=1147686) INFO 08-10 21:22:50 [loggers.py:259] Engine 000: Avg prompt throughput: 2160.5 tokens/s, Avg generation throughput: 3146.4 tokens/s, Running: 252 reqs, Waiting: 696 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 93.3%
(APIServer pid=1147686) INFO 08-10 21:23:00 [loggers.py:259] Engine 000: Avg prompt throughput: 2245.8 tokens/s, Avg generation throughput: 3093.6 tokens/s, Running: 255 reqs, Waiting: 409 reqs, GPU KV cache usage: 38.2%, Prefix cache hit rate: 93.4%
(APIServer pid=1147686) INFO 08-10 21:23:10 [loggers.py:259] Engine 000: Avg prompt throughput: 2095.3 tokens/s, Avg generation throughput: 3147.7 tokens/s, Running: 253 reqs, Waiting: 149 reqs, GPU KV cache usage: 40.2%, Prefix cache hit rate: 93.4%
(APIServer pid=1147686) INFO 08-10 21:23:20 [loggers.py:259] Engine 000: Avg prompt throughput: 1232.5 tokens/s, Avg generation throughput: 3092.0 tokens/s, Running: 88 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 93.4%
(APIServer pid=1147686) INFO:     127.0.0.1:37722 - "POST /v1/completions HTTP/1.1" 200 OK
(EngineCore pid=1148963) INFO 08-10 21:23:27 [core.py:1210] Shutdown initiated (timeout=0)
(EngineCore pid=1148963) INFO 08-10 21:23:27 [core.py:1233] Shutdown complete
(APIServer pid=1147686) INFO:     Shutting down
(APIServer pid=1147686) INFO:     Waiting for application shutdown.
(APIServer pid=1147686) INFO:     Application shutdown complete.
(APIServer pid=1147686) INFO:     Finished server process [1147686]