| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339]
|
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/release/healed/release-offpolicy-math-keep25/step0050
|
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:339] |
| (APIServer pid=178906) INFO 08-09 15:37:21 [api_utils.py:273] non-default args: {'model_tag': 'outputs/release/healed/release-offpolicy-math-keep25/step0050', 'host': '127.0.0.1', 'port': 8471, 'model': 'outputs/release/healed/release-offpolicy-math-keep25/step0050', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85} |
| (APIServer pid=178906) INFO 08-09 15:37:21 [model.py:619] Resolved architecture: PrunedOlmoeForCausalLM |
| (APIServer pid=178906) INFO 08-09 15:37:21 [model.py:1776] Using max model len 4096 |
| (APIServer pid=178906) INFO 08-09 15:37:21 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=178906) WARNING 08-09 15:37:21 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=178906) WARNING 08-09 15:37:21 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=178906) INFO 08-09 15:37:21 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=178906) INFO 08-09 15:37:21 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=178906) INFO 08-09 15:37:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (EngineCore pid=179352) INFO 08-09 15:37:28 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/release/healed/release-offpolicy-math-keep25/step0050', speculative_config=None, tokenizer='outputs/release/healed/release-offpolicy-math-keep25/step0050', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=179352) INFO 08-09 15:37:30 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.27:36165 backend=nccl |
| (EngineCore pid=179352) INFO 08-09 15:37:30 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (EngineCore pid=179352) INFO 08-09 15:37:30 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (EngineCore pid=179352) INFO 08-09 15:37:30 [gpu_model_runner.py:5209] Starting to load model outputs/release/healed/release-offpolicy-math-keep25/step0050... |
| (EngineCore pid=179352) INFO 08-09 15:37:31 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (EngineCore pid=179352) INFO 08-09 15:37:31 [flash_attn.py:718] Using FlashAttention version 2 |
| (EngineCore pid=179352) INFO 08-09 15:37:31 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 3.89 GiB. Available RAM: 106.02 GiB. |
| (EngineCore pid=179352) INFO 08-09 15:37:31 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (EngineCore pid=179352)
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] |
| (EngineCore pid=179352)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00, 7.23s/it] |
| (EngineCore pid=179352)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00, 7.23s/it] |
| (EngineCore pid=179352) |
| (EngineCore pid=179352) INFO 08-09 15:37:38 [default_loader.py:430] Loading weights took 7.25 seconds |
| (EngineCore pid=179352) INFO 08-09 15:37:39 [gpu_model_runner.py:5306] Model loading took 3.89 GiB memory and 7.476853 seconds |
| (EngineCore pid=179352) INFO 08-09 15:37:40 [gpu_worker.py:538] Available KV cache memory: 15.86 GiB |
| (EngineCore pid=179352) INFO 08-09 15:37:40 [kv_cache_utils.py:2146] GPU KV cache size: 129,920 tokens |
| (EngineCore pid=179352) INFO 08-09 15:37:40 [kv_cache_utils.py:2147] Maximum concurrency for 4,096 tokens per request: 31.72x |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.31 s |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=179352) WARNING 08-09 15:37:41 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=179352) WARNING 08-09 15:37:41 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=179352) INFO 08-09 15:37:41 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=178906) INFO 08-09 15:37:41 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=178906) INFO 08-09 15:37:41 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=178906) INFO 08-09 15:37:41 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8471 |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:37] Available routes are: |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /docs, Methods: HEAD, GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /redoc, Methods: HEAD, GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=178906) INFO 08-09 15:37:41 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=178906) INFO: Started server process [178906] |
| (APIServer pid=178906) INFO: Waiting for application startup. |
| (APIServer pid=178906) INFO: Application startup complete. |
| (APIServer pid=178906) INFO: 127.0.0.1:43626 - "GET /health HTTP/1.1" 200 OK |
| (EngineCore pid=179352) WARNING 08-09 15:37:58 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _plan_tiles. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=179352) WARNING 08-09 15:37:58 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _gate_up_silu. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=179352) WARNING 08-09 15:37:58 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _down_proj. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (EngineCore pid=179352) WARNING 08-09 15:38:02 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _count_experts. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=179352) WARNING 08-09 15:38:02 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _scatter_routes. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=178906) INFO 08-09 15:38:02 [loggers.py:273] Engine 000: Avg prompt throughput: 396.2 tokens/s, Avg generation throughput: 1092.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:38:12 [loggers.py:273] Engine 000: Avg prompt throughput: 697.1 tokens/s, Avg generation throughput: 3270.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:38:22 [loggers.py:273] Engine 000: Avg prompt throughput: 674.6 tokens/s, Avg generation throughput: 3348.9 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:38:32 [loggers.py:273] Engine 000: Avg prompt throughput: 699.3 tokens/s, Avg generation throughput: 3279.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:38:42 [loggers.py:273] Engine 000: Avg prompt throughput: 631.2 tokens/s, Avg generation throughput: 3286.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:38:52 [loggers.py:273] Engine 000: Avg prompt throughput: 644.5 tokens/s, Avg generation throughput: 3233.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:02 [loggers.py:273] Engine 000: Avg prompt throughput: 666.1 tokens/s, Avg generation throughput: 3290.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:12 [loggers.py:273] Engine 000: Avg prompt throughput: 675.6 tokens/s, Avg generation throughput: 3352.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:22 [loggers.py:273] Engine 000: Avg prompt throughput: 754.2 tokens/s, Avg generation throughput: 3335.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:32 [loggers.py:273] Engine 000: Avg prompt throughput: 676.0 tokens/s, Avg generation throughput: 3347.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:42 [loggers.py:273] Engine 000: Avg prompt throughput: 670.5 tokens/s, Avg generation throughput: 3314.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:39:52 [loggers.py:273] Engine 000: Avg prompt throughput: 661.1 tokens/s, Avg generation throughput: 3310.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:02 [loggers.py:273] Engine 000: Avg prompt throughput: 645.1 tokens/s, Avg generation throughput: 3345.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:12 [loggers.py:273] Engine 000: Avg prompt throughput: 682.1 tokens/s, Avg generation throughput: 3357.1 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:22 [loggers.py:273] Engine 000: Avg prompt throughput: 737.0 tokens/s, Avg generation throughput: 3370.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:32 [loggers.py:273] Engine 000: Avg prompt throughput: 701.6 tokens/s, Avg generation throughput: 3367.4 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52748 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:42 [loggers.py:273] Engine 000: Avg prompt throughput: 235.7 tokens/s, Avg generation throughput: 2154.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:40:52 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 169.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:02 [loggers.py:273] Engine 000: Avg prompt throughput: 779.1 tokens/s, Avg generation throughput: 2524.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 41.6% |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:12 [loggers.py:273] Engine 000: Avg prompt throughput: 475.0 tokens/s, Avg generation throughput: 2629.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 52.7% |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:22 [loggers.py:273] Engine 000: Avg prompt throughput: 477.2 tokens/s, Avg generation throughput: 2582.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 59.4% |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:32 [loggers.py:273] Engine 000: Avg prompt throughput: 552.5 tokens/s, Avg generation throughput: 2604.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 65.2% |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:42 [loggers.py:273] Engine 000: Avg prompt throughput: 441.0 tokens/s, Avg generation throughput: 2557.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 69.3% |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:41:52 [loggers.py:273] Engine 000: Avg prompt throughput: 477.5 tokens/s, Avg generation throughput: 2567.1 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 72.5% |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:02 [loggers.py:273] Engine 000: Avg prompt throughput: 380.1 tokens/s, Avg generation throughput: 2511.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 74.7% |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:12 [loggers.py:273] Engine 000: Avg prompt throughput: 395.5 tokens/s, Avg generation throughput: 2514.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 76.8% |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:22 [loggers.py:273] Engine 000: Avg prompt throughput: 491.5 tokens/s, Avg generation throughput: 2657.7 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.5%, Prefix cache hit rate: 78.8% |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:32 [loggers.py:273] Engine 000: Avg prompt throughput: 59.3 tokens/s, Avg generation throughput: 2055.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 79.1% |
| (APIServer pid=178906) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:42 [loggers.py:273] Engine 000: Avg prompt throughput: 264.8 tokens/s, Avg generation throughput: 522.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 79.3% |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:42:52 [loggers.py:273] Engine 000: Avg prompt throughput: 200.2 tokens/s, Avg generation throughput: 3314.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 79.4% |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:02 [loggers.py:273] Engine 000: Avg prompt throughput: 270.8 tokens/s, Avg generation throughput: 3208.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 79.6% |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:12 [loggers.py:273] Engine 000: Avg prompt throughput: 239.4 tokens/s, Avg generation throughput: 3015.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 79.8% |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:22 [loggers.py:273] Engine 000: Avg prompt throughput: 271.6 tokens/s, Avg generation throughput: 3273.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 80.0% |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:32 [loggers.py:273] Engine 000: Avg prompt throughput: 262.8 tokens/s, Avg generation throughput: 3326.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 80.2% |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:42 [loggers.py:273] Engine 000: Avg prompt throughput: 276.3 tokens/s, Avg generation throughput: 3345.8 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 80.4% |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:43:52 [loggers.py:273] Engine 000: Avg prompt throughput: 281.1 tokens/s, Avg generation throughput: 3349.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 80.7% |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:02 [loggers.py:273] Engine 000: Avg prompt throughput: 302.6 tokens/s, Avg generation throughput: 3306.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 80.9% |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:12 [loggers.py:273] Engine 000: Avg prompt throughput: 238.1 tokens/s, Avg generation throughput: 3289.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 81.1% |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:22 [loggers.py:273] Engine 000: Avg prompt throughput: 316.7 tokens/s, Avg generation throughput: 3311.5 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 81.3% |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:32 [loggers.py:273] Engine 000: Avg prompt throughput: 264.0 tokens/s, Avg generation throughput: 3293.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 80.5% |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59926 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60150 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:42 [loggers.py:273] Engine 000: Avg prompt throughput: 46.1 tokens/s, Avg generation throughput: 2098.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 80.4% |
| (APIServer pid=178906) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:60322 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:44:52 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 152.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.4% |
| (APIServer pid=178906) INFO 08-09 15:45:02 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.4% |
| (APIServer pid=178906) INFO: 127.0.0.1:56492 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=178906) INFO: 127.0.0.1:56492 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=178906) INFO: 127.0.0.1:56492 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44030 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44034 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44062 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44354 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44062 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44204 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44354 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44010 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44310 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:45:32 [loggers.py:273] Engine 000: Avg prompt throughput: 1837.0 tokens/s, Avg generation throughput: 548.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 73.6% |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44294 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44294 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44342 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44034 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44038 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44034 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44030 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44034 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44416 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44254 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44342 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44310 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44038 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44010 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44204 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44030 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44424 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44062 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44324 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44354 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44026 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:45:42 [loggers.py:273] Engine 000: Avg prompt throughput: 487.8 tokens/s, Avg generation throughput: 1932.8 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 70.3% |
| (APIServer pid=178906) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44400 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44292 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44388 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44410 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44294 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:44034 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:45:52 [loggers.py:273] Engine 000: Avg prompt throughput: 2045.7 tokens/s, Avg generation throughput: 1087.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.8%, Prefix cache hit rate: 66.1% |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:46:02 [loggers.py:273] Engine 000: Avg prompt throughput: 2754.4 tokens/s, Avg generation throughput: 2956.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 67.6% |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43300 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43404 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43466 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43212 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43270 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43482 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43418 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43322 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43510 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43306 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43370 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43170 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43144 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43540 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43508 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43132 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43362 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO: 127.0.0.1:43346 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=178906) INFO 08-09 15:46:12 [loggers.py:273] Engine 000: Avg prompt throughput: 748.2 tokens/s, Avg generation throughput: 1585.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=178906) INFO 08-09 15:46:22 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=178906) INFO 08-09 15:51:35 [launcher.py:100] [shutdown] API server: shutdown triggered |
| (APIServer pid=178906) INFO 08-09 15:51:35 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s |
| (APIServer pid=178906) INFO 08-09 15:51:35 [core_client.py:655] [shutdown] MPClient: start timeout=0s |
| (APIServer pid=178906) INFO 08-09 15:51:35 [core_client.py:657] [shutdown] MPClient: stopping engine manager |
| (APIServer pid=178906) WARNING 08-09 15:51:35 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 |
| (EngineCore pid=179352) INFO 08-09 15:51:35 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM |
| (EngineCore pid=179352) INFO 08-09 15:51:35 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s |
| (EngineCore pid=179352) INFO 08-09 15:51:35 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown |
| (EngineCore pid=179352) INFO 08-09 15:51:35 [core.py:1227] [shutdown] EngineCore: exiting busy loop |
| (APIServer pid=178906) INFO 08-09 15:51:35 [core_client.py:659] [shutdown] MPClient: engine manager stopped |
| (APIServer pid=178906) INFO 08-09 15:51:35 [core_client.py:660] [shutdown] MPClient: cleaning up background resources |
| (APIServer pid=178906) INFO 08-09 15:51:35 [core_client.py:662] [shutdown] MPClient: complete |
| (APIServer pid=178906) INFO 08-09 15:51:35 [launcher.py:125] [shutdown] API server: engine client stopped |
| (APIServer pid=178906) INFO 08-09 15:51:35 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown |
| (APIServer pid=178906) INFO 08-09 15:51:35 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server |
| (APIServer pid=178906) INFO: Shutting down |
| (APIServer pid=178906) INFO: Waiting for application shutdown. |
| (APIServer pid=178906) INFO: Application shutdown complete. |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
|
|
