| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339]
|
| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/release/healed/release-offpolicy-math-keep50/step0150
|
| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=77820) INFO 08-08 22:02:05 [api_utils.py:339] |
| (APIServer pid=77820) INFO 08-08 22:02:06 [api_utils.py:273] non-default args: {'model_tag': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'host': '127.0.0.1', 'port': 8461, 'model': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85} |
| (APIServer pid=77820) INFO 08-08 22:02:06 [model.py:619] Resolved architecture: PrunedOlmoeForCausalLM |
| (APIServer pid=77820) INFO 08-08 22:02:06 [model.py:1776] Using max model len 4096 |
| (APIServer pid=77820) INFO 08-08 22:02:06 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=77820) WARNING 08-08 22:02:06 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=77820) WARNING 08-08 22:02:06 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=77820) INFO 08-08 22:02:06 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=77820) INFO 08-08 22:02:06 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=77820) INFO 08-08 22:02:06 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (EngineCore pid=77950) INFO 08-08 22:02:13 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/release/healed/release-offpolicy-math-keep50/step0150', speculative_config=None, tokenizer='outputs/release/healed/release-offpolicy-math-keep50/step0150', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=77950) INFO 08-08 22:02:14 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.27:55219 backend=nccl |
| (EngineCore pid=77950) INFO 08-08 22:02:14 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (EngineCore pid=77950) INFO 08-08 22:02:15 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (EngineCore pid=77950) INFO 08-08 22:02:15 [gpu_model_runner.py:5209] Starting to load model outputs/release/healed/release-offpolicy-math-keep50/step0150... |
| (EngineCore pid=77950) INFO 08-08 22:02:16 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (EngineCore pid=77950) INFO 08-08 22:02:16 [flash_attn.py:718] Using FlashAttention version 2 |
| (EngineCore pid=77950) INFO 08-08 22:02:16 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 6.89 GiB. Available RAM: 107.59 GiB. |
| (EngineCore pid=77950) INFO 08-08 22:02:16 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (EngineCore pid=77950)
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] |
| (EngineCore pid=77950)
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:05<00:05, 5.02s/it] |
| (EngineCore pid=77950)
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:08<00:00, 4.15s/it] |
| (EngineCore pid=77950)
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:08<00:00, 4.28s/it] |
| (EngineCore pid=77950) |
| (EngineCore pid=77950) INFO 08-08 22:02:24 [default_loader.py:430] Loading weights took 8.57 seconds |
| (EngineCore pid=77950) INFO 08-08 22:02:25 [gpu_model_runner.py:5306] Model loading took 6.89 GiB memory and 8.834353 seconds |
| (EngineCore pid=77950) INFO 08-08 22:02:26 [gpu_worker.py:538] Available KV cache memory: 12.86 GiB |
| (EngineCore pid=77950) INFO 08-08 22:02:26 [kv_cache_utils.py:2146] GPU KV cache size: 105,328 tokens |
| (EngineCore pid=77950) INFO 08-08 22:02:26 [kv_cache_utils.py:2147] Maximum concurrency for 4,096 tokens per request: 25.71x |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.27 s |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=77950) WARNING 08-08 22:02:27 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=77950) WARNING 08-08 22:02:27 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=77950) INFO 08-08 22:02:27 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=77820) INFO 08-08 22:02:27 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=77820) INFO 08-08 22:02:28 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=77820) INFO 08-08 22:02:28 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8461 |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:37] Available routes are: |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /docs, Methods: HEAD, GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /redoc, Methods: HEAD, GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=77820) INFO 08-08 22:02:28 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=77820) INFO: Started server process [77820] |
| (APIServer pid=77820) INFO: Waiting for application startup. |
| (APIServer pid=77820) INFO: Application startup complete. |
| (APIServer pid=77820) INFO: 127.0.0.1:55202 - "GET /health HTTP/1.1" 200 OK |
| (EngineCore pid=77950) WARNING 08-08 22:02:48 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _plan_tiles. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=77950) WARNING 08-08 22:02:48 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _gate_up_silu. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=77950) WARNING 08-08 22:02:48 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _down_proj. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (EngineCore pid=77950) WARNING 08-08 22:02:49 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _count_experts. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=77950) WARNING 08-08 22:02:49 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _scatter_routes. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:02:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1656.9 tokens/s, Avg generation throughput: 3144.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:08 [loggers.py:273] Engine 000: Avg prompt throughput: 1174.1 tokens/s, Avg generation throughput: 3152.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:18 [loggers.py:273] Engine 000: Avg prompt throughput: 1159.6 tokens/s, Avg generation throughput: 3043.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:28 [loggers.py:273] Engine 000: Avg prompt throughput: 1256.2 tokens/s, Avg generation throughput: 3079.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:38 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.0 tokens/s, Avg generation throughput: 3164.4 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:48 [loggers.py:273] Engine 000: Avg prompt throughput: 1201.7 tokens/s, Avg generation throughput: 3115.2 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:03:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1264.2 tokens/s, Avg generation throughput: 3085.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:08 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.1 tokens/s, Avg generation throughput: 3136.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40098 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39972 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40168 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40220 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40204 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40188 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39906 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40148 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:40160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:18 [loggers.py:273] Engine 000: Avg prompt throughput: 507.4 tokens/s, Avg generation throughput: 1868.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:40194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:28 [loggers.py:273] Engine 000: Avg prompt throughput: 624.6 tokens/s, Avg generation throughput: 1567.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 34.2% |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:38 [loggers.py:273] Engine 000: Avg prompt throughput: 394.2 tokens/s, Avg generation throughput: 2285.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 46.4% |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:48 [loggers.py:273] Engine 000: Avg prompt throughput: 226.3 tokens/s, Avg generation throughput: 2115.2 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 52.5% |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:04:58 [loggers.py:273] Engine 000: Avg prompt throughput: 465.3 tokens/s, Avg generation throughput: 2150.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 59.3% |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:08 [loggers.py:273] Engine 000: Avg prompt throughput: 404.8 tokens/s, Avg generation throughput: 2175.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 63.5% |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:18 [loggers.py:273] Engine 000: Avg prompt throughput: 336.9 tokens/s, Avg generation throughput: 2123.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 67.2% |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:28 [loggers.py:273] Engine 000: Avg prompt throughput: 448.5 tokens/s, Avg generation throughput: 2123.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 69.7% |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:38 [loggers.py:273] Engine 000: Avg prompt throughput: 247.0 tokens/s, Avg generation throughput: 2129.1 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 71.9% |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:48 [loggers.py:273] Engine 000: Avg prompt throughput: 308.7 tokens/s, Avg generation throughput: 2123.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 73.9% |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:05:58 [loggers.py:273] Engine 000: Avg prompt throughput: 418.0 tokens/s, Avg generation throughput: 2198.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 76.1% |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:08 [loggers.py:273] Engine 000: Avg prompt throughput: 346.8 tokens/s, Avg generation throughput: 2175.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 77.9% |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49350 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49030 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:48988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:18 [loggers.py:273] Engine 000: Avg prompt throughput: 317.5 tokens/s, Avg generation throughput: 2137.4 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 79.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:49112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49282 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49146 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1488.5 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 79.0% |
| (APIServer pid=77820) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49088 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:38 [loggers.py:273] Engine 000: Avg prompt throughput: 485.0 tokens/s, Avg generation throughput: 1685.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 79.4% |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:48 [loggers.py:273] Engine 000: Avg prompt throughput: 453.8 tokens/s, Avg generation throughput: 2984.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 79.7% |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:06:58 [loggers.py:273] Engine 000: Avg prompt throughput: 368.0 tokens/s, Avg generation throughput: 2752.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 80.1% |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:08 [loggers.py:273] Engine 000: Avg prompt throughput: 478.8 tokens/s, Avg generation throughput: 2758.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 80.4% |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:18 [loggers.py:273] Engine 000: Avg prompt throughput: 482.9 tokens/s, Avg generation throughput: 2863.5 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.5%, Prefix cache hit rate: 80.7% |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:28 [loggers.py:273] Engine 000: Avg prompt throughput: 350.7 tokens/s, Avg generation throughput: 2805.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 81.1% |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:38 [loggers.py:273] Engine 000: Avg prompt throughput: 427.3 tokens/s, Avg generation throughput: 2798.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 81.3% |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:48 [loggers.py:273] Engine 000: Avg prompt throughput: 187.6 tokens/s, Avg generation throughput: 2426.0 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 80.3% |
| (APIServer pid=77820) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43682 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:07:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 830.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 80.3% |
| (APIServer pid=77820) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:08:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 56.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.3% |
| (APIServer pid=77820) INFO 08-08 22:08:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.3% |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46630 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46564 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46630 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46832 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46772 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46672 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46780 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46598 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46628 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46608 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46832 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46840 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46754 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46628 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46718 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46530 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46564 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46608 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46598 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46700 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46832 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46826 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46630 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46840 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46718 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46628 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46530 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:08:48 [loggers.py:273] Engine 000: Avg prompt throughput: 1911.1 tokens/s, Avg generation throughput: 873.3 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 73.1% |
| (APIServer pid=77820) INFO: 127.0.0.1:46832 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46598 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46796 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46608 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46630 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46754 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46780 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46840 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46744 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46700 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46826 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46628 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46718 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46564 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46796 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46516 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46584 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46744 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46678 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46838 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46516 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46840 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46564 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46516 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46840 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46678 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46630 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46598 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46608 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46718 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46700 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46780 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46584 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46832 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46744 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46826 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46560 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46632 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46672 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46796 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46838 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46530 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46772 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46754 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46502 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46732 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:08:58 [loggers.py:273] Engine 000: Avg prompt throughput: 413.8 tokens/s, Avg generation throughput: 681.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 70.2% |
| (APIServer pid=77820) INFO: 127.0.0.1:46576 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:46628 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:09:08 [loggers.py:273] Engine 000: Avg prompt throughput: 3525.3 tokens/s, Avg generation throughput: 1113.4 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 63.4% |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58934 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59264 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59056 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59166 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59066 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59276 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59042 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59014 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58920 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59082 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59258 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58980 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58966 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59168 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58954 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59006 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59226 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59210 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59260 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59000 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:58940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59290 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59152 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO: 127.0.0.1:59090 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=77820) INFO 08-08 22:09:18 [loggers.py:273] Engine 000: Avg prompt throughput: 2023.4 tokens/s, Avg generation throughput: 1586.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=77820) INFO 08-08 22:09:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=77820) INFO 08-08 22:14:14 [launcher.py:100] [shutdown] API server: shutdown triggered |
| (APIServer pid=77820) INFO 08-08 22:14:14 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s |
| (APIServer pid=77820) INFO 08-08 22:14:14 [core_client.py:655] [shutdown] MPClient: start timeout=0s |
| (APIServer pid=77820) INFO 08-08 22:14:14 [core_client.py:657] [shutdown] MPClient: stopping engine manager |
| (APIServer pid=77820) WARNING 08-08 22:14:14 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 |
| (EngineCore pid=77950) INFO 08-08 22:14:14 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM |
| (EngineCore pid=77950) INFO 08-08 22:14:14 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s |
| (EngineCore pid=77950) INFO 08-08 22:14:14 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown |
| (EngineCore pid=77950) INFO 08-08 22:14:14 [core.py:1227] [shutdown] EngineCore: exiting busy loop |
| (APIServer pid=77820) INFO 08-08 22:14:14 [core_client.py:659] [shutdown] MPClient: engine manager stopped |
| (APIServer pid=77820) INFO 08-08 22:14:14 [core_client.py:660] [shutdown] MPClient: cleaning up background resources |
| (APIServer pid=77820) INFO 08-08 22:14:14 [core_client.py:662] [shutdown] MPClient: complete |
| (APIServer pid=77820) INFO 08-08 22:14:14 [launcher.py:125] [shutdown] API server: engine client stopped |
| (APIServer pid=77820) INFO 08-08 22:14:14 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown |
| (APIServer pid=77820) INFO 08-08 22:14:15 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server |
| (APIServer pid=77820) INFO: Shutting down |
| (APIServer pid=77820) INFO: Waiting for application shutdown. |
| (APIServer pid=77820) INFO: Application shutdown complete. |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
|
|
