| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339]
|
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/qwen35_pruned_keep50
|
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:339] |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [api_utils.py:273] non-default args: {'model_tag': 'outputs/qwen35_pruned_keep50', 'host': '127.0.0.1', 'port': 8399, 'model': 'outputs/qwen35_pruned_keep50', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.9} |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [model.py:619] Resolved architecture: PrunedQwen3_5MoeForCausalLM |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [model.py:1776] Using max model len 8192 |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=1275713) WARNING 07-24 12:13:46 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=1275713) WARNING 07-24 12:13:46 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=1275713) INFO 07-24 12:13:46 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (EngineCore pid=1275876) INFO 07-24 12:13:58 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/qwen35_pruned_keep50', speculative_config=None, tokenizer='outputs/qwen35_pruned_keep50', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=1275876) WARNING 07-24 12:13:58 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. |
| (EngineCore pid=1275876) INFO 07-24 12:13:58 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=2, local_world_size=2 |
| (Worker pid=1275992) INFO 07-24 12:14:06 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:47861 backend=nccl |
| (Worker pid=1275993) INFO 07-24 12:14:06 [parallel_state.py:1607] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:47861 backend=nccl |
| (Worker pid=1275992) INFO 07-24 12:14:07 [pynccl.py:113] vLLM is using nccl==2.28.9 |
| (Worker pid=1275992) INFO 07-24 12:14:08 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. |
| (Worker pid=1275992) INFO 07-24 12:14:08 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (Worker pid=1275992) INFO 07-24 12:14:08 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:08 [gpu_model_runner.py:5209] Starting to load model outputs/qwen35_pruned_keep50... |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:09 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). |
| (Worker_PP0 pid=1275992) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| (Worker_PP0 pid=1275992) warnings.warn('Grouped GEMM not available.') |
| (Worker_PP1 pid=1275993) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| (Worker_PP1 pid=1275993) warnings.warn('Grouped GEMM not available.') |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:09 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:09 [flash_attn.py:718] Using FlashAttention version 2 |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:09 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 34.55 GiB. Available RAM: 106.53 GiB. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:09 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (Worker_PP0 pid=1275992)
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] |
| (Worker_PP0 pid=1275992)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:12<00:00, 12.44s/it] |
| (Worker_PP0 pid=1275992)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:12<00:00, 12.44s/it] |
| (Worker_PP0 pid=1275992) |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:22 [default_loader.py:430] Loading weights took 13.57 seconds |
| (Worker_PP1 pid=1275993) INFO 07-24 12:14:23 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP1 pid=1275993) INFO 07-24 12:14:23 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:23 [gpu_model_runner.py:5306] Model loading took 17.4 GiB memory and 13.896791 seconds |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:23 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:23 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:25 [gpu_worker.py:538] Available KV cache memory: 3.55 GiB |
| (EngineCore pid=1275876) INFO 07-24 12:14:25 [kv_cache_utils.py:2146] GPU KV cache size: 157,372 tokens |
| (EngineCore pid=1275876) INFO 07-24 12:14:25 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 19.21x |
| (Worker_PP1 pid=1275993) INFO 07-24 12:14:25 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:25 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP0 pid=1275992) INFO 07-24 12:14:25 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (Worker_PP1 pid=1275993) INFO 07-24 12:14:25 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=1275876) INFO 07-24 12:14:25 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.37 s |
| (EngineCore pid=1275876) INFO 07-24 12:14:28 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=1275876) WARNING 07-24 12:14:28 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=1275876) WARNING 07-24 12:14:28 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=1275876) INFO 07-24 12:14:28 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=1275876) INFO 07-24 12:14:28 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=1275876) INFO 07-24 12:14:28 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=1275713) INFO 07-24 12:14:28 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8399 |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:37] Available routes are: |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /docs, Methods: GET, HEAD |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /redoc, Methods: GET, HEAD |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=1275713) INFO 07-24 12:14:30 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=1275713) INFO: Started server process [1275713] |
| (APIServer pid=1275713) INFO: Waiting for application startup. |
| (APIServer pid=1275713) INFO: Application startup complete. |
| (APIServer pid=1275713) INFO: 127.0.0.1:49996 - "GET /health HTTP/1.1" 200 OK |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _build_route_rows. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/distributed/parallel_state.py:813: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1586.) |
| (Worker_PP0 pid=1275992) object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8) |
| [rank0]:[W724 12:14:59.097597705 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) |
| [rank1]:[W724 12:14:59.097875933 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1275992) WARNING 07-24 12:14:59 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=1275713) INFO 07-24 12:15:00 [loggers.py:273] Engine 000: Avg prompt throughput: 411.4 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:15:10 [loggers.py:273] Engine 000: Avg prompt throughput: 110.2 tokens/s, Avg generation throughput: 659.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:15:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 672.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:15:30 [loggers.py:273] Engine 000: Avg prompt throughput: 47.1 tokens/s, Avg generation throughput: 651.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:15:40 [loggers.py:273] Engine 000: Avg prompt throughput: 106.1 tokens/s, Avg generation throughput: 637.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:15:50 [loggers.py:273] Engine 000: Avg prompt throughput: 59.9 tokens/s, Avg generation throughput: 644.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:00 [loggers.py:273] Engine 000: Avg prompt throughput: 37.4 tokens/s, Avg generation throughput: 651.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:10 [loggers.py:273] Engine 000: Avg prompt throughput: 116.5 tokens/s, Avg generation throughput: 631.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:20 [loggers.py:273] Engine 000: Avg prompt throughput: 70.8 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:30 [loggers.py:273] Engine 000: Avg prompt throughput: 62.1 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:40 [loggers.py:273] Engine 000: Avg prompt throughput: 58.9 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:16:50 [loggers.py:273] Engine 000: Avg prompt throughput: 71.3 tokens/s, Avg generation throughput: 638.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:00 [loggers.py:273] Engine 000: Avg prompt throughput: 71.0 tokens/s, Avg generation throughput: 638.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 83.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:10 [loggers.py:273] Engine 000: Avg prompt throughput: 158.0 tokens/s, Avg generation throughput: 623.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:20 [loggers.py:273] Engine 000: Avg prompt throughput: 31.1 tokens/s, Avg generation throughput: 651.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:30 [loggers.py:273] Engine 000: Avg prompt throughput: 78.0 tokens/s, Avg generation throughput: 631.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:40 [loggers.py:273] Engine 000: Avg prompt throughput: 81.5 tokens/s, Avg generation throughput: 631.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:17:50 [loggers.py:273] Engine 000: Avg prompt throughput: 60.6 tokens/s, Avg generation throughput: 638.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:00 [loggers.py:273] Engine 000: Avg prompt throughput: 99.1 tokens/s, Avg generation throughput: 618.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:10 [loggers.py:273] Engine 000: Avg prompt throughput: 100.5 tokens/s, Avg generation throughput: 631.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:20 [loggers.py:273] Engine 000: Avg prompt throughput: 94.0 tokens/s, Avg generation throughput: 637.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:30 [loggers.py:273] Engine 000: Avg prompt throughput: 49.7 tokens/s, Avg generation throughput: 651.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:40 [loggers.py:273] Engine 000: Avg prompt throughput: 84.8 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:18:50 [loggers.py:273] Engine 000: Avg prompt throughput: 58.9 tokens/s, Avg generation throughput: 638.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:00 [loggers.py:273] Engine 000: Avg prompt throughput: 52.3 tokens/s, Avg generation throughput: 644.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:10 [loggers.py:273] Engine 000: Avg prompt throughput: 117.4 tokens/s, Avg generation throughput: 624.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:20 [loggers.py:273] Engine 000: Avg prompt throughput: 81.1 tokens/s, Avg generation throughput: 637.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:30 [loggers.py:273] Engine 000: Avg prompt throughput: 77.7 tokens/s, Avg generation throughput: 644.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:40 [loggers.py:273] Engine 000: Avg prompt throughput: 83.7 tokens/s, Avg generation throughput: 631.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:19:50 [loggers.py:273] Engine 000: Avg prompt throughput: 107.2 tokens/s, Avg generation throughput: 637.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:00 [loggers.py:273] Engine 000: Avg prompt throughput: 78.0 tokens/s, Avg generation throughput: 638.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:10 [loggers.py:273] Engine 000: Avg prompt throughput: 103.8 tokens/s, Avg generation throughput: 631.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:20 [loggers.py:273] Engine 000: Avg prompt throughput: 79.4 tokens/s, Avg generation throughput: 631.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:30 [loggers.py:273] Engine 000: Avg prompt throughput: 82.5 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:40 [loggers.py:273] Engine 000: Avg prompt throughput: 51.1 tokens/s, Avg generation throughput: 638.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:20:50 [loggers.py:273] Engine 000: Avg prompt throughput: 85.9 tokens/s, Avg generation throughput: 631.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:00 [loggers.py:273] Engine 000: Avg prompt throughput: 63.5 tokens/s, Avg generation throughput: 644.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:10 [loggers.py:273] Engine 000: Avg prompt throughput: 67.4 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:20 [loggers.py:273] Engine 000: Avg prompt throughput: 63.1 tokens/s, Avg generation throughput: 638.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:30 [loggers.py:273] Engine 000: Avg prompt throughput: 83.2 tokens/s, Avg generation throughput: 644.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:40 [loggers.py:273] Engine 000: Avg prompt throughput: 75.7 tokens/s, Avg generation throughput: 644.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:21:50 [loggers.py:273] Engine 000: Avg prompt throughput: 115.4 tokens/s, Avg generation throughput: 637.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:00 [loggers.py:273] Engine 000: Avg prompt throughput: 31.7 tokens/s, Avg generation throughput: 651.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:10 [loggers.py:273] Engine 000: Avg prompt throughput: 65.0 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:20 [loggers.py:273] Engine 000: Avg prompt throughput: 78.4 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:30 [loggers.py:273] Engine 000: Avg prompt throughput: 74.4 tokens/s, Avg generation throughput: 631.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:40 [loggers.py:273] Engine 000: Avg prompt throughput: 78.2 tokens/s, Avg generation throughput: 644.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:22:50 [loggers.py:273] Engine 000: Avg prompt throughput: 51.6 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:00 [loggers.py:273] Engine 000: Avg prompt throughput: 53.9 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:10 [loggers.py:273] Engine 000: Avg prompt throughput: 81.8 tokens/s, Avg generation throughput: 637.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:20 [loggers.py:273] Engine 000: Avg prompt throughput: 118.8 tokens/s, Avg generation throughput: 631.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:30 [loggers.py:273] Engine 000: Avg prompt throughput: 94.1 tokens/s, Avg generation throughput: 637.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:40 [loggers.py:273] Engine 000: Avg prompt throughput: 113.7 tokens/s, Avg generation throughput: 637.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:23:50 [loggers.py:273] Engine 000: Avg prompt throughput: 76.5 tokens/s, Avg generation throughput: 644.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:00 [loggers.py:273] Engine 000: Avg prompt throughput: 53.5 tokens/s, Avg generation throughput: 651.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:10 [loggers.py:273] Engine 000: Avg prompt throughput: 96.2 tokens/s, Avg generation throughput: 630.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:20 [loggers.py:273] Engine 000: Avg prompt throughput: 85.4 tokens/s, Avg generation throughput: 644.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:30 [loggers.py:273] Engine 000: Avg prompt throughput: 50.7 tokens/s, Avg generation throughput: 651.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:40 [loggers.py:273] Engine 000: Avg prompt throughput: 117.5 tokens/s, Avg generation throughput: 631.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:24:50 [loggers.py:273] Engine 000: Avg prompt throughput: 81.5 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:00 [loggers.py:273] Engine 000: Avg prompt throughput: 57.0 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:10 [loggers.py:273] Engine 000: Avg prompt throughput: 34.7 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:20 [loggers.py:273] Engine 000: Avg prompt throughput: 102.8 tokens/s, Avg generation throughput: 631.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:30 [loggers.py:273] Engine 000: Avg prompt throughput: 77.9 tokens/s, Avg generation throughput: 644.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:40 [loggers.py:273] Engine 000: Avg prompt throughput: 54.3 tokens/s, Avg generation throughput: 644.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:25:50 [loggers.py:273] Engine 000: Avg prompt throughput: 101.7 tokens/s, Avg generation throughput: 637.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:00 [loggers.py:273] Engine 000: Avg prompt throughput: 70.6 tokens/s, Avg generation throughput: 644.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:10 [loggers.py:273] Engine 000: Avg prompt throughput: 38.0 tokens/s, Avg generation throughput: 651.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:20 [loggers.py:273] Engine 000: Avg prompt throughput: 60.0 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:30 [loggers.py:273] Engine 000: Avg prompt throughput: 127.2 tokens/s, Avg generation throughput: 630.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:40 [loggers.py:273] Engine 000: Avg prompt throughput: 86.7 tokens/s, Avg generation throughput: 638.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:26:50 [loggers.py:273] Engine 000: Avg prompt throughput: 98.8 tokens/s, Avg generation throughput: 630.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:00 [loggers.py:273] Engine 000: Avg prompt throughput: 98.1 tokens/s, Avg generation throughput: 637.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:10 [loggers.py:273] Engine 000: Avg prompt throughput: 77.8 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:20 [loggers.py:273] Engine 000: Avg prompt throughput: 62.4 tokens/s, Avg generation throughput: 651.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:30 [loggers.py:273] Engine 000: Avg prompt throughput: 66.3 tokens/s, Avg generation throughput: 644.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:40 [loggers.py:273] Engine 000: Avg prompt throughput: 89.9 tokens/s, Avg generation throughput: 637.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:27:50 [loggers.py:273] Engine 000: Avg prompt throughput: 67.8 tokens/s, Avg generation throughput: 644.3 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:00 [loggers.py:273] Engine 000: Avg prompt throughput: 133.9 tokens/s, Avg generation throughput: 631.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:10 [loggers.py:273] Engine 000: Avg prompt throughput: 82.4 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:20 [loggers.py:273] Engine 000: Avg prompt throughput: 121.3 tokens/s, Avg generation throughput: 630.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:30 [loggers.py:273] Engine 000: Avg prompt throughput: 56.4 tokens/s, Avg generation throughput: 645.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:40 [loggers.py:273] Engine 000: Avg prompt throughput: 89.9 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:28:50 [loggers.py:273] Engine 000: Avg prompt throughput: 95.7 tokens/s, Avg generation throughput: 637.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:00 [loggers.py:273] Engine 000: Avg prompt throughput: 26.8 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:10 [loggers.py:273] Engine 000: Avg prompt throughput: 22.8 tokens/s, Avg generation throughput: 658.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:20 [loggers.py:273] Engine 000: Avg prompt throughput: 84.3 tokens/s, Avg generation throughput: 631.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:30 [loggers.py:273] Engine 000: Avg prompt throughput: 94.8 tokens/s, Avg generation throughput: 631.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:40 [loggers.py:273] Engine 000: Avg prompt throughput: 47.3 tokens/s, Avg generation throughput: 651.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:29:50 [loggers.py:273] Engine 000: Avg prompt throughput: 57.9 tokens/s, Avg generation throughput: 651.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:00 [loggers.py:273] Engine 000: Avg prompt throughput: 106.7 tokens/s, Avg generation throughput: 630.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:10 [loggers.py:273] Engine 000: Avg prompt throughput: 91.7 tokens/s, Avg generation throughput: 637.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:20 [loggers.py:273] Engine 000: Avg prompt throughput: 53.9 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:30 [loggers.py:273] Engine 000: Avg prompt throughput: 64.9 tokens/s, Avg generation throughput: 651.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:40 [loggers.py:273] Engine 000: Avg prompt throughput: 156.0 tokens/s, Avg generation throughput: 623.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:30:50 [loggers.py:273] Engine 000: Avg prompt throughput: 106.8 tokens/s, Avg generation throughput: 631.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:00 [loggers.py:273] Engine 000: Avg prompt throughput: 124.2 tokens/s, Avg generation throughput: 637.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:10 [loggers.py:273] Engine 000: Avg prompt throughput: 34.1 tokens/s, Avg generation throughput: 658.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:20 [loggers.py:273] Engine 000: Avg prompt throughput: 59.5 tokens/s, Avg generation throughput: 638.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:30 [loggers.py:273] Engine 000: Avg prompt throughput: 72.8 tokens/s, Avg generation throughput: 644.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:40 [loggers.py:273] Engine 000: Avg prompt throughput: 111.3 tokens/s, Avg generation throughput: 631.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:31:50 [loggers.py:273] Engine 000: Avg prompt throughput: 107.4 tokens/s, Avg generation throughput: 637.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:00 [loggers.py:273] Engine 000: Avg prompt throughput: 75.7 tokens/s, Avg generation throughput: 644.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:10 [loggers.py:273] Engine 000: Avg prompt throughput: 68.4 tokens/s, Avg generation throughput: 644.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:20 [loggers.py:273] Engine 000: Avg prompt throughput: 88.8 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:30 [loggers.py:273] Engine 000: Avg prompt throughput: 89.3 tokens/s, Avg generation throughput: 631.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:40 [loggers.py:273] Engine 000: Avg prompt throughput: 28.4 tokens/s, Avg generation throughput: 651.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:32:50 [loggers.py:273] Engine 000: Avg prompt throughput: 47.7 tokens/s, Avg generation throughput: 645.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:00 [loggers.py:273] Engine 000: Avg prompt throughput: 73.8 tokens/s, Avg generation throughput: 631.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:10 [loggers.py:273] Engine 000: Avg prompt throughput: 101.5 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:20 [loggers.py:273] Engine 000: Avg prompt throughput: 62.9 tokens/s, Avg generation throughput: 638.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:30 [loggers.py:273] Engine 000: Avg prompt throughput: 98.0 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:40 [loggers.py:273] Engine 000: Avg prompt throughput: 70.4 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:33:50 [loggers.py:273] Engine 000: Avg prompt throughput: 17.9 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:00 [loggers.py:273] Engine 000: Avg prompt throughput: 66.8 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:10 [loggers.py:273] Engine 000: Avg prompt throughput: 79.0 tokens/s, Avg generation throughput: 638.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:20 [loggers.py:273] Engine 000: Avg prompt throughput: 89.5 tokens/s, Avg generation throughput: 631.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:30 [loggers.py:273] Engine 000: Avg prompt throughput: 85.2 tokens/s, Avg generation throughput: 631.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:40 [loggers.py:273] Engine 000: Avg prompt throughput: 65.3 tokens/s, Avg generation throughput: 631.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:34:50 [loggers.py:273] Engine 000: Avg prompt throughput: 80.6 tokens/s, Avg generation throughput: 638.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:00 [loggers.py:273] Engine 000: Avg prompt throughput: 63.5 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:10 [loggers.py:273] Engine 000: Avg prompt throughput: 77.4 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:20 [loggers.py:273] Engine 000: Avg prompt throughput: 100.7 tokens/s, Avg generation throughput: 624.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:30 [loggers.py:273] Engine 000: Avg prompt throughput: 63.7 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:40 [loggers.py:273] Engine 000: Avg prompt throughput: 42.5 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:35:50 [loggers.py:273] Engine 000: Avg prompt throughput: 117.5 tokens/s, Avg generation throughput: 631.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:00 [loggers.py:273] Engine 000: Avg prompt throughput: 36.6 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:10 [loggers.py:273] Engine 000: Avg prompt throughput: 68.5 tokens/s, Avg generation throughput: 638.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:20 [loggers.py:273] Engine 000: Avg prompt throughput: 56.4 tokens/s, Avg generation throughput: 644.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:30 [loggers.py:273] Engine 000: Avg prompt throughput: 85.8 tokens/s, Avg generation throughput: 638.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:40 [loggers.py:273] Engine 000: Avg prompt throughput: 95.7 tokens/s, Avg generation throughput: 631.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:36:50 [loggers.py:273] Engine 000: Avg prompt throughput: 93.1 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:00 [loggers.py:273] Engine 000: Avg prompt throughput: 78.5 tokens/s, Avg generation throughput: 637.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:10 [loggers.py:273] Engine 000: Avg prompt throughput: 58.6 tokens/s, Avg generation throughput: 645.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:20 [loggers.py:273] Engine 000: Avg prompt throughput: 65.9 tokens/s, Avg generation throughput: 644.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:30 [loggers.py:273] Engine 000: Avg prompt throughput: 138.1 tokens/s, Avg generation throughput: 623.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:40 [loggers.py:273] Engine 000: Avg prompt throughput: 44.4 tokens/s, Avg generation throughput: 631.2 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:37:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 553.3 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 59.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 470.0 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 45.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 355.2 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 248.0 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 212.5 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 147.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:38:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 79.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:39:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:39:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:39:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:39:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:39:40 [loggers.py:273] Engine 000: Avg prompt throughput: 5018.6 tokens/s, Avg generation throughput: 213.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:39:50 [loggers.py:273] Engine 000: Avg prompt throughput: 147.4 tokens/s, Avg generation throughput: 639.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:40:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:40:10 [loggers.py:273] Engine 000: Avg prompt throughput: 75.7 tokens/s, Avg generation throughput: 626.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:40:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 605.3 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:40:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:40:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 617.9 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:40:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 618.0 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:41:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 597.5 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:41:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 578.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:41:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 551.1 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:41:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 550.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:41:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 525.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:41:50 [loggers.py:273] Engine 000: Avg prompt throughput: 4120.2 tokens/s, Avg generation throughput: 188.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:42:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 652.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:42:10 [loggers.py:273] Engine 000: Avg prompt throughput: 150.6 tokens/s, Avg generation throughput: 626.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:42:20 [loggers.py:273] Engine 000: Avg prompt throughput: 97.6 tokens/s, Avg generation throughput: 633.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:42:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 581.5 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:42:40 [loggers.py:273] Engine 000: Avg prompt throughput: 521.5 tokens/s, Avg generation throughput: 574.0 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:42:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.2 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:43:00 [loggers.py:273] Engine 000: Avg prompt throughput: 306.5 tokens/s, Avg generation throughput: 606.4 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:43:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 625.7 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:43:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 583.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:43:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 557.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:43:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 540.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:43:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 551.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:44:00 [loggers.py:273] Engine 000: Avg prompt throughput: 2179.9 tokens/s, Avg generation throughput: 347.9 tokens/s, Running: 53 reqs, Waiting: 7 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:44:10 [loggers.py:273] Engine 000: Avg prompt throughput: 1910.9 tokens/s, Avg generation throughput: 510.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:44:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 646.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:44:30 [loggers.py:273] Engine 000: Avg prompt throughput: 287.0 tokens/s, Avg generation throughput: 620.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:44:40 [loggers.py:273] Engine 000: Avg prompt throughput: 73.7 tokens/s, Avg generation throughput: 621.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:44:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 587.5 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:45:00 [loggers.py:273] Engine 000: Avg prompt throughput: 74.9 tokens/s, Avg generation throughput: 605.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:45:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 609.4 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:45:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 583.1 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:45:30 [loggers.py:273] Engine 000: Avg prompt throughput: 74.6 tokens/s, Avg generation throughput: 561.2 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:45:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 524.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:45:50 [loggers.py:273] Engine 000: Avg prompt throughput: 76.1 tokens/s, Avg generation throughput: 517.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:46:00 [loggers.py:273] Engine 000: Avg prompt throughput: 149.9 tokens/s, Avg generation throughput: 528.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:46:10 [loggers.py:273] Engine 000: Avg prompt throughput: 449.6 tokens/s, Avg generation throughput: 511.1 tokens/s, Running: 51 reqs, Waiting: 8 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:46:20 [loggers.py:273] Engine 000: Avg prompt throughput: 3946.5 tokens/s, Avg generation throughput: 346.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:46:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:46:40 [loggers.py:273] Engine 000: Avg prompt throughput: 145.9 tokens/s, Avg generation throughput: 633.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:46:50 [loggers.py:273] Engine 000: Avg prompt throughput: 77.1 tokens/s, Avg generation throughput: 619.9 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:47:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 601.7 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:47:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 617.9 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:47:20 [loggers.py:273] Engine 000: Avg prompt throughput: 149.6 tokens/s, Avg generation throughput: 602.0 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:47:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 604.4 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:47:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 604.4 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:47:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 558.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 530.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 513.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 544.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:30 [loggers.py:273] Engine 000: Avg prompt throughput: 4516.4 tokens/s, Avg generation throughput: 255.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:40 [loggers.py:273] Engine 000: Avg prompt throughput: 151.7 tokens/s, Avg generation throughput: 633.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:48:50 [loggers.py:273] Engine 000: Avg prompt throughput: 75.3 tokens/s, Avg generation throughput: 646.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:49:00 [loggers.py:273] Engine 000: Avg prompt throughput: 153.7 tokens/s, Avg generation throughput: 626.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:49:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:49:20 [loggers.py:273] Engine 000: Avg prompt throughput: 73.6 tokens/s, Avg generation throughput: 581.6 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:49:30 [loggers.py:273] Engine 000: Avg prompt throughput: 84.3 tokens/s, Avg generation throughput: 609.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:49:40 [loggers.py:273] Engine 000: Avg prompt throughput: 150.6 tokens/s, Avg generation throughput: 602.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:49:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 605.8 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:50:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 571.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:50:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 535.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:50:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 535.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:50:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 556.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:50:40 [loggers.py:273] Engine 000: Avg prompt throughput: 3865.4 tokens/s, Avg generation throughput: 248.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:50:50 [loggers.py:273] Engine 000: Avg prompt throughput: 379.5 tokens/s, Avg generation throughput: 639.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:51:00 [loggers.py:273] Engine 000: Avg prompt throughput: 240.5 tokens/s, Avg generation throughput: 607.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:51:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 646.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:51:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 618.9 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:51:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 605.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:51:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 611.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:51:50 [loggers.py:273] Engine 000: Avg prompt throughput: 164.4 tokens/s, Avg generation throughput: 599.6 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:52:00 [loggers.py:273] Engine 000: Avg prompt throughput: 74.3 tokens/s, Avg generation throughput: 611.9 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:52:10 [loggers.py:273] Engine 000: Avg prompt throughput: 73.2 tokens/s, Avg generation throughput: 581.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:52:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 551.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:52:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 530.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:52:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 525.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:52:50 [loggers.py:273] Engine 000: Avg prompt throughput: 2708.7 tokens/s, Avg generation throughput: 325.3 tokens/s, Running: 51 reqs, Waiting: 10 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:53:00 [loggers.py:273] Engine 000: Avg prompt throughput: 1679.7 tokens/s, Avg generation throughput: 524.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:53:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 646.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:53:20 [loggers.py:273] Engine 000: Avg prompt throughput: 332.1 tokens/s, Avg generation throughput: 613.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:53:30 [loggers.py:273] Engine 000: Avg prompt throughput: 151.9 tokens/s, Avg generation throughput: 624.1 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:53:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 596.2 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:53:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 614.1 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:54:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 605.8 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:54:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 583.0 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:54:20 [loggers.py:273] Engine 000: Avg prompt throughput: 76.2 tokens/s, Avg generation throughput: 582.7 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:54:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 539.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:54:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 540.5 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:54:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 540.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:55:00 [loggers.py:273] Engine 000: Avg prompt throughput: 1091.2 tokens/s, Avg generation throughput: 440.2 tokens/s, Running: 52 reqs, Waiting: 6 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45316 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:55:10 [loggers.py:273] Engine 000: Avg prompt throughput: 2881.5 tokens/s, Avg generation throughput: 387.8 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:55:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 571.1 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:55:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 565.5 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:55:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 563.4 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:55:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 555.5 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45372 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:56:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 549.8 tokens/s, Running: 53 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:56:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 545.8 tokens/s, Running: 53 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45172 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:56:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 532.3 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45412 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:56:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 513.7 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:56:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 516.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:56:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 513.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:57:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 508.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45274 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45230 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45156 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:57:10 [loggers.py:273] Engine 000: Avg prompt throughput: 342.4 tokens/s, Avg generation throughput: 445.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:57:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 665.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:57:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:57:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 639.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:57:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 595.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6.6 tokens/s, Avg generation throughput: 601.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 83.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:10 [loggers.py:273] Engine 000: Avg prompt throughput: 13.0 tokens/s, Avg generation throughput: 626.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6.1 tokens/s, Avg generation throughput: 620.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:30 [loggers.py:273] Engine 000: Avg prompt throughput: 8.4 tokens/s, Avg generation throughput: 620.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6.3 tokens/s, Avg generation throughput: 613.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:58:50 [loggers.py:273] Engine 000: Avg prompt throughput: 13.8 tokens/s, Avg generation throughput: 613.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:59:00 [loggers.py:273] Engine 000: Avg prompt throughput: 9.8 tokens/s, Avg generation throughput: 611.7 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:59:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 604.6 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:59:20 [loggers.py:273] Engine 000: Avg prompt throughput: 9.6 tokens/s, Avg generation throughput: 605.1 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 12:59:30 [loggers.py:273] Engine 000: Avg prompt throughput: 293.4 tokens/s, Avg generation throughput: 629.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:59:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 646.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 12:59:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:00:00 [loggers.py:273] Engine 000: Avg prompt throughput: 4.7 tokens/s, Avg generation throughput: 633.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:00:10 [loggers.py:273] Engine 000: Avg prompt throughput: 11.6 tokens/s, Avg generation throughput: 633.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:00:20 [loggers.py:273] Engine 000: Avg prompt throughput: 9.3 tokens/s, Avg generation throughput: 626.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:00:30 [loggers.py:273] Engine 000: Avg prompt throughput: 16.4 tokens/s, Avg generation throughput: 633.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 13:00:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 608.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:00:50 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 607.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:01:00 [loggers.py:273] Engine 000: Avg prompt throughput: 25.4 tokens/s, Avg generation throughput: 613.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:01:10 [loggers.py:273] Engine 000: Avg prompt throughput: 16.5 tokens/s, Avg generation throughput: 620.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:01:20 [loggers.py:273] Engine 000: Avg prompt throughput: 24.2 tokens/s, Avg generation throughput: 613.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:01:30 [loggers.py:273] Engine 000: Avg prompt throughput: 22.3 tokens/s, Avg generation throughput: 613.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:01:40 [loggers.py:273] Engine 000: Avg prompt throughput: 241.0 tokens/s, Avg generation throughput: 612.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 13:01:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 13:02:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:02:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6.5 tokens/s, Avg generation throughput: 627.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:02:20 [loggers.py:273] Engine 000: Avg prompt throughput: 13.4 tokens/s, Avg generation throughput: 620.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:02:30 [loggers.py:273] Engine 000: Avg prompt throughput: 22.4 tokens/s, Avg generation throughput: 620.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:02:40 [loggers.py:273] Engine 000: Avg prompt throughput: 18.0 tokens/s, Avg generation throughput: 613.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:02:50 [loggers.py:273] Engine 000: Avg prompt throughput: 8.4 tokens/s, Avg generation throughput: 614.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:00 [loggers.py:273] Engine 000: Avg prompt throughput: 5.1 tokens/s, Avg generation throughput: 614.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:10 [loggers.py:273] Engine 000: Avg prompt throughput: 28.7 tokens/s, Avg generation throughput: 607.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:20 [loggers.py:273] Engine 000: Avg prompt throughput: 21.7 tokens/s, Avg generation throughput: 613.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:30 [loggers.py:273] Engine 000: Avg prompt throughput: 54.9 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:40 [loggers.py:273] Engine 000: Avg prompt throughput: 8.0 tokens/s, Avg generation throughput: 620.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:03:50 [loggers.py:273] Engine 000: Avg prompt throughput: 162.5 tokens/s, Avg generation throughput: 614.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:04:00 [loggers.py:273] Engine 000: Avg prompt throughput: 4.2 tokens/s, Avg generation throughput: 626.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:04:10 [loggers.py:273] Engine 000: Avg prompt throughput: 12.6 tokens/s, Avg generation throughput: 620.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 13:04:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:04:30 [loggers.py:273] Engine 000: Avg prompt throughput: 12.1 tokens/s, Avg generation throughput: 620.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:04:40 [loggers.py:273] Engine 000: Avg prompt throughput: 16.9 tokens/s, Avg generation throughput: 613.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:04:50 [loggers.py:273] Engine 000: Avg prompt throughput: 38.0 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:00 [loggers.py:273] Engine 000: Avg prompt throughput: 17.6 tokens/s, Avg generation throughput: 613.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:10 [loggers.py:273] Engine 000: Avg prompt throughput: 3.2 tokens/s, Avg generation throughput: 620.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:20 [loggers.py:273] Engine 000: Avg prompt throughput: 32.4 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:30 [loggers.py:273] Engine 000: Avg prompt throughput: 24.8 tokens/s, Avg generation throughput: 613.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:40 [loggers.py:273] Engine 000: Avg prompt throughput: 35.8 tokens/s, Avg generation throughput: 587.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:05:50 [loggers.py:273] Engine 000: Avg prompt throughput: 26.1 tokens/s, Avg generation throughput: 613.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:06:00 [loggers.py:273] Engine 000: Avg prompt throughput: 134.3 tokens/s, Avg generation throughput: 615.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:06:10 [loggers.py:273] Engine 000: Avg prompt throughput: 25.4 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:06:20 [loggers.py:273] Engine 000: Avg prompt throughput: 13.6 tokens/s, Avg generation throughput: 601.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO 07-24 13:06:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:06:40 [loggers.py:273] Engine 000: Avg prompt throughput: 4.4 tokens/s, Avg generation throughput: 620.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:06:50 [loggers.py:273] Engine 000: Avg prompt throughput: 18.1 tokens/s, Avg generation throughput: 620.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46520 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:00 [loggers.py:273] Engine 000: Avg prompt throughput: 35.4 tokens/s, Avg generation throughput: 612.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:10 [loggers.py:273] Engine 000: Avg prompt throughput: 17.8 tokens/s, Avg generation throughput: 620.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:20 [loggers.py:273] Engine 000: Avg prompt throughput: 17.4 tokens/s, Avg generation throughput: 607.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:30 [loggers.py:273] Engine 000: Avg prompt throughput: 23.6 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:40 [loggers.py:273] Engine 000: Avg prompt throughput: 69.6 tokens/s, Avg generation throughput: 599.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:07:50 [loggers.py:273] Engine 000: Avg prompt throughput: 11.1 tokens/s, Avg generation throughput: 620.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:08:00 [loggers.py:273] Engine 000: Avg prompt throughput: 32.6 tokens/s, Avg generation throughput: 613.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:08:10 [loggers.py:273] Engine 000: Avg prompt throughput: 17.3 tokens/s, Avg generation throughput: 613.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46934 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1275713) INFO 07-24 13:08:20 [loggers.py:273] Engine 000: Avg prompt throughput: 105.4 tokens/s, Avg generation throughput: 623.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1275713) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
|
|
