| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339]
|
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/release/healed/release-offpolicy-math-keep75/step0150
|
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:339] |
| (APIServer pid=112067) INFO 08-08 22:15:20 [api_utils.py:273] non-default args: {'model_tag': 'outputs/release/healed/release-offpolicy-math-keep75/step0150', 'host': '127.0.0.1', 'port': 8460, 'model': 'outputs/release/healed/release-offpolicy-math-keep75/step0150', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85} |
| (APIServer pid=112067) INFO 08-08 22:15:20 [model.py:619] Resolved architecture: PrunedOlmoeForCausalLM |
| (APIServer pid=112067) INFO 08-08 22:15:20 [model.py:1776] Using max model len 4096 |
| (APIServer pid=112067) INFO 08-08 22:15:20 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=112067) WARNING 08-08 22:15:20 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=112067) WARNING 08-08 22:15:20 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=112067) INFO 08-08 22:15:20 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=112067) INFO 08-08 22:15:20 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=112067) INFO 08-08 22:15:20 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (EngineCore pid=112194) INFO 08-08 22:15:27 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/release/healed/release-offpolicy-math-keep75/step0150', speculative_config=None, tokenizer='outputs/release/healed/release-offpolicy-math-keep75/step0150', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=112194) INFO 08-08 22:15:28 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.27:36275 backend=nccl |
| (EngineCore pid=112194) INFO 08-08 22:15:28 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (EngineCore pid=112194) INFO 08-08 22:15:29 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (EngineCore pid=112194) INFO 08-08 22:15:29 [gpu_model_runner.py:5209] Starting to load model outputs/release/healed/release-offpolicy-math-keep75/step0150... |
| (EngineCore pid=112194) INFO 08-08 22:15:30 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (EngineCore pid=112194) INFO 08-08 22:15:30 [flash_attn.py:718] Using FlashAttention version 2 |
| (EngineCore pid=112194) INFO 08-08 22:15:30 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 9.89 GiB. Available RAM: 106.85 GiB. |
| (EngineCore pid=112194) INFO 08-08 22:15:30 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (EngineCore pid=112194)
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s] |
| (EngineCore pid=112194)
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:03<00:06, 3.24s/it] |
| (EngineCore pid=112194)
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:07<00:03, 3.58s/it] |
| (EngineCore pid=112194)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:07<00:00, 2.19s/it] |
| (EngineCore pid=112194)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:07<00:00, 2.53s/it] |
| (EngineCore pid=112194) |
| (EngineCore pid=112194) INFO 08-08 22:15:37 [default_loader.py:430] Loading weights took 7.60 seconds |
| (EngineCore pid=112194) INFO 08-08 22:15:38 [gpu_model_runner.py:5306] Model loading took 9.89 GiB memory and 7.895527 seconds |
| (EngineCore pid=112194) INFO 08-08 22:15:40 [gpu_worker.py:538] Available KV cache memory: 9.86 GiB |
| (EngineCore pid=112194) INFO 08-08 22:15:40 [kv_cache_utils.py:2146] GPU KV cache size: 80,752 tokens |
| (EngineCore pid=112194) INFO 08-08 22:15:40 [kv_cache_utils.py:2147] Maximum concurrency for 4,096 tokens per request: 19.71x |
| (EngineCore pid=112194) INFO 08-08 22:15:40 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (EngineCore pid=112194) INFO 08-08 22:15:40 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=112194) INFO 08-08 22:15:41 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.65 s |
| (EngineCore pid=112194) INFO 08-08 22:15:41 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=112194) WARNING 08-08 22:15:41 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=112194) WARNING 08-08 22:15:41 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=112194) INFO 08-08 22:15:41 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=112194) INFO 08-08 22:15:41 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=112194) INFO 08-08 22:15:41 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=112067) INFO 08-08 22:15:41 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=112067) INFO 08-08 22:15:41 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=112067) INFO 08-08 22:15:41 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8460 |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:37] Available routes are: |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /docs, Methods: GET, HEAD |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /redoc, Methods: GET, HEAD |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=112067) INFO 08-08 22:15:41 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=112067) INFO: Started server process [112067] |
| (APIServer pid=112067) INFO: Waiting for application startup. |
| (APIServer pid=112067) INFO: Application startup complete. |
| (APIServer pid=112067) INFO: 127.0.0.1:42364 - "GET /health HTTP/1.1" 200 OK |
| (EngineCore pid=112194) WARNING 08-08 22:15:57 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _plan_tiles. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=112194) WARNING 08-08 22:15:57 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _gate_up_silu. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=112194) WARNING 08-08 22:15:57 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _down_proj. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (EngineCore pid=112194) WARNING 08-08 22:15:58 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _count_experts. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (EngineCore pid=112194) WARNING 08-08 22:15:58 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _scatter_routes. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:01 [loggers.py:273] Engine 000: Avg prompt throughput: 736.3 tokens/s, Avg generation throughput: 742.3 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:11 [loggers.py:273] Engine 000: Avg prompt throughput: 1888.3 tokens/s, Avg generation throughput: 2512.3 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:21 [loggers.py:273] Engine 000: Avg prompt throughput: 1784.6 tokens/s, Avg generation throughput: 2514.7 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:31 [loggers.py:273] Engine 000: Avg prompt throughput: 1835.8 tokens/s, Avg generation throughput: 2474.3 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:41 [loggers.py:273] Engine 000: Avg prompt throughput: 1881.6 tokens/s, Avg generation throughput: 2527.9 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:16:51 [loggers.py:273] Engine 000: Avg prompt throughput: 1597.5 tokens/s, Avg generation throughput: 2452.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38134 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37818 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:01 [loggers.py:273] Engine 000: Avg prompt throughput: 1124.6 tokens/s, Avg generation throughput: 1948.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:38078 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 78.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:21 [loggers.py:273] Engine 000: Avg prompt throughput: 557.5 tokens/s, Avg generation throughput: 1221.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 31.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:31 [loggers.py:273] Engine 000: Avg prompt throughput: 361.7 tokens/s, Avg generation throughput: 1997.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 44.6% |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:41 [loggers.py:273] Engine 000: Avg prompt throughput: 299.1 tokens/s, Avg generation throughput: 1946.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 51.8% |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:17:51 [loggers.py:273] Engine 000: Avg prompt throughput: 232.0 tokens/s, Avg generation throughput: 1847.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.0%, Prefix cache hit rate: 56.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:01 [loggers.py:273] Engine 000: Avg prompt throughput: 371.1 tokens/s, Avg generation throughput: 1821.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 60.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:11 [loggers.py:273] Engine 000: Avg prompt throughput: 360.8 tokens/s, Avg generation throughput: 1879.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 64.1% |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:21 [loggers.py:273] Engine 000: Avg prompt throughput: 226.5 tokens/s, Avg generation throughput: 1865.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.2%, Prefix cache hit rate: 66.9% |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:31 [loggers.py:273] Engine 000: Avg prompt throughput: 346.8 tokens/s, Avg generation throughput: 1850.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 69.4% |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:41 [loggers.py:273] Engine 000: Avg prompt throughput: 365.7 tokens/s, Avg generation throughput: 1879.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 71.6% |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:18:51 [loggers.py:273] Engine 000: Avg prompt throughput: 259.8 tokens/s, Avg generation throughput: 1927.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 73.5% |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:01 [loggers.py:273] Engine 000: Avg prompt throughput: 411.4 tokens/s, Avg generation throughput: 1896.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 75.6% |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:11 [loggers.py:273] Engine 000: Avg prompt throughput: 214.5 tokens/s, Avg generation throughput: 1904.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.9%, Prefix cache hit rate: 76.9% |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:21 [loggers.py:273] Engine 000: Avg prompt throughput: 374.7 tokens/s, Avg generation throughput: 1878.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 78.4% |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49008 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48756 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48986 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:31 [loggers.py:273] Engine 000: Avg prompt throughput: 167.8 tokens/s, Avg generation throughput: 1858.3 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.0%, Prefix cache hit rate: 79.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:48960 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49024 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48840 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48742 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48814 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48772 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1291.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 79.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48850 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:49042 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48842 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:19:51 [loggers.py:273] Engine 000: Avg prompt throughput: 446.1 tokens/s, Avg generation throughput: 1353.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 79.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:01 [loggers.py:273] Engine 000: Avg prompt throughput: 281.7 tokens/s, Avg generation throughput: 1968.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 79.5% |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:11 [loggers.py:273] Engine 000: Avg prompt throughput: 260.2 tokens/s, Avg generation throughput: 1877.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 79.7% |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:21 [loggers.py:273] Engine 000: Avg prompt throughput: 306.8 tokens/s, Avg generation throughput: 1847.5 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 80.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:31 [loggers.py:273] Engine 000: Avg prompt throughput: 339.3 tokens/s, Avg generation throughput: 1869.6 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.8%, Prefix cache hit rate: 80.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:41 [loggers.py:273] Engine 000: Avg prompt throughput: 327.8 tokens/s, Avg generation throughput: 1678.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 80.5% |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:20:51 [loggers.py:273] Engine 000: Avg prompt throughput: 285.4 tokens/s, Avg generation throughput: 1718.3 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 80.7% |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:01 [loggers.py:273] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 1722.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 81.0% |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:11 [loggers.py:273] Engine 000: Avg prompt throughput: 261.1 tokens/s, Avg generation throughput: 1657.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 81.1% |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:21 [loggers.py:273] Engine 000: Avg prompt throughput: 259.1 tokens/s, Avg generation throughput: 1665.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 81.2% |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52620 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:31 [loggers.py:273] Engine 000: Avg prompt throughput: 182.8 tokens/s, Avg generation throughput: 1550.9 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 80.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:52820 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52684 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52468 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52660 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52522 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 776.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 80.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52556 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:52740 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:21:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 203.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.3% |
| (APIServer pid=112067) INFO 08-08 22:22:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.3% |
| (APIServer pid=112067) INFO: 127.0.0.1:35086 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=112067) INFO: 127.0.0.1:35086 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=112067) INFO: 127.0.0.1:35086 - "GET /tokenizer_info HTTP/1.1" 404 Not Found |
| (APIServer pid=112067) INFO: 127.0.0.1:38432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38514 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38602 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38622 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38652 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38452 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38436 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:22:31 [loggers.py:273] Engine 000: Avg prompt throughput: 1429.9 tokens/s, Avg generation throughput: 296.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 75.6% |
| (APIServer pid=112067) INFO: 127.0.0.1:38634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38680 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38696 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38514 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38658 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38662 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38652 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38602 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38452 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38464 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38518 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38398 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38680 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38592 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38380 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38622 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38544 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38592 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38704 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38398 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38650 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38374 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38398 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38696 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38704 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38436 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38652 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38662 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38658 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38680 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38374 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38602 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38544 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38622 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38380 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38518 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38658 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38398 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38544 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38464 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38704 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38544 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38380 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38578 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38696 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38658 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38514 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38436 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38372 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38702 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38544 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38462 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38478 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38374 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38712 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38452 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38518 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38612 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38624 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38622 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38648 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38356 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38680 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38464 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38634 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38398 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38664 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38662 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38592 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38650 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38388 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38402 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38602 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38704 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38568 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:38652 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:22:41 [loggers.py:273] Engine 000: Avg prompt throughput: 894.8 tokens/s, Avg generation throughput: 787.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 70.2% |
| (APIServer pid=112067) INFO: 127.0.0.1:38530 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:22:51 [loggers.py:273] Engine 000: Avg prompt throughput: 3602.8 tokens/s, Avg generation throughput: 1132.9 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 63.2% |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43858 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43768 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43804 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44002 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43758 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43784 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43922 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43798 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43786 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44028 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43946 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44008 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43936 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43876 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43972 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43986 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43924 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44154 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44184 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44176 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44054 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44016 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44164 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43918 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:44120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO: 127.0.0.1:43940 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=112067) INFO 08-08 22:23:01 [loggers.py:273] Engine 000: Avg prompt throughput: 1945.9 tokens/s, Avg generation throughput: 1576.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=112067) INFO 08-08 22:23:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3% |
| (APIServer pid=112067) INFO 08-08 22:26:29 [launcher.py:100] [shutdown] API server: shutdown triggered |
| (APIServer pid=112067) INFO 08-08 22:26:29 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s |
| (APIServer pid=112067) INFO 08-08 22:26:29 [core_client.py:655] [shutdown] MPClient: start timeout=0s |
| (APIServer pid=112067) INFO 08-08 22:26:29 [core_client.py:657] [shutdown] MPClient: stopping engine manager |
| (APIServer pid=112067) WARNING 08-08 22:26:29 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 |
| (EngineCore pid=112194) INFO 08-08 22:26:29 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM |
| (EngineCore pid=112194) INFO 08-08 22:26:29 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s |
| (EngineCore pid=112194) INFO 08-08 22:26:29 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown |
| (EngineCore pid=112194) INFO 08-08 22:26:29 [core.py:1227] [shutdown] EngineCore: exiting busy loop |
| (APIServer pid=112067) INFO 08-08 22:26:29 [core_client.py:659] [shutdown] MPClient: engine manager stopped |
| (APIServer pid=112067) INFO 08-08 22:26:29 [core_client.py:660] [shutdown] MPClient: cleaning up background resources |
| (APIServer pid=112067) INFO 08-08 22:26:29 [core_client.py:662] [shutdown] MPClient: complete |
| (APIServer pid=112067) INFO 08-08 22:26:29 [launcher.py:125] [shutdown] API server: engine client stopped |
| (APIServer pid=112067) INFO 08-08 22:26:29 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown |
| (APIServer pid=112067) INFO 08-08 22:26:29 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server |
| (APIServer pid=112067) INFO: Shutting down |
| (APIServer pid=112067) INFO: Waiting for application shutdown. |
| (APIServer pid=112067) INFO: Application shutdown complete. |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
|
|
