Spaces:

Sumkh
/

AgenticRAG

Sleeping

Sumkh commited on Feb 25, 2025

Commit

19ad6b0

verified ·

1 Parent(s): 5c87648

Update start.sh

Files changed (1) hide show

start.sh CHANGED Viewed

@@ -14,6 +14,7 @@ vllm serve unsloth/llama-3-70b-Instruct-bnb-4bit \
   --chat-template examples/tool_chat_template_llama3.1_json.jinja \
   --quantization bitsandbytes \
   --load-format bitsandbytes \
   --enforce-eager &
 # Wait to ensure the vLLM server is fully started (adjust if needed)

   --chat-template examples/tool_chat_template_llama3.1_json.jinja \
   --quantization bitsandbytes \
   --load-format bitsandbytes \
+  --gpu_memory_utilization 0.9 \
   --enforce-eager &
 # Wait to ensure the vLLM server is fully started (adjust if needed)