Upload folder using huggingface_hub

README.md

@@ -49,13 +49,15 @@ If the env vars are set on the server and you send the master key in the Authori
 
 **503 "The model is overloaded"** – Returned by the provider (e.g. Gemini). The proxy retries and then follows fallbacks. Clients receive a short error message only (no traceback). Retrying the request later usually works.
 
+**429 Rate limit exceeded** – Returned when a provider (e.g. OpenRouter) throttles requests. The proxy retries and uses fallbacks. Clients receive a short rate-limit message only. If all fallbacks hit rate limits, retry after a short delay.
+
 **Context length exceeded (e.g. 139k > 128k)** – The config uses a large-context model group (`my-large-context`) as the first fallback when context is exceeded. Ensure `OPENROUTER_API_KEY` is set so that fallback can be used. If the request is very long (e.g. 140k+ tokens including tool input), consider trimming history or using a client that supports prompt compression so fallbacks have a chance to succeed.
 
 **Hugging Face: "property 'prefix' is unsupported"** – Some clients (e.g. agent mode) send assistant messages with a `prefix` field. The proxy strips that field from every message in a pre-call hook before any provider sees it, so all providers (including HF on fallback) receive messages without prefix. If you still see this error, ensure the trim-messages hook is loaded in config (callbacks) and the proxy was restarted after config changes.
 
 **Hugging Face: "Credit balance is depleted"** – Your HF Inference credits are used up. Add pre-paid credits or subscribe to PRO; until then, `my-hf-models` fallbacks will fail and the proxy will try other groups (e.g. `my-large-context`, `my-free-models`).
 
-**Request fails after many fallbacks / "tokens lost"** – If the primary model and every fallback fail (context too long, HF credits depleted, Gemini 503, etc.), the request returns an error and no completion is streamed. Clients see a plain error message only. To avoid wasting tokens: (1) keep conversations or tool payloads within context limits where possible, (2) ensure at least one fallback provider (OpenRouter, HF, or Gemini) has quota and supports your message format, (3) retry during peak overload (e.g. Gemini 503).
+**Request fails after many fallbacks / "tokens lost"** – If the primary model and every fallback fail (context too long, HF credits depleted, Gemini 503, rate limits, etc.), the request returns an error and no completion is streamed. Clients see a plain error message only. Server logs may still show full tracebacks for debugging, especially when the failure happens during streaming. To avoid wasting tokens: (1) keep conversations or tool payloads within context limits where possible, (2) ensure at least one fallback provider (OpenRouter, HF, or Gemini) has quota and supports your message format, (3) retry during peak overload (e.g. Gemini 503) or after rate limits reset.
 
 **Client cancels or disconnects** – If the client closes the connection or cancels the request, the proxy may still continue calling the provider until the request finishes. That can use extra tokens until LiteLLM supports cancelling upstream on disconnect.
 

trim_messages_hook.py

@@ -9,6 +9,7 @@ import os
 from typing import TYPE_CHECKING, Any, Literal, Optional
 
 from fastapi import HTTPException
+from litellm import RateLimitError
 from litellm.integrations.custom_logger import CustomLogger
 
 if TYPE_CHECKING:
@@ -98,16 +99,21 @@ class ErrorHandler(CustomLogger):
             _log.error("LLM call failed: %s\n%s", original_exception, traceback_str)
         else:
             _log.error("LLM call failed: %s", original_exception, exc_info=True)
-        status_code = 500
-        message = "An error occurred while processing your request. Please retry."
-        if getattr(original_exception, "response", None) is not None:
-            resp = getattr(original_exception, "response")
-            if getattr(resp, "status_code", None) == 503:
-                status_code = 503
-                message = "Service temporarily unavailable. Please retry."
-        elif "503" in str(original_exception):
+        status_code = getattr(original_exception, "status_code", None)
+        if status_code is not None and 400 <= status_code < 600:
+            pass
+        else:
+            status_code = 500
+        resp = getattr(original_exception, "response", None)
+        if resp is not None and getattr(resp, "status_code", None) == 503:
             status_code = 503
+        if status_code == 503:
             message = "Service temporarily unavailable. Please retry."
+        elif status_code == 429 or isinstance(original_exception, RateLimitError):
+            status_code = 429
+            message = "Rate limit exceeded. Please retry shortly."
+        else:
+            message = "An error occurred while processing your request. Please retry."
         return HTTPException(status_code=status_code, detail=message)