Hugging Face's logo

Spaces:
Modelflare
/
blog
Running

App Files Community
blog / api-recipes-streaming-fallback.html
modelflaredev's picture
modelflaredev
Upload 11 files
e6f9c10 verified
Raw History Blame Contribute Delete
14.6 kB
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Modelflare API recipes: streaming, routing fallback, images, and async video | Modelflare Blog</title>
<meta name="description" content="Request examples for streaming chat, gateway-side fallback with group failover and Smart API Keys, client-side model fallback, image generation, and the asynchronous video task flow."><meta property="og:image" content="images/api-recipes-streaming-fallback-cover.jpg">
<style>
:root{--bg:#0b1020;--card:#121a2f;--text:#e8ecf7;--muted:#9aa6c2;--accent:#8b5cf6;--accent2:#ec4899;--border:rgba(255,255,255,.08)}
*{box-sizing:border-box}
body{margin:0;font-family:ui-sans-serif,system-ui,-apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif;background:radial-gradient(1200px 600px at 10% -10%,#2a1850 0%,transparent 50%),radial-gradient(900px 500px at 100% 0%,#3b0f2e 0%,transparent 45%),var(--bg);color:var(--text);min-height:100vh;line-height:1.65}
.wrap{max-width:860px;margin:0 auto;padding:28px 18px 64px}
header.top{display:flex;flex-wrap:wrap;gap:14px;align-items:center;justify-content:space-between;margin-bottom:28px;font-size:14px}
header.top a{color:var(--muted);text-decoration:none}header.top a:hover{color:#fff}
header.top .brand{color:#fff;font-weight:700;font-size:16px}
header.top nav{display:flex;flex-wrap:wrap;gap:14px}
h1{font-size:clamp(26px,4.6vw,38px);line-height:1.2;margin:0 0 12px}
h2{font-size:22px;margin:36px 0 10px}h3{font-size:17px;margin:24px 0 8px;color:#d6c7ff}
p,li{color:#cfd6e8}a{color:#c4b5fd}
.lead{color:var(--muted);font-size:17px}
article img{max-width:100%;height:auto;border-radius:12px;border:1px solid var(--border);display:block;margin:18px auto 6px}
article p em:only-child{display:block;text-align:center;color:var(--muted);font-size:14px}
pre{background:#070b16;border:1px solid var(--border);border-radius:12px;padding:0;overflow-x:auto}
pre code.hljs{background:transparent;padding:14px 16px;font-size:13.5px;line-height:1.55}
code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.92em}
p code,li code,td code{background:rgba(139,92,246,.14);padding:1px 5px;border-radius:5px}
table{border-collapse:collapse;width:100%;font-size:14px;margin:14px 0;display:block;overflow-x:auto}
th,td{border:1px solid var(--border);padding:8px 10px;text-align:left;vertical-align:top}th{background:var(--card)}
hr{border:0;border-top:1px solid var(--border);margin:32px 0}
.list{display:grid;gap:14px}
.card{display:flex;gap:16px;background:var(--card);border:1px solid var(--border);border-radius:16px;padding:18px;text-decoration:none;color:inherit}
.card:hover{border-color:rgba(139,92,246,.5)}
.card img{width:200px;height:105px;object-fit:cover;border-radius:10px;flex-shrink:0}
.card h2{font-size:18px;margin:0 0 6px;color:#fff}.card p{margin:0;color:var(--muted);font-size:14px}
footer{margin-top:48px;padding-top:18px;border-top:1px solid var(--border);color:var(--muted);font-size:13px}
footer a{color:var(--muted)}
@media(max-width:600px){.card{flex-direction:column}.card img{width:100%;height:auto;aspect-ratio:1200/630}}
</style>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@highlightjs/cdn-assets@11.9.0/styles/github-dark.min.css"><script src="https://cdn.jsdelivr.net/npm/@highlightjs/cdn-assets@11.9.0/highlight.min.js"></script><script>document.addEventListener("DOMContentLoaded",()=>hljs.highlightAll());</script>
</head>
<body><div class="wrap"><header class="top"><a class="brand" href="index.html">Modelflare Blog</a><nav><a href="https://modelflare.dev/?utm_source=huggingface&utm_medium=space&utm_campaign=blog">modelflare.dev</a><a href="https://docs.modelflare.dev/?utm_source=huggingface&utm_medium=space&utm_campaign=blog">Docs</a><a href="https://castaly.modelflare.dev/?utm_source=huggingface&utm_medium=space&utm_campaign=blog">Castaly</a><a href="https://huggingface.co/Modelflare" target="_top">Hugging Face org</a></nav></header>
<article><h1>Modelflare API recipes: streaming, routing fallback, images, and async video</h1>
<p class="lead">Request examples for streaming chat, gateway-side fallback with group failover and Smart API Keys, client-side model fallback, image generation, and the asynchronous video task flow.</p>
<img src="images/api-recipes-streaming-fallback-cover.jpg" alt="Diagram: parallel routes converging with fallback paths">
<hr />
<p>The recipes below use <a href="https://modelflare.dev/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">Modelflare</a>'s OpenAI-compatible base URL, <code>https://modelflare.dev/v1</code>, and a placeholder key. Each model ID must be enabled on the key's routing group; check <code>GET /v1/models</code> or <a href="https://modelflare.dev/pricing?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">Models &amp; Pricing</a> before sending requests.</p>
<pre><code class="language-python">from openai import OpenAI
client = OpenAI(api_key=&quot;YOUR_MODELFLARE_API_KEY&quot;, base_url=&quot;https://modelflare.dev/v1&quot;)
</code></pre>
<h2>Streaming chat completions</h2>
<p>The streaming request follows the <a href="https://huggingface.co/datasets/Modelflare/api-examples" target="_top">Modelflare/api-examples</a> dataset: it sets <code>"stream": true</code> on Chat Completions and consumes server-sent events the same way the OpenAI SDK does with <code>stream=True</code>. The Modelflare documentation does not include a separate streaming example.</p>
<pre><code class="language-python">stream = client.chat.completions.create(
 model=&quot;gpt-5.6-sol&quot;,
 messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Say hi in one line&quot;}],
 stream=True,
)
for chunk in stream:
 delta = chunk.choices[0].delta.content if chunk.choices else None
 if delta:
 print(delta, end=&quot;&quot;, flush=True)
</code></pre>
<p>The equivalent curl request:</p>
<pre><code class="language-bash">curl -N https://modelflare.dev/v1/chat/completions \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot; \
 -H &quot;Content-Type: application/json&quot; \
 -d '{&quot;model&quot;:&quot;gpt-5.6-sol&quot;,&quot;messages&quot;:[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:&quot;Say hi in one line&quot;}],&quot;stream&quot;:true}'
</code></pre>
<p>Streaming interacts with failover in one specific way. Once a streamed response has written content bytes to the client, the gateway does not replay the request on another route; heartbeats alone do not count as content. A stream that fails after content has started must be retried by the client.</p>
<h2>Fallback at the gateway</h2>
<p>Modelflare implements fallback in the routing layer, so the request body does not change.</p>
<p>An ordinary key names a primary routing group and, optionally, fallback groups in priority order. The chain is configured in API Keys and can be reordered there. The gateway moves to the next group when the current group has no channel for the requested model, when the group is at its rate limit for the account, or when every channel priority in the group has returned a retryable error. It does not fail over on client errors (malformed JSON, an invalid parameter, context length exceeded), on a cancelled client connection, on a request pinned to one channel, or on a stream that has already sent content. The request is billed at the multiplier of the group that served it, and that group is recorded in the consume log. A key whose primary group is a limited-time campaign route must have at least one fallback group before it can be saved.</p>
<p>A Smart API Key is bound to <code>auto</code>. On each request it collects the groups the account can use, keeps those that serve the requested model, orders them by the key's strategy (Balanced, the default; Stability first; or Low price first), and tries them in order. Smart API Keys do not take a manual fallback list. Client code is unchanged: same base URL, same <code>Authorization: Bearer</code> header.</p>
<p><img alt="Diagram: primary group followed by two fallback groups, with the conditions that trigger and stop failover" src="images/fig-failover.svg" />
<em>Group failover on an ordinary key: when the gateway moves to the next group, and when it stops.</em></p>
<h2>Fallback across models in client code</h2>
<p>Switching to a different model after a failure is application logic, not a gateway feature. A minimal version:</p>
<pre><code class="language-python">from openai import APIStatusError
MODELS = [&quot;gpt-5.6-sol&quot;, &quot;YOUR_SECOND_MODEL_ID&quot;] # exact IDs your key can access
def chat_with_fallback(messages):
 last = None
 for m in MODELS:
 try:
 return client.chat.completions.create(model=m, messages=messages)
 except APIStatusError as e:
 last = e
 if e.status_code in (400, 401, 403): # bad request, bad key, model/group not allowed
 raise
 raise last
</code></pre>
<p>Status codes 400, 401, and 403 indicate a malformed request, an invalid key, or a model or group the key may not use. They are raised immediately rather than masked by a retry.</p>
<h2>Image generation</h2>
<pre><code class="language-bash">curl https://modelflare.dev/v1/images/generations \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot; \
 -H &quot;Content-Type: application/json&quot; \
 -d '{
 &quot;model&quot;: &quot;gpt-image-2&quot;,
 &quot;prompt&quot;: &quot;A white cat runs through a neon-lit street in the rain&quot;,
 &quot;size&quot;: &quot;1024x1024&quot;, &quot;n&quot;: 1
 }'
</code></pre>
<p><code>gpt-image-2</code> returns Base64 image data in <code>data[].b64_json</code>. This endpoint serves GPT Images, Wan Image, and Seedream; the accepted fields vary by model and are listed on each <a href="https://docs.modelflare.dev/models/image/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">image model page</a>. Gemini image models use the native <code>POST /v1/models/{model}:generateContent</code> endpoint instead. Requests sent to the wrong family are rejected with <code>400 unsupported_endpoint</code>. Use a Smart API Key or a key on the <code>media</code> group.</p>
<h2>Asynchronous video</h2>
<p><img alt="Diagram: video task created, then queued, in_progress, and completed or failed, followed by download" src="images/fig-video-states.svg" />
<em>Video task states returned by GET /v1/videos/{task_id}.</em></p>
<p>Video generation is a three-step task: create, poll, download.</p>
<pre><code class="language-bash"># 1) create a task (Seedance example; the body differs per family)
curl https://modelflare.dev/v1/videos \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot; \
 -H &quot;Content-Type: application/json&quot; \
 -d '{
 &quot;model&quot;: &quot;seedance-2.5&quot;,
 &quot;content&quot;: [{&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;A cat walks through a sunlit room&quot;}],
 &quot;resolution&quot;: &quot;720p&quot;,
 &quot;duration&quot;: 5,
 &quot;ratio&quot;: &quot;16:9&quot;
 }'
# save &quot;id&quot; from the response (compatibility responses may use &quot;task_id&quot;)
# 2) poll while status is &quot;queued&quot; or &quot;in_progress&quot;; stop at &quot;completed&quot; or &quot;failed&quot;
curl https://modelflare.dev/v1/videos/TASK_ID \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot;
# 3) download
curl --fail --location \
 https://modelflare.dev/v1/videos/TASK_ID/content \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot; \
 --output seedance-result.mp4
</code></pre>
<p>Task creation can return an ID before provider validation finishes, so a later poll may report <code>failed</code>; keep the request ID for investigation. The request body depends on the model family (Seedance, Wan, Kling, MiniMax, HappyHorse) and is documented on each <a href="https://docs.modelflare.dev/models/video/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">video model page</a>. Result URLs can expire, so store the file promptly.</p>
<h2>Responses API</h2>
<p><code>POST /v1/responses</code> is the endpoint used by Codex and Grok Build. The documentation does not show a raw request body; the request below comes from the <a href="https://huggingface.co/datasets/Modelflare/api-examples" target="_top">Modelflare/api-examples</a> dataset.</p>
<pre><code class="language-bash">curl https://modelflare.dev/v1/responses \
 -H &quot;Authorization: Bearer YOUR_MODELFLARE_API_KEY&quot; \
 -H &quot;Content-Type: application/json&quot; \
 -d '{&quot;model&quot;: &quot;gpt-5.6-sol&quot;, &quot;input&quot;: &quot;Reply with one sentence confirming the model.&quot;}'
</code></pre>
<h2>References</h2>
<p>The <a href="https://huggingface.co/spaces/Modelflare/api-recipes" target="_top">Modelflare/api-recipes</a> Space and the <a href="https://huggingface.co/datasets/Modelflare/api-examples" target="_top">Modelflare/api-examples</a> dataset contain these requests. Background reading: <a href="https://docs.modelflare.dev/guides/endpoints/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">Which endpoint to call</a>, <a href="https://docs.modelflare.dev/guides/features/group-failover/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">Group failover</a>, and <a href="https://docs.modelflare.dev/guides/features/smart-api-key/?utm_source=huggingface&amp;utm_medium=space&amp;utm_campaign=api-recipes-streaming-fallback">Smart API Key</a>. Higher limits or volume arrangements: <a href="mailto:business@modelflare.dev">business@modelflare.dev</a>.</p></article><p><a href="index.html">&larr; All articles</a></p>
<footer>Modelflare on Hugging Face: <a href="https://huggingface.co/spaces/Modelflare/start" target="_top">start</a> 路 <a href="https://huggingface.co/spaces/Modelflare/openai-sdk" target="_top">openai-sdk</a> 路 <a href="https://huggingface.co/spaces/Modelflare/coding-agents" target="_top">coding-agents</a> 路 <a href="https://huggingface.co/spaces/Modelflare/api-recipes" target="_top">api-recipes</a> 路 <a href="https://huggingface.co/datasets/Modelflare/api-examples" target="_top">api-examples dataset</a><br>Business: <a href="mailto:business@modelflare.dev">business@modelflare.dev</a></footer></div></body>
</html>