Add local GPU sidecars for tiny-aya + klein (off-ZeroGPU)

Run the polats/tiny-army-{tiny-aya,klein}-zerogpu Spaces locally on the
3090 so the main app doesn't burn HF ZeroGPU quota. Each mirrors its
hosted Space's Gradio API contract, so the main app's gradio_client talks
to them unchanged — only the .env TINY_{AYA,KLEIN}_SPACE vars switch from
Space IDs to local URLs (.env is gitignored, so deploys are unaffected).

- spaces/tiny-aya-zerogpu: spaces.GPU -> no-op shim, 4-bit NF4 load, VRAM
cap, .env/token, host/port launch block.
- spaces/klein-zerogpu: sequential CPU offload (fits the desktop-shared
card; 4-bit OOMs at load-time warmup), weiner LoRA pre-applied + default.
- run_sidecars.sh launcher (start/stop/aya/klein; binds 0.0.0.0 for LAN).
- docs/local-sidecars.md.

This is a local-dev branch; it is not the deployed Space branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/local-sidecars.md

@@ -0,0 +1,65 @@
+# Local model sidecars (no HuggingFace quota)
+
+Two of the hosted ZeroGPU Spaces now run **locally on the 3090** so the main app doesn't burn
+HF quota. They mirror each hosted Space's Gradio API exactly, so the main app's `gradio_client`
+talks to them unchanged — only the `*_SPACE` env var changes from a Space ID to a local URL.
+
+| Sidecar    | Source dir                  | Port | Model                          | `.env` var        |
+|------------|-----------------------------|------|--------------------------------|-------------------|
+| Tiny Aya   | `spaces/tiny-aya-zerogpu/`  | 7864 | `CohereLabs/tiny-aya-global`   | `TINY_AYA_SPACE`  |
+| Klein      | `spaces/klein-zerogpu/`     | 7865 | `black-forest-labs/FLUX.2-klein-4B` + `polats/weiner-klein-lora` | `TINY_KLEIN_SPACE` |
+
+Mirrors of `polats/tiny-army-tiny-aya-zerogpu` and `polats/tiny-army-klein-zerogpu`, adapted for
+local hardware (no ZeroGPU). The hosted `app.py`s are unchanged in spirit; the local diffs are
+documented in each file's header.
+
+## Run
+
+```bash
+./run_sidecars.sh          # start both, wait until ready (logs in logs/)
+./run_sidecars.sh aya      # just tiny-aya
+./run_sidecars.sh klein    # just klein
+./run_sidecars.sh stop     # stop both
+```
+
+`.env` already points `TINY_AYA_SPACE` / `TINY_KLEIN_SPACE` at the local URLs.
+
+### Network access
+
+The sidecars bind to `0.0.0.0` by default, so they're reachable from the **local network** at this
+host's LAN IP (currently `10.0.0.36`, also the tailnet `100.88.3.43`):
+
+- Tiny Aya:  `http://10.0.0.36:7864`
+- Klein:     `http://10.0.0.36:7865`
+
+The main app talks to them over loopback (`127.0.0.1`) for speed regardless. Set `BIND=127.0.0.1`
+before `./run_sidecars.sh` to restrict them to this machine only. (They are NOT on the public
+cloudflared/nginx path that exposes kimodo — LAN/tailnet only.) To go back to the
+hosted ZeroGPU Spaces, comment those two lines back to the `polats/...` Space IDs.
+
+## Why these settings (shared-GPU constraints)
+
+The 3090 also drives the desktop **and** is shared with a separate ~15 GB text-encoder server
+(`kimodo`), leaving only ~5 GB free. So:
+
+- **Tiny Aya** loads in **4-bit NF4** (~7 GB → ~3.5 GB resident). `TINY_AYA_QUANT=bf16` for full
+  precision when the card is free.
+- **Klein** loads in **bf16 with `enable_sequential_cpu_offload()`** — the pipeline stays on CPU
+  and streams to the GPU submodule-by-submodule, so **peak VRAM is only ~2.5 GB** and it never
+  contends with the shared card at load time. (4-bit quant was tried but its load-time GPU warmup
+  allocation OOMs against the tight free headroom; sequential offload sidesteps that and keeps full
+  bf16 quality.) ~11 s/image at 1024². `TINY_KLEIN_OFFLOAD=model` uses the faster
+  whole-component offload (~8 GB peak) when the card is free.
+- The **weiner LoRA** is pre-applied at startup and is the default for the `lora` input, so the
+  main app's `/generate(prompt, seed)` calls (which don't pass a LoRA) get weiner portraits.
+  Other checkpoints: `weiner750`, `weiner500`, `weiner250`.
+
+## Verify
+
+```bash
+# both endpoints, via the main app (which must be running on :7860)
+curl -s -X POST localhost:7860/portrait -H 'content-type: application/json' \
+  -d '{"prompt":"portrait of a knight","engine":"klein"}' -o /tmp/p.webp
+curl -sN -X POST localhost:7860/text/generate/stream -H 'content-type: application/json' \
+  -d '{"model":"tiny-aya-global-zerogpu","user":"hi","max_tokens":20}'
+```

run_sidecars.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Run the Tiny Army model sidecars LOCALLY on the 3090 instead of on ZeroGPU Spaces,
+# so the main app doesn't burn HuggingFace quota.
+#
+# These mirror the hosted Spaces' Gradio API exactly, so the main app talks to them via
+# gradio_client unchanged — just point TINY_AYA_SPACE / TINY_KLEIN_SPACE at the local URLs
+# (already done in .env):
+#   TINY_AYA_SPACE=http://127.0.0.1:7864
+#   TINY_KLEIN_SPACE=http://127.0.0.1:7865
+#
+# Usage:
+#   ./run_sidecars.sh          # start both, stream logs to logs/, wait until ready
+#   ./run_sidecars.sh aya      # just tiny-aya
+#   ./run_sidecars.sh klein    # just klein
+#   ./run_sidecars.sh stop     # stop both
+set -euo pipefail
+cd "$(dirname "$0")"
+
+PY="${PY:-.venv/bin/python}"
+AYA_PORT="${AYA_PORT:-7864}"
+KLEIN_PORT="${KLEIN_PORT:-7865}"
+# Bind address. 0.0.0.0 = reachable from the LAN (other machines on the network) at this host's
+# IP:PORT. Set BIND=127.0.0.1 to restrict to localhost (main app on the same box only).
+BIND="${BIND:-0.0.0.0}"
+mkdir -p logs
+
+start_one() {
+  local name="$1" dir="$2" port="$3"
+  if [ -f "logs/$name.pid" ] && kill -0 "$(cat "logs/$name.pid")" 2>/dev/null; then
+    echo "[$name] already running (pid $(cat "logs/$name.pid")) on :$port"
+    return
+  fi
+  echo "[$name] starting on $BIND:$port (logs/$name.log)"
+  PORT="$port" GRADIO_SERVER_NAME="$BIND" nohup "$PY" "spaces/$dir/app.py" \
+    > "logs/$name.log" 2>&1 &
+  echo $! > "logs/$name.pid"
+}
+
+wait_ready() {
+  local name="$1" port="$2"
+  echo -n "[$name] loading model"
+  for _ in $(seq 1 600); do  # up to ~10 min (cold model download/load)
+    if curl -fsS "http://127.0.0.1:$port/config" >/dev/null 2>&1; then
+      echo " — ready at http://127.0.0.1:$port"
+      return 0
+    fi
+    if [ -f "logs/$name.pid" ] && ! kill -0 "$(cat "logs/$name.pid")" 2>/dev/null; then
+      echo " — DIED, tail of logs/$name.log:"; tail -n 30 "logs/$name.log"; return 1
+    fi
+    echo -n "."; sleep 1
+  done
+  echo " — timed out"; return 1
+}
+
+stop_one() {
+  local name="$1"
+  if [ -f "logs/$name.pid" ]; then
+    kill "$(cat "logs/$name.pid")" 2>/dev/null || true
+    rm -f "logs/$name.pid"
+    echo "[$name] stopped"
+  fi
+}
+
+case "${1:-all}" in
+  stop)  stop_one aya; stop_one klein ;;
+  aya)   start_one aya  tiny-aya-zerogpu "$AYA_PORT";   wait_ready aya  "$AYA_PORT" ;;
+  klein) start_one klein klein-zerogpu   "$KLEIN_PORT"; wait_ready klein "$KLEIN_PORT" ;;
+  all|*)
+    start_one aya  tiny-aya-zerogpu "$AYA_PORT"
+    start_one klein klein-zerogpu   "$KLEIN_PORT"
+    wait_ready aya  "$AYA_PORT"
+    wait_ready klein "$KLEIN_PORT"
+    echo "both sidecars up. point the main app at them (already set in .env)."
+    ;;
+esac

spaces/klein-zerogpu/README.md

@@ -0,0 +1,18 @@
+---
+title: Tiny Army Klein ZeroGPU
+emoji: 🤏
+colorFrom: green
+colorTo: gray
+sdk: gradio
+sdk_version: 5.49.1
+app_file: app.py
+suggested_hardware: zero-a10g
+pinned: false
+license: apache-2.0
+models:
+  - black-forest-labs/FLUX.2-klein-4B
+---
+
+# Tiny Army Klein ZeroGPU
+
+Private ZeroGPU sidecar for Tiny Army portrait generation.

spaces/klein-zerogpu/app.py

@@ -0,0 +1,162 @@
+from __future__ import annotations
+
+# LOCAL SIDECAR variant of the polats/tiny-army-klein-zerogpu Space.
+#
+# Same Gradio API contract as the hosted Space (so the main app's gradio_client talks to it
+# unchanged):
+#   /generate(prompt, seed:int, lora:str) -> image   # returns a PIL image / file path
+#
+# Differences from the hosted ZeroGPU app:
+#   * No ZeroGPU. `@spaces.GPU` is a no-op passthrough.
+#   * FLUX.2-klein is ~16 GB (transformer + text_encoder) and won't fit on the 3090, which is
+#     ALSO shared with a ~15 GB text-encoder server. We load in bf16 on CPU and stream to the GPU
+#     with `enable_sequential_cpu_offload()` (submodule granularity), so peak VRAM is only ~1-2 GB
+#     and the model never lands on the GPU all at once. (4-bit quantization was tried but its
+#     load-time GPU "warmup" allocation spikes above the ~5 GB of free headroom and OOMs; sequential
+#     offload sidesteps that AND keeps full bf16 quality. TINY_KLEIN_OFFLOAD=model uses the faster
+#     component-granularity offload instead — only safe when the card is mostly free.)
+#   * The weiner LoRA (polats/weiner-klein-lora) is loaded by DEFAULT and pre-applied at startup,
+#     so the main app's /generate calls (which don't pass a lora) get weiner portraits.
+#   * Loads the repo .env for HF_TOKEN.
+import os
+import random
+import time
+
+# Reduce CUDA fragmentation under the shared-card memory pressure (the offload churn allocates
+# and frees repeatedly). Must be set before torch initializes CUDA.
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv(os.path.join(os.path.dirname(__file__), "..", "..", ".env"))
+    load_dotenv()
+except Exception:  # noqa: BLE001
+    pass
+
+
+# Off ZeroGPU there's no spaces.GPU; make the decorator a no-op passthrough.
+def GPU(*dargs, **dkwargs):  # noqa: N802
+    def wrap(fn):
+        return fn
+    if len(dargs) == 1 and callable(dargs[0]) and not dkwargs:
+        return dargs[0]
+    return wrap
+
+
+import gradio as gr
+import torch
+from diffusers import Flux2KleinPipeline
+
+MODEL_ID = os.environ.get("TINY_KLEIN_MODEL", "black-forest-labs/FLUX.2-klein-4B")
+STEPS = int(os.environ.get("TINY_KLEIN_STEPS", "4"))
+GUIDANCE = float(os.environ.get("TINY_KLEIN_GUIDANCE", "1.0"))
+MAX_SEED = 2_147_483_647
+_HF_TOKEN = os.environ.get("HF_TOKEN") or None
+# Offload strategy. "sequential" (default) streams submodules → ~1-2 GB peak, fits alongside the
+# shared text-encoder server, slower. "model" streams whole components → ~8 GB peak, faster, only
+# safe when the card is mostly free.
+OFFLOAD = os.environ.get("TINY_KLEIN_OFFLOAD", "sequential").strip().lower()
+
+# Optional character/style LoRAs, loadable per-request via the `lora` argument.
+# Trained on klein-base; serving on the distilled model is the documented fast path.
+LORA_REPO = os.environ.get("KLEIN_LORA_REPO", "polats/weiner-klein-lora")
+LORAS = {
+    "weiner": "weiner_klein_4b_v1.safetensors",          # step 1000 (final)
+    "weiner750": "weiner_klein_4b_v1_000000750.safetensors",
+    "weiner500": "weiner_klein_4b_v1_000000500.safetensors",
+    "weiner250": "weiner_klein_4b_v1_000000250.safetensors",
+}
+# Default LoRA applied at startup (and the component default so no-arg /generate calls use it).
+DEFAULT_LORA = os.environ.get("KLEIN_LORA_DEFAULT", "weiner").strip()
+
+print(f"[klein] loading {MODEL_ID} (bf16, offload={OFFLOAD if torch.cuda.is_available() else 'cpu'})", flush=True)
+_t0 = time.time()
+# bf16 on CPU — no GPU allocation at load time, so it never contends with the shared card.
+pipe = Flux2KleinPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, token=_HF_TOKEN)
+print(f"[klein] loaded in {time.time() - _t0:.1f}s", flush=True)
+
+_lora_active: str | None = None
+_offload_enabled = False
+
+
+def _load_lora(name: str) -> None:
+    pipe.load_lora_weights(LORA_REPO, weight_name=LORAS[name], token=_HF_TOKEN)
+
+
+def _ensure_lora(name: str) -> None:
+    global _lora_active
+    name = (name or "").strip()
+    if name == (_lora_active or ""):
+        return
+    if _lora_active:
+        pipe.unload_lora_weights()
+        _lora_active = None
+    if name:
+        if name not in LORAS:
+            raise gr.Error(f"unknown lora '{name}' (have: {', '.join(LORAS)})")
+        _load_lora(name)
+        _lora_active = name
+
+
+# Pre-apply the default LoRA, THEN install the offload hooks (load_lora before offload is the
+# reliable order). After this the pipe manages its own device placement — never call .to(cuda).
+if DEFAULT_LORA:
+    try:
+        _load_lora(DEFAULT_LORA)
+        _lora_active = DEFAULT_LORA
+        print(f"[klein] applied default LoRA: {DEFAULT_LORA}", flush=True)
+    except Exception as e:  # noqa: BLE001
+        print(f"[klein] WARN: could not preload default LoRA {DEFAULT_LORA!r}: {e}", flush=True)
+
+if torch.cuda.is_available():
+    if OFFLOAD == "model":
+        pipe.enable_model_cpu_offload()  # whole-component granularity, ~8 GB peak, faster
+        print("[klein] enabled model CPU offload (peak ~ one component)", flush=True)
+    else:
+        pipe.enable_sequential_cpu_offload()  # submodule granularity, ~1-2 GB peak, slower
+        print("[klein] enabled sequential CPU offload (peak ~1-2 GB)", flush=True)
+    _offload_enabled = True
+
+
+@GPU(duration=60)
+def generate(prompt: str, seed: int = 42, lora: str = DEFAULT_LORA):
+    if not prompt or not prompt.strip():
+        raise gr.Error("prompt required")
+    _ensure_lora(lora)
+    # With offload enabled the pipeline places its own modules; the generator must live on the
+    # device the latents end up on (cuda when offloading, else cpu).
+    dev = "cuda" if torch.cuda.is_available() else "cpu"
+    if seed is None or int(seed) < 0:
+        seed = random.randint(0, MAX_SEED)
+    img = pipe(
+        prompt=prompt.strip(),
+        width=1024,
+        height=1024,
+        num_inference_steps=STEPS,
+        guidance_scale=GUIDANCE,
+        generator=torch.Generator(device=dev).manual_seed(int(seed)),
+    ).images[0]
+    if dev == "cuda":
+        torch.cuda.empty_cache()
+    return img
+
+
+demo = gr.Interface(
+    fn=generate,
+    inputs=[
+        gr.Textbox(label="Prompt", lines=4),
+        gr.Number(label="Seed", value=42, precision=0),
+        gr.Textbox(label="LoRA", value=DEFAULT_LORA,
+                   placeholder="blank = none; weiner / weiner750 / weiner500 / weiner250"),
+    ],
+    outputs=gr.Image(type="pil", label="Portrait"),
+    api_name="generate",
+    title="Tiny Army Klein — local sidecar",
+)
+
+if __name__ == "__main__":
+    demo.queue().launch(
+        server_name=os.environ.get("GRADIO_SERVER_NAME", "127.0.0.1"),
+        server_port=int(os.environ.get("PORT", "7865")),
+        show_error=True,
+    )

spaces/klein-zerogpu/requirements.txt

@@ -0,0 +1,9 @@
+git+https://github.com/huggingface/diffusers.git
+torch
+transformers
+accelerate
+sentencepiece
+protobuf
+spaces
+peft
+python-dotenv

spaces/tiny-aya-zerogpu/README.md

@@ -0,0 +1,19 @@
+---
+title: Tiny Army Tiny Aya ZeroGPU
+emoji: 🌍
+colorFrom: green
+colorTo: blue
+sdk: gradio
+sdk_version: 6.15.2
+app_file: app.py
+python_version: "3.11"
+suggested_hardware: zero-a10g
+pinned: false
+license: apache-2.0
+models:
+  - CohereLabs/tiny-aya-global
+---
+
+# Tiny Army Tiny Aya ZeroGPU
+
+ZeroGPU sidecar for Tiny Army text generation using Cohere Labs Tiny Aya Global.

spaces/tiny-aya-zerogpu/app.py

@@ -0,0 +1,182 @@
+from __future__ import annotations
+
+# LOCAL SIDECAR variant of the polats/tiny-army-tiny-aya-zerogpu Space.
+#
+# Same Gradio API contract as the hosted Space (so the main app's gradio_client talks to it
+# unchanged):
+#   /generate(system, user, max_tokens:int, temperature:float)        -> str
+#   /generate_stream(system, user, max_tokens:int, temperature:float) -> str   # CUMULATIVE, streamed
+#
+# Differences from the hosted ZeroGPU app:
+#   * No ZeroGPU. `@spaces.GPU` is replaced by a no-op passthrough (the locally-installed
+#     `spaces` package has no `.GPU` off-platform), so generate() just runs on the local CUDA
+#     device the model is already resident on.
+#   * Loads a sibling/repo .env for HF_TOKEN (tiny-aya-global is a gated repo).
+#   * Caps this process's VRAM so a spike can't grab the whole 3090 and crash the desktop.
+import os
+import threading
+
+os.environ.setdefault("OPENBLAS_NUM_THREADS", "4")
+os.environ.setdefault("OMP_NUM_THREADS", "4")
+os.environ.setdefault("MKL_NUM_THREADS", "4")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ.setdefault("GRADIO_SSR_MODE", "false")
+
+try:
+    from dotenv import load_dotenv
+    # repo root .env (two dirs up: spaces/tiny-aya-zerogpu/ -> repo root)
+    load_dotenv(os.path.join(os.path.dirname(__file__), "..", "..", ".env"))
+    load_dotenv()  # also any .env in CWD
+except Exception:  # noqa: BLE001
+    pass
+
+import gradio as gr
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
+
+
+# Off ZeroGPU there's no spaces.GPU; make the decorator a no-op passthrough. The model is
+# already on the local CUDA device, so the wrapped fn runs there directly.
+def GPU(*dargs, **dkwargs):  # noqa: N802
+    def wrap(fn):
+        return fn
+    if len(dargs) == 1 and callable(dargs[0]) and not dkwargs:
+        return dargs[0]
+    return wrap
+
+
+MODEL_ID = os.environ.get("TINY_AYA_MODEL", "CohereLabs/tiny-aya-global")
+DEFAULT_MAX_TOKENS = int(os.environ.get("TINY_AYA_MAX_TOKENS", "400"))
+_HF_TOKEN = os.environ.get("HF_TOKEN") or None
+# 4-bit by default: the 3090 is shared with a ~15 GB text-encoder server, so we NF4-quantize
+# tiny-aya (~7 GB bf16 -> ~2.5 GB) to leave room. TINY_AYA_QUANT=bf16 disables it (best quality).
+QUANT = os.environ.get("TINY_AYA_QUANT", "4bit").strip().lower()
+
+# GUARDRAIL: the 3090 also drives the desktop. Cap THIS process's share of the card so an OOM
+# errors out instead of taking down the display.
+if torch.cuda.is_available():
+    try:
+        torch.cuda.set_per_process_memory_fraction(
+            float(os.environ.get("TINY_AYA_VRAM_FRAC", "0.5")), 0
+        )
+    except Exception:  # noqa: BLE001
+        pass
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=_HF_TOKEN)
+
+
+def _load_model():
+    if not torch.cuda.is_available():
+        return AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", token=_HF_TOKEN)
+    if QUANT == "bf16":
+        m = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, token=_HF_TOKEN)
+        return m.to("cuda")
+    # 4-bit NF4, loaded straight onto the GPU (bnb models can't be .to()'d afterwards).
+    from transformers import BitsAndBytesConfig
+    quant = BitsAndBytesConfig(
+        load_in_4bit=True,
+        bnb_4bit_quant_type="nf4",
+        bnb_4bit_compute_dtype=torch.bfloat16,
+        bnb_4bit_use_double_quant=True,
+    )
+    return AutoModelForCausalLM.from_pretrained(
+        MODEL_ID, quantization_config=quant, torch_dtype=torch.bfloat16,
+        device_map="cuda", token=_HF_TOKEN,
+    )
+
+
+print(f"[tiny-aya] loading {MODEL_ID} quant={QUANT if torch.cuda.is_available() else 'cpu'}", flush=True)
+model = _load_model()
+model.eval()
+print("[tiny-aya] model ready", flush=True)
+
+_lock = threading.Lock()
+
+
+def _messages(system: str, user: str):
+    messages = []
+    if system and system.strip():
+        messages.append({"role": "system", "content": system.strip()})
+    messages.append({"role": "user", "content": (user or "").strip()})
+    return messages
+
+
+@GPU(duration=120)
+def generate(system: str, user: str, max_tokens: int = DEFAULT_MAX_TOKENS, temperature: float = 0.8):
+    if not user or not user.strip():
+        raise gr.Error("user prompt required")
+    max_tokens = max(1, min(int(max_tokens or DEFAULT_MAX_TOKENS), 1024))
+    temperature = max(0.0, min(float(temperature if temperature is not None else 0.8), 2.0))
+    inputs = tokenizer.apply_chat_template(
+        _messages(system, user),
+        tokenize=True,
+        add_generation_prompt=True,
+        return_dict=True,
+        return_tensors="pt",
+    ).to(model.device)
+    with _lock, torch.inference_mode():
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=max_tokens,
+            do_sample=temperature > 0,
+            temperature=max(temperature, 1e-5),
+            top_p=0.95,
+            pad_token_id=tokenizer.eos_token_id,
+        )
+    return tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()
+
+
+@GPU(duration=120)
+def generate_stream(system: str, user: str, max_tokens: int = DEFAULT_MAX_TOKENS, temperature: float = 0.8):
+    if not user or not user.strip():
+        raise gr.Error("user prompt required")
+    max_tokens = max(1, min(int(max_tokens or DEFAULT_MAX_TOKENS), 1024))
+    temperature = max(0.0, min(float(temperature if temperature is not None else 0.8), 2.0))
+    inputs = tokenizer.apply_chat_template(
+        _messages(system, user),
+        tokenize=True,
+        add_generation_prompt=True,
+        return_dict=True,
+        return_tensors="pt",
+    ).to(model.device)
+    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
+
+    def run():
+        with _lock, torch.inference_mode():
+            model.generate(
+                **inputs,
+                max_new_tokens=max_tokens,
+                do_sample=temperature > 0,
+                temperature=max(temperature, 1e-5),
+                top_p=0.95,
+                pad_token_id=tokenizer.eos_token_id,
+                streamer=streamer,
+            )
+
+    thread = threading.Thread(target=run, daemon=True)
+    thread.start()
+    acc = ""
+    for token in streamer:
+        acc += token
+        yield acc
+    thread.join(timeout=1)
+
+
+with gr.Blocks(title="Tiny Army Tiny Aya (local)") as demo:
+    gr.Markdown("# Tiny Army Tiny Aya — local sidecar")
+    system = gr.Textbox(label="System", lines=5)
+    user = gr.Textbox(label="User", lines=5)
+    max_tokens = gr.Slider(1, 1024, value=DEFAULT_MAX_TOKENS, step=1, label="Max new tokens")
+    temperature = gr.Slider(0, 2, value=0.8, step=0.05, label="Temperature")
+    btn = gr.Button("Generate")
+    out = gr.Textbox(label="Output", lines=10)
+    btn.click(generate, inputs=[system, user, max_tokens, temperature], outputs=out, api_name="generate")
+    btn.click(generate_stream, inputs=[system, user, max_tokens, temperature], outputs=out, api_name="generate_stream")
+
+
+if __name__ == "__main__":
+    demo.queue().launch(
+        server_name=os.environ.get("GRADIO_SERVER_NAME", "127.0.0.1"),
+        server_port=int(os.environ.get("PORT", "7864")),
+        show_error=True,
+    )

spaces/tiny-aya-zerogpu/requirements.txt

@@ -0,0 +1,10 @@
+gradio==6.15.2
+spaces
+torch
+transformers>=5.4.0
+accelerate
+sentencepiece
+protobuf
+huggingface_hub
+python-dotenv
+bitsandbytes