| --- |
| title: React |
| emoji: 🌍 |
| colorFrom: yellow |
| colorTo: gray |
| sdk: gradio |
| sdk_version: 6.9.0 |
| app_file: app.py |
| pinned: false |
| --- |
| |
| Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
|
|
|
|
| # HF Space Backend (Streaming LLM Server) |
|
|
| This folder contains Hugging Face Space backends for two model deployments that share the same production runtime. |
|
|
| ## Files |
| - `app.py`: Nanbeige deployment entrypoint (`Nanbeige/Nanbeige4.1-3B`) |
| - `main.py`: LiquidAI deployment entrypoint (`LiquidAI/LFM2.5-1.2B-Thinking`) |
| - `server_runtime.py`: shared queue + worker + streaming runtime used by both entrypoints |
| - `index.html`: lightweight local streaming test UI |
| - `requirements.txt`: runtime dependencies |
|
|
| ## Runtime Architecture |
| Both servers use the same execution flow: |
|
|
| Client Request |
| -> FastAPI `/chat` |
| -> `asyncio.Queue` request buffer |
| -> worker pool (`asyncio` tasks) |
| -> concurrency gate (`asyncio.Semaphore`) |
| -> one generation thread per request (`model.generate`) |
| -> per-request `TextIteratorStreamer` |
| -> SSE token stream to client |
|
|
| ### Why this structure |
| - Keeps the event loop responsive. |
| - Prevents response mixing across users (isolated request objects). |
| - Supports controlled concurrency under CPU/GPU. |
| - Queues overflow load instead of hard failing during bursts. |
|
|
| ## Concurrency |
| Hardware-aware worker count: |
| - CPU: `1..4` workers (core-based) |
| - GPU: `3..5` workers (VRAM tier-based) |
|
|
| Override at runtime: |
| - `HF_MAX_WORKERS` |
|
|
| Queue settings: |
| - `HF_QUEUE_MAX_SIZE` (default: `512`) |
|
|
| ## Thread Lifecycle and Safety |
| - Each request gets its own generation thread. |
| - Each request has a cancellation event. |
| - `CancelAwareStoppingCriteria` stops generation when client disconnects/cancels. |
| - Streamer is explicitly ended in `finally` block. |
| - Generation thread is joined with a long timeout (`HF_GENERATION_JOIN_TIMEOUT_SECONDS`, default `180`) to avoid orphaned work. |
|
|
| This fixes the old short-join behavior that produced frequent: |
| - `Generation thread did not finish within timeout` |
|
|
| ## Metrics and Logging |
| Per request logs include: |
| - request queued |
| - worker start/end |
| - first token latency |
| - generated token count |
| - tokens/sec |
| - active workers |
| - queue size |
|
|
| Debug token-by-token logging is optional: |
| - `HF_DEBUG_TOKEN_LOGS=1` |
|
|
| ## API |
| ### `POST /chat` |
| Body: |
| - `messages`: chat messages |
| - `stream`: `true` for SSE streaming |
| - `max_tokens`: max new tokens requested |
| - `temperature`: optional; if omitted model default is used |
| - `tools`: optional tool schemas for chat template |
|
|
| Streaming response format: |
| - SSE `data: {"type":"token","content":"..."}` chunks |
| - final `{"type":"done","content":""}` event |
|
|
| ### `GET /health` |
| Returns: |
| - `status` |
| - `model_loaded` |
| - `device` |
| - `active_workers` |
| - `queue_size` |
| - `max_workers` |
|
|
| ### `GET /index` |
| Serves `index.html` test page. |
|
|
| ## Model-Specific Settings |
| ### `app.py` (Nanbeige4.1-3B) |
| - `max_input_tokens=32768` |
| - `eos_token_id=166101` |
| - `default_temperature=0.6` |
| - `top_p=0.95` |
| - `repetition_penalty=1.0` |
| - `tokenizer_use_fast=False` |
|
|
| ### `main.py` (LFM2.5-1.2B-Thinking) |
| - `max_input_tokens=32768` |
| - `default_temperature=0.1` |
| - `top_p=0.1` |
| - `top_k=50` |
| - `repetition_penalty=1.05` |
| - `eos_token_id` from tokenizer config |
|
|
| ## Environment Variables |
| - `HF_MAX_WORKERS` |
| - `HF_QUEUE_MAX_SIZE` |
| - `HF_STREAMER_TIMEOUT_SECONDS` |
| - `HF_GENERATION_JOIN_TIMEOUT_SECONDS` |
| - `HF_MAX_INPUT_TOKENS` |
| - `HF_MAX_NEW_TOKENS` |
| - `HF_DEBUG_TOKEN_LOGS` |
|
|
| ## Model Documentation References |
| ### Nanbeige / `app.py` |
| - https://huggingface.co/Nanbeige/Nanbeige4.1-3B |
| - https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/README.md |
| - https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/Nanbeige4.1-3B-Report.pdf |
| - https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/generation_config.json |
| - https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/config.json |
| |
| ### LiquidAI / `main.py` |
| - https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking |
| - https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/README.md |
| - https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/chat_template.jinja |
| - https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/config.json |
| - https://docs.liquid.ai/lfm/key-concepts/chat-template |
| - https://docs.liquid.ai/lfm/key-concepts/text-generation-and-prompting |
| - https://docs.liquid.ai/lfm/key-concepts/tool-use |
| - https://huggingface.co/docs/transformers/en/chat_templating#using-applychattemplate |
| |
| ## Notes |
| - Model is loaded once per process during FastAPI lifespan startup. |
| - `index.html` is intentionally a simple streaming test page, not the production frontend. |
| - Both entrypoints (`app.py`, `main.py`) now behave consistently by design. |
