MiniCPM5 ZeroGPU migration

.env.example

@@ -1,6 +1,10 @@
 EXA_API_KEY=
 FIRECRAWL_API_KEY=
-BORDERLESS_MODEL_ID=Qwen/Qwen3.6-27B
+HF_TOKEN=
+BORDERLESS_MODEL_ID=openbmb/MiniCPM5-1B
+BORDERLESS_INFERENCE_MODE=local
+BORDERLESS_ENABLE_THINKING=1
+BORDERLESS_GPU_DURATION=120
 BORDERLESS_MAX_TOOL_ROUNDS=7
 BORDERLESS_TRACE_DIR=agent_traces
 # Set to 1 if you do not want local JSONL trace logs.

README.md

@@ -8,7 +8,7 @@ sdk_version: 6.16.0
 app_file: app.py
 pinned: false
 license: apache-2.0
-hardware: cpu-basic
+hardware: zero-gpu
 short_description: Agentic immigration research for global movers
 tags:
   - agents
@@ -17,12 +17,13 @@ tags:
   - travel
   - research
   - tool-use
-  - qwen
+  - minicpm
+  - openbmb
   - langgraph
   - maplibre
   - geospatial
 models:
-  - Qwen/Qwen3.6-27B
+  - openbmb/MiniCPM5-1B
 datasets: []
 hf_oauth: true
 hf_oauth_scopes:
@@ -54,7 +55,7 @@ No forms to decode. No keyword guessing. Just a research session that meets you
 
 ## How it works
 
-Borderless uses a **[LangGraph](https://docs.langchain.com/oss/python/langgraph/overview)** workflow tuned for **[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)** (27B parameters — within the hackathon's 32B cap):
+Borderless uses a **[LangGraph](https://docs.langchain.com/oss/python/langgraph/overview)** workflow powered by **[openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B)** running locally on **ZeroGPU** (1B parameters):
 
 ```
 User profile → Planner → parallel Researchers (one per to-do) → Consolidator → final answer
@@ -78,7 +79,7 @@ Progress streams live in the chat: thinking steps, the research plan, tool calls
 | `crawl_web_site` | Multiple pages from an official immigration website section (Firecrawl) |
 | `update_globe` | Marks, highlights, and flies to countries on the MapLibre globe |
 
-Sign in with your Hugging Face account to run inference through the Inference API.
+Sign in with your Hugging Face account for the app OAuth flow. LLM inference runs locally on ZeroGPU when `BORDERLESS_INFERENCE_MODE=local` (default); set `HF_TOKEN` as a Space secret for model download.
 
 ## Features
 
@@ -100,7 +101,7 @@ Sign in with your Hugging Face account to run inference through the Inference AP
 
 - **[Gradio Server](https://gradio.app)** — custom HTML/JS UI, OAuth, and streaming API endpoints
 - **[LangGraph](https://docs.langchain.com/oss/python/langgraph/overview)** — planner / parallel researcher / consolidator workflow
-- **[Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)** — planning and synthesis via Hugging Face Inference API (OpenAI-compatible router)
+- **[MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B)** — local planning and synthesis via transformers on ZeroGPU
 - **[MapLibre GL JS](https://maplibre.org/)** — interactive 3D globe
 - **[Exa](https://exa.ai)** — neural web search for discovering immigration sources
 - **[Firecrawl](https://firecrawl.dev)** — scrape and crawl official web pages
@@ -141,7 +142,7 @@ data/
 
 | Constraint | Borderless |
 |------------|------------|
-| Model ≤ 32B | Qwen3.6-27B (27B) |
+| Model ≤ 32B | MiniCPM5-1B (1B) |
 | Gradio on HF Spaces | Yes — [live Space](https://huggingface.co/spaces/build-small-hackathon/borderless) |
 | Agentic | LangGraph multi-agent research with visible tool traces |
 | Sharing is Caring | JSONL tool traces can be sanitized and published — see `TRACE_SHARING.md` |
@@ -157,15 +158,19 @@ cp .env.example .env   # then fill in API keys
 python app.py
 ```
 
-Set a Hugging Face token with Inference API access, or sign in through the app's OAuth flow when deployed.
+Set `HF_TOKEN` in `.env` for model download. For router-based inference (no local GPU), set `BORDERLESS_INFERENCE_MODE=router` and sign in through OAuth.
 
 For web research tools, set API keys from [dashboard.exa.ai](https://dashboard.exa.ai/api-keys) and [firecrawl.dev](https://firecrawl.dev):
 
 | Variable | Purpose |
 |----------|---------|
+| `HF_TOKEN` | Hugging Face Hub login + MiniCPM weight download |
 | `EXA_API_KEY` | `search_immigration_info` |
 | `FIRECRAWL_API_KEY` | `scrape_web_page`, `crawl_web_site` |
-| `BORDERLESS_MODEL_ID` | Model override (default `Qwen/Qwen3.6-27B`) |
+| `BORDERLESS_MODEL_ID` | Model override (default `openbmb/MiniCPM5-1B`) |
+| `BORDERLESS_INFERENCE_MODE` | `local` (default, ZeroGPU) or `router` (HF Inference API) |
+| `BORDERLESS_ENABLE_THINKING` | `1` (default) enables MiniCPM thinking mode in chat template |
+| `BORDERLESS_GPU_DURATION` | ZeroGPU quota reservation per chat run (default `120`) |
 | `BORDERLESS_AGENT_MODE` | `graph` (default) or `legacy` for the single-agent loop |
 | `BORDERLESS_TRACE_DIR` | JSONL trace output directory |
 | `BORDERLESS_DISABLE_TRACE_LOGS` | Set to `1` to disable local trace logs |
@@ -174,4 +179,4 @@ On Hugging Face Spaces, add API keys as **Space secrets** (Settings → Secrets)
 
 ## License
 
-Apache-2.0 (model: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B))
+Apache-2.0 (model: [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B))

requirements.txt

@@ -6,6 +6,7 @@ openai==2.41.0
 langchain-core==1.4.6
 langchain-openai==1.3.0
 langgraph==1.2.4
+transformers==4.57.3
 mcp==1.27.2
 
 # Tools / APIs

tests/test_llm_wrapper.py

@@ -0,0 +1,51 @@
+# tests/test_llm_wrapper.py
+from __future__ import annotations
+
+import os
+import unittest
+from unittest.mock import patch
+
+os.environ.setdefault("BORDERLESS_INFERENCE_MODE", "router")
+os.environ.setdefault("BORDERLESS_PRELOAD_MODEL", "0")
+
+from langchain_openai import ChatOpenAI
+
+from ui.agent.graph.llm import MiniCPMChatModel, build_llm
+
+
+class BuildLlmTests(unittest.TestCase):
+    @patch("ui.agent.graph.llm.local_inference_enabled", return_value=True)
+    def test_local_mode_returns_minicpm_wrapper(self, _mock_local: unittest.mock.Mock) -> None:
+        llm = build_llm({"configurable": {"max_tokens": 512, "temperature": 0.2, "top_p": 0.8}})
+        self.assertIsInstance(llm, MiniCPMChatModel)
+        self.assertEqual(llm.max_tokens, 512)
+
+    @patch("ui.agent.graph.llm.local_inference_enabled", return_value=False)
+    def test_router_mode_returns_chat_openai(self, _mock_local: unittest.mock.Mock) -> None:
+        llm = build_llm(
+            {
+                "configurable": {
+                    "hf_token": "test-token",
+                    "max_tokens": 512,
+                    "temperature": 0.2,
+                    "top_p": 0.8,
+                }
+            }
+        )
+        self.assertIsInstance(llm, ChatOpenAI)
+
+    @patch("ui.agent.graph.llm.local_inference_enabled", return_value=True)
+    def test_minicpm_bind_tools_preserves_generation_params(
+        self,
+        _mock_local: unittest.mock.Mock,
+    ) -> None:
+        llm = MiniCPMChatModel(max_tokens=128, temperature=0.1, top_p=0.5)
+        bound = llm.bind_tools(
+            [{"type": "function", "function": {"name": "search_immigration_info"}}]
+        )
+        self.assertEqual(bound.max_tokens, 128)
+        self.assertEqual(len(bound.tools), 1)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_minicpm_messages.py

@@ -0,0 +1,69 @@
+# tests/test_minicpm_messages.py
+from __future__ import annotations
+
+import json
+import os
+import unittest
+
+os.environ.setdefault("BORDERLESS_INFERENCE_MODE", "router")
+os.environ.setdefault("BORDERLESS_PRELOAD_MODEL", "0")
+
+from langchain_core.messages import AIMessage, ToolMessage
+
+from ui.agent.minicpm.messages import (
+    append_tool_instructions,
+    normalize_messages,
+)
+
+
+class NormalizeMessagesTests(unittest.TestCase):
+    def test_dict_messages(self) -> None:
+        messages = [
+            {"role": "system", "content": "You are helpful."},
+            {"role": "user", "content": "Hello"},
+        ]
+        normalized = normalize_messages(messages)
+        self.assertEqual(normalized[0]["role"], "system")
+        self.assertEqual(normalized[1]["content"], "Hello")
+
+    def test_tool_message_becomes_user_context(self) -> None:
+        messages = [
+            ToolMessage(content='{"results": []}', tool_call_id="call-1"),
+        ]
+        normalized = normalize_messages(messages)
+        self.assertEqual(normalized[0]["role"], "user")
+        self.assertIn("Tool result", normalized[0]["content"])
+
+    def test_ai_message(self) -> None:
+        messages = [AIMessage(content="Done.")]
+        normalized = normalize_messages(messages)
+        self.assertEqual(normalized[0]["role"], "assistant")
+        self.assertEqual(normalized[0]["content"], "Done.")
+
+
+class ToolInstructionTests(unittest.TestCase):
+    def test_appends_tool_block_to_system(self) -> None:
+        tools = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "search_immigration_info",
+                    "description": "Search immigration sources",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {"query": {"type": "string"}},
+                    },
+                },
+            }
+        ]
+        messages = [
+            {"role": "system", "content": "Base prompt"},
+            {"role": "user", "content": "Research Canada"},
+        ]
+        updated = append_tool_instructions(messages, tools)
+        self.assertIn("search_immigration_info", updated[0]["content"])
+        self.assertIn(json.dumps(tools[0]["function"]["parameters"]), updated[0]["content"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_research_findings.py

@@ -2,9 +2,13 @@
 from __future__ import annotations
 
 import json
+import os
 import unittest
 from unittest.mock import patch
 
+os.environ.setdefault("BORDERLESS_INFERENCE_MODE", "router")
+os.environ.setdefault("BORDERLESS_PRELOAD_MODEL", "0")
+
 from langchain_core.messages import AIMessage, ToolMessage
 
 from ui.agent.graph.nodes.helpers import extract_assistant_text, research_tool_calls

ui/agent/config.py

@@ -6,8 +6,23 @@ import os
 from .tool_schemas import TOOL_SCHEMAS
 from .tool_schemas import think as THINK_SCHEMA
 
-MODEL_ID = os.environ.get("BORDERLESS_MODEL_ID", "Qwen/Qwen3.6-27B")
+MODEL_ID = os.environ.get("BORDERLESS_MODEL_ID", "openbmb/MiniCPM5-1B")
+INFERENCE_MODE = os.environ.get("BORDERLESS_INFERENCE_MODE", "local").strip().lower()
+ENABLE_THINKING = os.environ.get("BORDERLESS_ENABLE_THINKING", "1").strip().lower() in {
+    "1",
+    "true",
+    "yes",
+}
+GPU_DURATION = int(os.environ.get("BORDERLESS_GPU_DURATION", "120"))
 MAX_TOOL_ROUNDS = int(os.environ.get("BORDERLESS_MAX_TOOL_ROUNDS", "7"))
 
 TOOLS = TOOL_SCHEMAS
 THINK_TOOLS = [THINK_SCHEMA]
+
+
+def local_inference_enabled() -> bool:
+    return INFERENCE_MODE == "local"
+
+
+def hub_token_available() -> bool:
+    return bool(os.environ.get("HF_TOKEN", "").strip())

ui/agent/graph/llm.py

@@ -1,26 +1,68 @@
 # ui/agent/graph/llm.py
 from __future__ import annotations
 
+from dataclasses import dataclass, field
 from typing import Any
 
+from langchain_core.messages import AIMessage
 from langchain_core.runnables import RunnableConfig
 from langchain_openai import ChatOpenAI
 
-from ..config import MODEL_ID
+from ..config import MODEL_ID, local_inference_enabled
 
 HF_ROUTER_BASE_URL = "https://router.huggingface.co/v1"
 
 
-def build_llm(config: RunnableConfig, **overrides: Any) -> ChatOpenAI:
+@dataclass
+class MiniCPMChatModel:
+    max_tokens: int = 1800
+    temperature: float = 0.35
+    top_p: float = 0.9
+    tools: list[dict[str, Any]] = field(default_factory=list)
+
+    def bind_tools(self, tools: list[dict[str, Any]]) -> MiniCPMChatModel:
+        return MiniCPMChatModel(
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            top_p=self.top_p,
+            tools=list(tools),
+        )
+
+    def invoke(self, messages: list[Any]) -> AIMessage:
+        from ..minicpm.model import chat_complete
+
+        return chat_complete(
+            messages,
+            tools=self.tools or None,
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            top_p=self.top_p,
+        )
+
+
+def build_llm(config: RunnableConfig, **overrides: Any) -> MiniCPMChatModel | ChatOpenAI:
     """Build a chat model from the per-request configurable values."""
     configurable = config.get("configurable", {})
+    max_tokens = int(overrides.pop("max_tokens", configurable.get("max_tokens", 1800)))
+    temperature = float(
+        overrides.pop("temperature", configurable.get("temperature", 0.35))
+    )
+    top_p = float(overrides.pop("top_p", configurable.get("top_p", 0.9)))
+
+    if local_inference_enabled():
+        return MiniCPMChatModel(
+            max_tokens=max_tokens,
+            temperature=temperature,
+            top_p=top_p,
+        )
+
     params: dict[str, Any] = {
         "model": MODEL_ID,
-        "api_key": configurable["hf_token"],
+        "api_key": configurable.get("hf_token") or "",
         "base_url": HF_ROUTER_BASE_URL,
-        "max_tokens": configurable.get("max_tokens", 1800),
-        "temperature": configurable.get("temperature", 0.35),
-        "top_p": configurable.get("top_p", 0.9),
+        "max_tokens": max_tokens,
+        "temperature": temperature,
+        "top_p": top_p,
     }
     params.update(overrides)
     return ChatOpenAI(**params)

ui/agent/graph/nodes/consolidator.py

@@ -10,12 +10,10 @@ from ..llm import build_llm
 from ..state import AgentState
 from ...synthesis import build_structured_final_answer
 from .config import CONSOLIDATOR_MAX_TOKENS, FINDING_SUMMARY_LIMIT
+from .helpers import extract_assistant_text, format_todo_label
 from .prompts import CONSOLIDATOR_SYSTEM_PROMPT
 
 
-from .helpers import format_todo_label
-
-
 def consolidator_node(state: AgentState, config: RunnableConfig) -> dict[str, Any]:
     findings = sorted(state.get("findings", []), key=lambda item: item["todo_id"])
     todo_by_id = {todo["id"]: todo for todo in state.get("todos") or []}
@@ -44,7 +42,7 @@ def consolidator_node(state: AgentState, config: RunnableConfig) -> dict[str, An
         },
     ]
     response = llm.invoke(messages)
-    answer = str(response.content or "").strip()
+    answer = extract_assistant_text(response)
     if not answer:
         answer = build_structured_final_answer(
             profile_summary=str(state.get("profile_summary") or "").strip(),

ui/agent/graph/nodes/planner.py

@@ -11,6 +11,7 @@ from ..llm import build_llm
 from ..state import AgentState, CandidateCountry, TodoItem
 from .config import MAX_TODOS, PLANNER_MAX_TOKENS, PLANNER_TEMPERATURE
 from .helpers import (
+    extract_assistant_text,
     extract_json,
     heuristic_candidate_countries,
     normalize_plan,
@@ -69,7 +70,7 @@ def planner_node(state: AgentState, config: RunnableConfig) -> dict[str, Any]:
     raw_plan: dict[str, Any] | None = None
     for _ in range(2):
         response = llm.invoke(messages)
-        raw_plan = extract_json(str(response.content or ""))
+        raw_plan = extract_json(extract_assistant_text(response))
         if raw_plan and raw_plan.get("todos"):
             break
         raw_plan = None

ui/agent/graph/respond.py

@@ -11,6 +11,7 @@ from ui.globe_commands import apply_update_globe
 from ui.globe_details import attach_finding_to_globe, attach_research_progress_to_globe
 
 from ..messages import history_to_api_messages, multimodal_input_to_api_content
+from ..config import hub_token_available, local_inference_enabled
 from ..respond import (
     AUTH_REQUIRED_MESSAGE,
     SESSION_EXPIRED_MESSAGE,
@@ -243,11 +244,11 @@ def respond_with_graph(
     hf_token: gr.OAuthToken | None,
 ):
     """LangGraph-backed drop-in replacement for ui.agent.respond.respond."""
-    if hf_token is None:
+    if hf_token is None and not (local_inference_enabled() and hub_token_available()):
         yield from yield_response([], AUTH_REQUIRED_MESSAGE, globe_state)
         return
 
-    if hf_token.expires_at <= time.time():
+    if hf_token is not None and hf_token.expires_at <= time.time():
         yield from yield_response([], SESSION_EXPIRED_MESSAGE, globe_state)
         return
 
@@ -267,7 +268,7 @@ def respond_with_graph(
 
     config = {
         "configurable": {
-            "hf_token": hf_token.token,
+            "hf_token": hf_token.token if hf_token is not None else "",
             "max_tokens": max_tokens,
             "temperature": temperature,
             "top_p": top_p,

ui/agent/minicpm/__init__.py

@@ -0,0 +1,4 @@
+# ui/agent/minicpm/__init__.py
+from __future__ import annotations
+
+__all__: list[str] = []

ui/agent/minicpm/messages.py

@@ -0,0 +1,100 @@
+# ui/agent/minicpm/messages.py
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from langchain_core.messages import AIMessage, BaseMessage, ToolMessage
+
+
+def _flatten_content(content: Any) -> str:
+    if content is None:
+        return ""
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts: list[str] = []
+        for block in content:
+            if isinstance(block, str):
+                parts.append(block)
+                continue
+            if isinstance(block, dict):
+                if block.get("type") == "text":
+                    parts.append(str(block.get("text") or ""))
+                elif "text" in block:
+                    parts.append(str(block["text"]))
+        return "\n".join(part for part in parts if part)
+    return str(content)
+
+
+def normalize_messages(messages: list[Any]) -> list[dict[str, str]]:
+    """Convert LangChain or OpenAI-style messages to MiniCPM chat messages."""
+    normalized: list[dict[str, str]] = []
+    for message in messages:
+        if isinstance(message, dict):
+            role = str(message.get("role") or "user")
+            content = _flatten_content(message.get("content"))
+            if content:
+                normalized.append({"role": role, "content": content})
+            continue
+
+        if isinstance(message, BaseMessage):
+            if isinstance(message, ToolMessage):
+                tool_id = getattr(message, "tool_call_id", "") or "tool"
+                content = _flatten_content(message.content)
+                normalized.append(
+                    {
+                        "role": "user",
+                        "content": f"Tool result ({tool_id}):\n{content}",
+                    }
+                )
+                continue
+
+            role = getattr(message, "type", "user")
+            if role == "human":
+                role = "user"
+            elif role == "ai":
+                role = "assistant"
+            content = _flatten_content(message.content)
+            if content:
+                normalized.append({"role": role, "content": content})
+            continue
+
+        normalized.append({"role": "user", "content": str(message)})
+    return normalized
+
+
+def append_tool_instructions(
+    messages: list[dict[str, str]],
+    tools: list[dict[str, Any]],
+) -> list[dict[str, str]]:
+    if not tools:
+        return messages
+
+    lines = [
+        "You may call tools by emitting a single line in this exact format:",
+        "tool_name{\"arg\": \"value\"}",
+        "",
+        "Available tools:",
+    ]
+    for tool in tools:
+        function = tool.get("function") or {}
+        name = str(function.get("name") or "")
+        description = str(function.get("description") or "").strip()
+        parameters = function.get("parameters") or {}
+        lines.append(f"- {name}: {description}")
+        lines.append(f"  parameters schema: {json.dumps(parameters, ensure_ascii=True)}")
+
+    tool_block = "\n".join(lines)
+    if not messages:
+        return [{"role": "system", "content": tool_block}]
+
+    updated = list(messages)
+    if updated[0]["role"] == "system":
+        updated[0] = {
+            "role": "system",
+            "content": f"{updated[0]['content']}\n\n{tool_block}",
+        }
+    else:
+        updated.insert(0, {"role": "system", "content": tool_block})
+    return updated

ui/agent/minicpm/model.py

@@ -0,0 +1,141 @@
+# ui/agent/minicpm/model.py
+from __future__ import annotations
+
+import logging
+import os
+import threading
+from typing import Any
+
+import torch
+from huggingface_hub import login
+from langchain_core.messages import AIMessage
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from ..config import ENABLE_THINKING, MODEL_ID
+from .messages import append_tool_instructions, normalize_messages
+
+logger = logging.getLogger(__name__)
+
+_GENERATE_LOCK = threading.Lock()
+_MODEL: AutoModelForCausalLM | None = None
+_TOKENIZER: AutoTokenizer | None = None
+_DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+
+def _hub_login() -> None:
+    hf_token = os.environ.get("HF_TOKEN")
+    if hf_token:
+        login(token=hf_token)
+        logger.info("Logged in to Hugging Face Hub for MiniCPM weights")
+    else:
+        logger.warning("HF_TOKEN not set — gated MiniCPM weights may be inaccessible")
+
+
+def _load_model() -> tuple[AutoTokenizer, AutoModelForCausalLM]:
+    global _MODEL, _TOKENIZER
+    if _MODEL is not None and _TOKENIZER is not None:
+        return _TOKENIZER, _MODEL
+
+    _hub_login()
+    logger.info("Loading MiniCPM model %s on %s", MODEL_ID, _DEVICE)
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
+    model = AutoModelForCausalLM.from_pretrained(
+        MODEL_ID,
+        torch_dtype=torch.bfloat16,
+        trust_remote_code=True,
+    ).to(_DEVICE)
+    _TOKENIZER = tokenizer
+    _MODEL = model
+    return tokenizer, model
+
+
+def _apply_chat_template(
+    tokenizer: AutoTokenizer,
+    messages: list[dict[str, str]],
+    *,
+    enable_thinking: bool,
+) -> str:
+    kwargs: dict[str, Any] = {
+        "tokenize": False,
+        "add_generation_prompt": True,
+    }
+    try:
+        return tokenizer.apply_chat_template(
+            messages,
+            enable_thinking=enable_thinking,
+            **kwargs,
+        )
+    except TypeError:
+        return tokenizer.apply_chat_template(messages, **kwargs)
+
+
+def _split_think_output(text: str) -> tuple[str, str]:
+    open_tag = "<" + "think" + ">"
+    close_tag = "</" + "think" + ">"
+    start = text.find(open_tag)
+    end = text.find(close_tag)
+    if start != -1 and end != -1 and end > start:
+        reasoning = text[start + len(open_tag) : end].strip()
+        content = (text[:start] + text[end + len(close_tag) :]).strip()
+        return content, reasoning
+    return text.strip(), ""
+
+
+def chat_complete(
+    messages: list[Any],
+    *,
+    tools: list[dict[str, Any]] | None = None,
+    max_tokens: int = 1800,
+    temperature: float = 0.35,
+    top_p: float = 0.9,
+    enable_thinking: bool | None = None,
+) -> AIMessage:
+    """Run one MiniCPM chat turn and return a LangChain AIMessage."""
+    tokenizer, model = _load_model()
+    normalized = normalize_messages(messages)
+    if tools:
+        normalized = append_tool_instructions(normalized, tools)
+
+    thinking = ENABLE_THINKING if enable_thinking is None else enable_thinking
+    prompt_text = _apply_chat_template(tokenizer, normalized, enable_thinking=thinking)
+    model_inputs = tokenizer([prompt_text], return_tensors="pt").to(_DEVICE)
+
+    gen_kwargs: dict[str, Any] = {
+        **model_inputs,
+        "max_new_tokens": max_tokens,
+    }
+    if temperature > 0:
+        gen_kwargs.update(
+            temperature=temperature,
+            top_p=top_p,
+            do_sample=True,
+        )
+    else:
+        gen_kwargs["do_sample"] = False
+
+    with _GENERATE_LOCK:
+        output_ids = model.generate(**gen_kwargs)
+
+    generated = output_ids[0][model_inputs["input_ids"].shape[1] :]
+    raw_text = tokenizer.decode(generated, skip_special_tokens=False)
+    content, reasoning = _split_think_output(raw_text)
+
+    additional_kwargs: dict[str, Any] = {}
+    if reasoning:
+        additional_kwargs["reasoning_content"] = reasoning
+
+    return AIMessage(content=content or raw_text, additional_kwargs=additional_kwargs)
+
+
+def _maybe_preload() -> None:
+    if os.environ.get("BORDERLESS_INFERENCE_MODE", "local") != "local":
+        return
+    if os.environ.get("BORDERLESS_PRELOAD_MODEL", "1") != "1":
+        return
+    try:
+        _load_model()
+    except Exception as exc:
+        logger.warning("MiniCPM eager preload skipped: %s", exc)
+
+
+_maybe_preload()

ui/server_api.py

@@ -2,11 +2,14 @@
 from __future__ import annotations
 
 import os
+from collections.abc import Iterator
 from typing import Any
 
 import gradio as gr
+import spaces
 from gradio import ChatMessage
 
+from ui.agent.config import GPU_DURATION, local_inference_enabled
 from ui.agent.graph import respond_with_graph
 from ui.agent.respond import respond as respond_legacy
 from ui.agent.system_prompt import BORDERLESS_SYSTEM_PROMPT
@@ -127,26 +130,51 @@ def _merge_chat_history(
     return updated_history
 
 
-def stream_chat(
+def _stream_agent_response(
     message: str,
     history: list[dict[str, Any]],
-    globe_state: dict[str, Any] | None,
+    globe_state: dict[str, Any],
     hf_token: gr.OAuthToken | None,
-):
-    state = globe_state if globe_state else empty_globe_state()
-    ui_messages: list[ChatMessage] = []
-    assistant_text = ""
-
-    for chunk in _respond_fn()(
+) -> Iterator[tuple[Any, dict[str, Any]]]:
+    yield from _respond_fn()(
         message,
         history,
         BORDERLESS_SYSTEM_PROMPT,
         DEFAULT_MAX_TOKENS,
         DEFAULT_TEMPERATURE,
         DEFAULT_TOP_P,
-        state,
+        globe_state,
         hf_token,
-    ):
+    )
+
+
+@spaces.GPU(duration=GPU_DURATION)
+def _run_agent_graph_gpu(
+    message: str,
+    history: list[dict[str, Any]],
+    globe_state: dict[str, Any],
+    hf_token: gr.OAuthToken | None,
+) -> Iterator[tuple[Any, dict[str, Any]]]:
+    yield from _stream_agent_response(message, history, globe_state, hf_token)
+
+
+def stream_chat(
+    message: str,
+    history: list[dict[str, Any]],
+    globe_state: dict[str, Any] | None,
+    hf_token: gr.OAuthToken | None,
+):
+    state = globe_state if globe_state else empty_globe_state()
+    ui_messages: list[ChatMessage] = []
+    assistant_text = ""
+
+    agent_stream = (
+        _run_agent_graph_gpu(message, history, state, hf_token)
+        if local_inference_enabled() and AGENT_MODE != "legacy"
+        else _stream_agent_response(message, history, state, hf_token)
+    )
+
+    for chunk in agent_stream:
         payload, state = chunk
         if isinstance(payload, list):
             ui_messages = payload