Upload folder using huggingface_hub

README.md

@@ -17,7 +17,7 @@ tags:
 models:
   - google/gemma-4-12B-it
   - bosonai/higgs-audio-v2-generation-3B-base
-  - nvidia/nemotron-3.5-asr-streaming-0.6b
+  - nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
   - openai/whisper-large-v3-turbo
 ---
 
@@ -30,40 +30,40 @@ with its voice.**
 Built for the [HF Build Small Hackathon](https://huggingface.co/build-small-hackathon):
 every model runs **on this Space itself** via ZeroGPU. No cloud APIs, no keys.
 
-## The stack (Σ 16.4B params — well under the 32B cap)
+## The stack (Σ 19.8B params — well under the 32B cap)
 
 | Role | Model | Params |
 |---|---|---|
-| 🧠 Agent brain + vision | [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) | 12B |
+| 🧠 Teacher + vision | [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) | 12B |
+| 🔎 Research planner | [nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16) | 4B |
 | 🗣️ Expressive voice | [bosonai/higgs-audio-v2-generation-3B-base](https://huggingface.co/bosonai/higgs-audio-v2-generation-3B-base) | 3B |
-| 👂 Speech recognition | [nvidia/nemotron-3.5-asr-streaming-0.6b](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) | 0.6B |
-| 👂 ASR fallback | [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | 0.8B |
-
-### The Nemotron CPU sidecar 🟩
-
-The fun engineering bit: **NeMo and ZeroGPU don't mix.** Import NeMo in the
-main process and ZeroGPU's forked GPU workers die ("GPU task aborted" —
-verified the hard way); lazy-load it inside the workers instead and every
-fresh worker spends 30-60 s restoring the model on its first voice turn.
-
-So Nemotron runs in a **sidecar**: an isolated process spawned at startup
-(`asr_sidecar.py`) that loads NeMo once and serves transcriptions over a
-pipe — on **CPU**, where the 0.6B FastConformer transcribes an utterance in
-a couple of seconds. Your words appear *before the GPU is even requested*,
-the whole `@spaces.GPU` window is spent on teaching, and voice input costs
-**zero GPU quota**. Whisper turbo rides along inside the GPU as a silent
-fallback if the sidecar is ever unavailable.
-
-> One more honest swap forced by the same constraint: Higgs Audio **v3** TTS
-> ships only for the SGLang-Omni serving stack (needs a persistent GPU), so
-> we use v2 — same family, natively in transformers.
+| 👂 Speech recognition | [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | 0.8B |
+
+**Gemma teaches. Nemotron researches.** Every turn, NVIDIA's Nemotron 3 Nano
+decides whether the question needs fresh facts and writes the search queries —
+the agentic step that turns a chatbot into a researcher. Gemma 4 then teaches
+from what Nemotron found, Higgs speaks it, Whisper listens.
+
+### Engineering notes (the honest kind)
+
+- We first shipped **Nemotron 3.5 ASR** as the ears. ZeroGPU said no, twice:
+  NeMo in the main process crashes the forked GPU workers ("GPU task
+  aborted"), and lazy-loading it inside workers cost every fresh worker
+  30-60 s on its first voice turn. We then ran it in an isolated **CPU
+  sidecar** — measured RTF ≈ 24 on the Space's shared CPU (a 10 s clip took
+  ~240 s, int8-quantized). Not interactive. So Nemotron moved to the role
+  where it shines on this hardware — the research brain — and Whisper turbo
+  took the ears (preloads with everything, transcribes in ~1 s).
+- Higgs Audio **v3** TTS ships only for the SGLang-Omni serving stack (needs
+  a persistent GPU), so we use v2 — same family, natively in transformers.
 
 ## How a turn works
 
-1. **You talk** (or type). Nemotron ASR transcribes you on-device.
-2. **The agent gathers context.** Gemma 4 decides whether the question needs
-   fresh facts; if so it writes search queries and pulls snippets from the web
-   (DuckDuckGo, keyless). Timeless topics skip straight to teaching.
+1. **You talk** (or type). Whisper turbo transcribes you on-device.
+2. **Nemotron researches.** NVIDIA's Nemotron 3 Nano decides whether the
+   question needs fresh facts; if so it writes the search queries and Tutori
+   pulls snippets + page text from the web (DuckDuckGo, keyless). Timeless
+   topics skip straight to teaching.
 3. **It teaches in steps.** Gemma 4 emits a JSON lesson script — each step is
    a sentence to *say* plus whiteboard ops to *draw* (boxes, arrows, curves,
    axes, highlights in a 100×75 coordinate space).
@@ -86,7 +86,7 @@ fallback if the sidecar is ever unavailable.
 
 ## Running it
 
-- **On Spaces (the real thing):** select **ZeroGPU** hardware. All three
+- **On Spaces (the real thing):** select **ZeroGPU** hardware. All four
   models load at startup and get packed by ZeroGPU; each turn runs in a
   single `@spaces.GPU(duration=59)` generator call, sized so even
   **logged-out visitors** (120s/day ZeroGPU quota) get a full lesson. Sign in

app.py

@@ -9,9 +9,6 @@ this Space itself (ZeroGPU) — 16.6B parameters total, zero cloud APIs.
 
 import json
 import os
-import subprocess
-import sys
-import threading
 import uuid
 from pathlib import Path
 
@@ -46,76 +43,6 @@ BOARD_JS = (ROOT / "static" / "board.js").read_text()
 CSS = (ROOT / "static" / "style.css").read_text()
 
 
-class AsrSidecar:
-    """Client for the Nemotron ASR sidecar process (see asr_sidecar.py).
-
-    The sidecar loads NeMo once, on CPU, in its own process — so voice turns
-    transcribe in a couple of seconds, before any GPU is even requested.
-    """
-
-    def __init__(self, script):
-        self.script = script
-        self.proc = subprocess.Popen(
-            [sys.executable, str(script)],
-            stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-            stderr=None, text=True, bufsize=1,
-        )
-        self.lock = threading.Lock()
-        self.ready = False
-        threading.Thread(target=self._await_ready, daemon=True).start()
-
-    def _await_ready(self):
-        line = self.proc.stdout.readline()
-        if '"ready"' in (line or ""):
-            self.ready = True
-            print("[tutori] Nemotron ASR sidecar ready", flush=True)
-
-    def transcribe(self, path, timeout=45):
-        if not self.ready or self.proc.poll() is not None:
-            return None
-        with self.lock:
-            try:
-                self.proc.stdin.write(json.dumps({"path": path}) + "\n")
-                self.proc.stdin.flush()
-            except Exception:
-                return None
-            box = {}
-
-            def _read():
-                box["line"] = self.proc.stdout.readline()
-
-            t = threading.Thread(target=_read, daemon=True)
-            t.start()
-            t.join(timeout)
-            line = box.get("line")
-            if not line:
-                # the late reply would desync the protocol — restart instead
-                print("[tutori] sidecar timed out; restarting it", flush=True)
-                try:
-                    self.proc.kill()
-                except Exception:
-                    pass
-                self.ready = False
-                self.__init__(self.script)
-                return None
-        if not line:
-            return None
-        try:
-            out = json.loads(line)
-        except Exception:
-            return None
-        if out.get("error"):
-            print(f"[tutori] sidecar ASR error: {out['error']}", flush=True)
-            return None
-        return out.get("text") or None
-
-
-SIDECAR = None
-if ENGINE.MODELS_INFO.get("mode") == "zerogpu" and \
-        os.environ.get("TUTORI_SIDECAR", "1") == "1":
-    print("[tutori] starting Nemotron ASR sidecar …", flush=True)
-    SIDECAR = AsrSidecar(ROOT / "asr_sidecar.py")
-
 HEAD = f"""
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link href="https://fonts.googleapis.com/css2?family=Caveat:wght@500;600;700&family=Inter:wght@400;500;600&family=Space+Grotesk:wght@600;700&display=swap" rel="stylesheet">
@@ -132,8 +59,9 @@ HEADER_HTML = f"""
     <span class="badge hot">⚡ ZeroGPU · no cloud APIs</span>
     <span class="badge">🧠 Gemma 4 12B</span>
     <span class="badge">🗣️ Higgs Audio v2 TTS 3B</span>
-    <span class="badge">👂 Nemotron 3.5 ASR 0.6B</span>
-    <span class="badge">Σ 16.4B params — Build Small ✅</span>
+    <span class="badge">🔎 Nemotron 3 Nano 4B</span>
+    <span class="badge">👂 Whisper v3 turbo 0.8B</span>
+    <span class="badge">Σ 19.8B params — Build Small ✅</span>
   </div>
 </div>
 """
@@ -201,20 +129,6 @@ def run_turn(audio_path, typed_text, snapshot, chat, convo, profile,
 
     yield render(detail="Thinking…")
 
-    # Nemotron sidecar hears you BEFORE any GPU is requested — the transcript
-    # appears in a couple of seconds and the GPU window is spent on teaching.
-    if audio_path and SIDECAR is not None:
-        yield render("thinking", "Listening…")
-        heard = SIDECAR.transcribe(audio_path)
-        if heard:
-            typed_text = f"{heard} {typed_text or ''}".strip()
-            question_for_context = typed_text
-            chat[-2]["content"] = f"🎙️ {typed_text}"
-            audio_path = None
-            yield render("thinking", "Heard you!")
-        # if the sidecar couldn't transcribe, the audio passes through to the
-        # engine's GPU fallback ASR
-
     seq = 0
     error = None
     print(f"[tutori] turn {turn_id}: starting engine", flush=True)
@@ -356,8 +270,8 @@ with gr.Blocks(
     gr.HTML(
         '<div id="tutori-foot">Built small for the '
         '<a href="https://huggingface.co/build-small-hackathon" target="_blank">'
-        "HF Build Small Hackathon</a> · Gemma 4 12B + Higgs Audio v2 + Nemotron ASR "
-        "= 16.4B params, all on this Space · draw on the board with your mouse, "
+        "HF Build Small Hackathon</a> · Gemma 4 teaches · Nemotron 3 researches · Higgs speaks · Whisper listens "
+        "= 19.8B params, all on this Space · draw on the board with your mouse, "
         "Tutori can see it 👀"
         '<div id="made-by">Made by SSH/ProCreations</div></div>'
     )

engine.py

@@ -1,12 +1,13 @@
 """
 Tutori engine — the real, ZeroGPU-backed brains.
 
-Models (15.8B params total, all running on the Space itself — no API calls):
+Models (19.8B params total, all running on the Space itself — no API calls):
   * google/gemma-4-12B-it ............ agent / lesson planner / vision (12B)
   * bosonai/higgs-audio-v2-generation-3B-base ... expressive speech (3B)
     (v3 TTS only ships for the SGLang-Omni serving stack, which needs a
      persistent GPU — incompatible with ZeroGPU. v2 is the same Higgs Audio
      family, natively supported by transformers.)
+  * nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 ... research planner (4B)
   * openai/whisper-large-v3-turbo .... speech recognition (0.8B)
 
 run_turn() is a generator that yields event dicts; app.py turns those into
@@ -32,7 +33,8 @@ import soundfile as sf
 import spaces
 import torch
 from PIL import Image
-from transformers import (AutoModelForMultimodalLM, AutoProcessor,
+from transformers import (AutoModelForCausalLM, AutoModelForMultimodalLM,
+                          AutoProcessor, AutoTokenizer,
                           HiggsAudioV2ForConditionalGeneration,
                           TextIteratorStreamer, pipeline)
 
@@ -43,6 +45,7 @@ if os.environ.get("TUTORI_DEBUG") == "1":
 LLM_ID = "google/gemma-4-12B-it"
 TTS_ID = "bosonai/higgs-audio-v2-generation-3B-base"
 ASR_ID = "openai/whisper-large-v3-turbo"
+RESEARCH_ID = "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
 
 print(f"[tutori] torch {torch.__version__} | spaces {getattr(spaces, '__version__', '?')}",
       flush=True)
@@ -71,6 +74,12 @@ asr_pipe = pipeline(
     dtype=torch.bfloat16, device="cuda",
 )
 
+print("[tutori] loading Nemotron 3 Nano research planner …")
+research_tok = AutoTokenizer.from_pretrained(RESEARCH_ID)
+research_llm = AutoModelForCausalLM.from_pretrained(
+    RESEARCH_ID, dtype=torch.bfloat16, device_map="cuda"
+)
+
 print("[tutori] all models ready")
 
 
@@ -959,6 +968,19 @@ def synthesize(text):
 # web research (DuckDuckGo, no key needed)
 # --------------------------------------------------------------------------
 
+def nemotron_generate(messages, max_new_tokens=220):
+    """Tutori's research planner runs on NVIDIA Nemotron 3 Nano."""
+    ids = research_tok.apply_chat_template(
+        messages, add_generation_prompt=True, return_tensors="pt"
+    ).to(research_llm.device)
+    with torch.inference_mode():
+        out = research_llm.generate(
+            ids, max_new_tokens=max_new_tokens, do_sample=False,
+            pad_token_id=research_tok.eos_token_id,
+        )
+    return research_tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)
+
+
 def decide_search(question, profile, notes):
     today = datetime.date.today().isoformat()
     notes_hint = f"\nEXISTING NOTES (from earlier this session): {notes[:500]}" if notes else ""
@@ -969,12 +991,17 @@ def decide_search(question, profile, notes):
                     f"Learner profile: {json.dumps(profile)[:400]}{notes_hint}"},
     ]
     try:
-        raw = llm_generate(msgs, max_new_tokens=96, temperature=0.0)
-        obj = parse_lesson_json(raw) or {}
-        if obj.get("search") and isinstance(obj.get("queries"), list):
-            return [str(q)[:120] for q in obj["queries"][:2]]
+        raw = nemotron_generate(msgs, max_new_tokens=220)
     except Exception as e:
-        print(f"[tutori] search decision failed: {e}")
+        print(f"[tutori] Nemotron planner failed ({e!r}); falling back to Gemma")
+        try:
+            raw = llm_generate(msgs, max_new_tokens=96, temperature=0.0)
+        except Exception as e2:
+            print(f"[tutori] search decision failed: {e2}")
+            return []
+    obj = parse_lesson_json(raw) or {}
+    if obj.get("search") and isinstance(obj.get("queries"), list):
+        return [str(q)[:120] for q in obj["queries"][:2]]
     return []
 
 
@@ -1195,6 +1222,6 @@ def tts_only(text):
 
 MODELS_INFO = {
     "llm": LLM_ID, "tts": TTS_ID, "asr": ASR_ID,
-    "total_params": "15.8B (12B + 3B + 0.8B)",
+    "total_params": "19.8B (12B + 4B + 3B + 0.8B)",
     "mode": "zerogpu",
 }