spike: MolmoAct2 Docker Space (server + Dockerfile)

DEPLOY.md

@@ -0,0 +1,67 @@
+# Deploy MolmoAct2 as a HuggingFace Docker Space
+
+This folder IS the Space contents (`Dockerfile`, `README.md`, `requirements.txt`,
+`molmoact2_server.py`). `molmoact2_server.py` is a copy of the canonical
+`../molmoact2_server.py` — after editing the canonical server, re-copy it:
+
+```bash
+cp ../molmoact2_server.py molmoact2_server.py
+```
+
+## Target
+Space `NoriRobotics/molmoact2-space` (private, Docker SDK, GPU hardware).
+
+## Path A — scripted (create repo + upload folder)
+Uses the org token already in `nori-backend/.env` (`HF_ORG_ADMIN_TOKEN`), read
+in-process, never printed. From `cloud_inference/space/`:
+
+```bash
+python - <<'PY'
+import os
+from dotenv import load_dotenv
+from huggingface_hub import HfApi
+load_dotenv("/Users/michael/Documents/Nori-Robotics/nori-backend/.env")
+api = HfApi(token=os.environ["HF_ORG_ADMIN_TOKEN"])
+repo = "NoriRobotics/molmoact2-space"
+api.create_repo(repo, repo_type="space", space_sdk="docker", private=True, exist_ok=True)
+api.upload_folder(repo_id=repo, repo_type="space", folder_path=".")
+print("uploaded ->", repo)
+PY
+```
+
+Creating the Space defaults to **free CPU** hardware; a CPU box cannot load the
+model. After upload, finish in the UI (Path B steps 2-3).
+
+## Path B — UI
+1. huggingface.co → New Space → owner `NoriRobotics`, SDK **Docker**, **private**.
+   Clone it, copy this folder's files in, `git add . && git commit && git push`.
+2. **Settings → Variables and secrets**: add secret `NORI_INFER_TOKEN` (the bearer
+   token the rollout sends). Add `HF_TOKEN` only if the model repo is gated.
+3. **Settings → Hardware**: pick a GPU — `a10g-small` (~$1/hr) is enough.
+
+## Verify
+First boot downloads ~21GB. Poll health (no auth needed):
+
+```bash
+curl -s https://noribotics-molmoact2-space.hf.space/health
+# {"ok":false,"status":"loading",...}  ->  {"ok":true,"status":"ready",...}
+```
+
+Smoke-test `/act` (needs the token; keep it out of shell history — read from a file):
+
+```bash
+TOKEN=$(cat ~/.nori_infer_token)   # or your secret manager
+curl -s https://noribotics-molmoact2-space.hf.space/act \
+  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
+  -d '{"images":["<b64-jpeg>","<b64-jpeg>"],"state":[0,0,0,0,0,0],
+       "instruction":"pick up the red cup","num_steps":10}'
+# -> {"actions":[[...6...], ...]}   ~30 moves, robot scale
+```
+
+The exact subdomain is shown on the Space page (Embed → Direct URL). It is
+`https://<owner>-<space-name>.hf.space`, lowercased with `/` → `-`.
+
+## Cost note
+A GPU Space bills while **Running**. Set **Sleep after inactivity** in Settings
+(e.g. 15 min) for the spike so it pauses when idle; the robot rollout wakes it
+(cold start ≈ first-boot download unless persistent storage is attached).

Dockerfile

@@ -0,0 +1,41 @@
+# MolmoAct2-SO100_101 cloud-inference server as a HuggingFace **Docker** Space.
+#
+# WHY A DOCKER SPACE (not an Inference Endpoint): the managed Inference-Endpoint
+# container bakes in `huggingface_inference_toolkit`, whose bootstrap imports
+# `transformers.file_utils.is_tf_available` — REMOVED in the transformers >=4.57
+# that MolmoAct2's processor requires (it needs `transformers.video_utils`). The
+# two are mutually exclusive, so the toolkit path is a dead end. A Docker Space
+# runs OUR uvicorn directly — the toolkit never enters the picture, and we own
+# the exact transformers version.
+#
+# Base image = the SAME torch stack proven on HF Jobs (torch 2.5.1 + cu121):
+# pinning transformers on top of it does NOT upgrade torch, so torchvision /
+# torchaudio ABIs stay intact (the "-U torchvision" ABI break we hit earlier).
+FROM pytorch/pytorch:2.5.1-cuda12.1-cudnn9-runtime
+
+# HF Spaces run the container as uid 1000 with $HOME=/home/user. Point every
+# cache at a writable dir so the ~21GB model download doesn't hit a read-only FS.
+ENV HOME=/home/user \
+    PYTHONUNBUFFERED=1 \
+    HF_HOME=/home/user/.cache/huggingface \
+    HF_HUB_ENABLE_HF_TRANSFER=1
+
+# libGL / glib for PIL+torchvision image ops; ffmpeg libs for PyAV (av) decode.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        ffmpeg libgl1 libglib2.0-0 && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN useradd -m -u 1000 user
+
+WORKDIR /app
+COPY --chown=user:user requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt hf_transfer
+
+COPY --chown=user:user molmoact2_server.py .
+
+RUN mkdir -p /home/user/.cache/huggingface && chown -R user:user /home/user /app
+USER user
+
+# HF Spaces expose the container on 7860 (declared as app_port in README.md).
+EXPOSE 7860
+CMD ["uvicorn", "molmoact2_server:app", "--host", "0.0.0.0", "--port", "7860"]

README.md

@@ -1,10 +1,33 @@
 ---
-title: Molmoact2 Space
-emoji: 📚
-colorFrom: blue
-colorTo: green
+title: MolmoAct2 SO100 101 Inference
+emoji: 🤖
+colorFrom: indigo
+colorTo: blue
 sdk: docker
+app_port: 7860
 pinned: false
+short_description: Nori cloud-inference server for MolmoAct2-SO100_101 (/act)
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# MolmoAct2-SO100_101 — Nori cloud inference
+
+Private Docker Space serving `allenai/MolmoAct2-SO100_101` for Nori robot rollout.
+It runs our own FastAPI/uvicorn server (`molmoact2_server.py`) — **not** the HF
+Inference-Endpoint toolkit, which is incompatible with the transformers version
+this model needs.
+
+## Endpoints
+- `GET /health` → `{"ok", "status": "loading|ready|error", "error", "repo", "dtype"}`
+- `POST /act` (Bearer `NORI_INFER_TOKEN`) →
+  `{ images:[b64...], state:[6 floats], instruction:str, num_steps? }`
+  → `{ actions: [[...6 DOF...], ... up to 30 moves] }` (robot scale).
+
+## Required setup (Space **Settings**)
+1. **Hardware**: a GPU tier — `a10g-small` (A10G 24GB, ~$1/hr) is enough (bf16 <16GB).
+2. **Secrets**:
+   - `NORI_INFER_TOKEN` — the bearer token the rollout client sends (required).
+   - `HF_TOKEN` — only if the model repo is gated (allenai's is public; usually not needed).
+3. First boot downloads ~21GB, so `/health` reports `"loading"` for a few minutes,
+   then `"ready"`. Add **persistent storage** later to skip re-downloads on restart.
+
+Deploy/update instructions: see `cloud_inference/space/DEPLOY.md` in the Nori-Lab repo.

molmoact2_server.py

@@ -0,0 +1,133 @@
+"""
+Nori cloud-inference server for MolmoAct2-SO100_101  (spike — task #38).
+
+Runs on an AWS GPU instance (g5.xlarge / A10G 24GB is enough in bf16 <16GB).
+Serves the robot rollout over plain JSON (NO pickle on the wire — avoids the
+LeRobot PolicyServer CVE-2026-25874 class):
+
+    POST /act  { images:[b64...], state:[6 floats], instruction:str, num_steps? }
+      -> { actions: [[...DOF...], ...] }   # a 10-30 move chunk, ROBOT SCALE
+
+The model is loaded once at startup. Inference is serialized behind a lock
+(single GPU). Bearer-token auth (NORI_INFER_TOKEN) on every call.
+
+The exact model API mirrors the allenai/MolmoAct2-SO100_101 model card:
+    model.predict_action(processor=..., images=[...], task=..., state=...,
+        norm_tag="so100_so101_molmoact2", inference_action_mode="continuous",
+        num_steps=10, normalize_language=True, enable_cuda_graph=True).actions
+
+Deploy + test: see README.md in this directory.
+"""
+
+import base64
+import io
+import os
+import threading
+from typing import Optional
+
+import numpy as np
+import torch
+from fastapi import FastAPI, Header, HTTPException
+from PIL import Image
+from pydantic import BaseModel
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+REPO_ID = os.environ.get("MOLMOACT_REPO", "allenai/MolmoAct2-SO100_101")
+NORM_TAG = os.environ.get("MOLMOACT_NORM_TAG", "so100_so101_molmoact2")
+AUTH_TOKEN = os.environ.get("NORI_INFER_TOKEN")  # REQUIRED — the rollout sends it
+# bf16 fits <16GB (A10G/L4). Set MOLMOACT_BF16=0 to run fp32 (~26GB, needs L40S/48GB).
+DTYPE = torch.bfloat16 if os.environ.get("MOLMOACT_BF16", "1") == "1" else torch.float32
+
+app = FastAPI(title="nori-molmoact2")
+_model = None
+_processor = None
+_lock = threading.Lock()  # single GPU: serialize predict_action calls
+_load_error: Optional[str] = None  # set if the background load failed
+
+
+def _load_model() -> None:
+    """Load weights in a background thread so the HTTP port is up immediately.
+
+    MolmoAct2 is ~21GB — a blocking startup event would keep the port dark for
+    minutes and a HuggingFace Space health-probe would kill the container as
+    unhealthy before the model ever finishes loading. /health reports progress.
+    """
+    global _model, _processor, _load_error
+    try:
+        proc = AutoProcessor.from_pretrained(REPO_ID, trust_remote_code=True)
+        model = (
+            AutoModelForImageTextToText.from_pretrained(
+                REPO_ID, trust_remote_code=True, dtype=DTYPE
+            )
+            .to("cuda")
+            .eval()
+        )
+        _processor, _model = proc, model
+        print(f"[molmoact2] loaded {REPO_ID} dtype={DTYPE}", flush=True)
+    except Exception as exc:  # surface load failures via /health instead of a dead port
+        _load_error = f"{type(exc).__name__}: {exc}"
+        print(f"[molmoact2] LOAD FAILED — {_load_error}", flush=True)
+
+
+@app.on_event("startup")
+def _startup() -> None:
+    if not AUTH_TOKEN:
+        raise RuntimeError("NORI_INFER_TOKEN must be set (bearer token for /act)")
+    threading.Thread(target=_load_model, name="molmoact2-load", daemon=True).start()
+
+
+class ActRequest(BaseModel):
+    images: list[str]      # base64 JPEG/PNG (optionally a data: URL), 2+ camera views
+    state: list[float]     # robot joint state (6 for a single SO-100/101 arm)
+    instruction: str       # natural-language task, e.g. "pick up the red cup"
+    num_steps: int = 10    # flow-matching integration steps (latency <-> quality)
+
+
+class ActResponse(BaseModel):
+    actions: list[list[float]]  # chunk: N moves x DOF, ROBOT SCALE (already de-normalized)
+
+
+def _decode(b64: str) -> np.ndarray:
+    if b64.lstrip().startswith("data:") and "," in b64[:64]:
+        b64 = b64.split(",", 1)[1]
+    img = Image.open(io.BytesIO(base64.b64decode(b64))).convert("RGB")
+    return np.asarray(img)
+
+
+@app.get("/health")
+def health() -> dict:
+    status = "ready" if _model is not None else ("error" if _load_error else "loading")
+    return {"ok": _model is not None, "status": status, "error": _load_error,
+            "repo": REPO_ID, "dtype": str(DTYPE)}
+
+
+@app.post("/act", response_model=ActResponse)
+def act(req: ActRequest, authorization: Optional[str] = Header(None)) -> ActResponse:
+    if authorization != f"Bearer {AUTH_TOKEN}":
+        raise HTTPException(status_code=401, detail="bad or missing bearer token")
+    if _model is None:
+        detail = f"model load failed: {_load_error}" if _load_error else "model not loaded yet"
+        raise HTTPException(status_code=503, detail=detail)
+    if len(req.images) < 1:
+        raise HTTPException(status_code=422, detail="need at least one camera image")
+    images = [_decode(b) for b in req.images]
+    state = np.asarray(req.state, dtype=np.float32)
+    with _lock, torch.no_grad():
+        out = _model.predict_action(
+            processor=_processor,
+            images=images,
+            task=req.instruction,
+            state=state,
+            norm_tag=NORM_TAG,
+            inference_action_mode="continuous",
+            num_steps=req.num_steps,
+            normalize_language=True,
+            enable_cuda_graph=True,
+        )
+    acts = out.actions
+    if torch.is_tensor(acts):  # predict_action returns a CUDA tensor — move to host first
+        acts = acts.detach().float().cpu().numpy()
+    acts = np.asarray(acts, dtype=np.float32)
+    if acts.ndim == 3 and acts.shape[0] == 1:  # (1, chunk, DOF) -> (chunk, DOF)
+        acts = acts[0]
+    return ActResponse(actions=acts.tolist())

requirements.txt

@@ -0,0 +1,20 @@
+# Installed ON TOP OF the pytorch/pytorch:2.5.1-cuda12.1 base image.
+#
+# Do NOT list torch / torchvision / torchaudio here: the base image already
+# ships a matched, ABI-compatible trio. Reinstalling/upgrading any one of them
+# re-triggers the "libtorchaudio.so: undefined symbol" ABI break we hit on HF Jobs.
+#
+# transformers>=4.57.0 is MANDATORY — MolmoAct2's trust_remote_code processor
+# imports transformers.video_utils.VideoInput (added in 4.57.0). Pinning it here
+# is SAFE in a Docker Space (unlike the Inference-Endpoint toolkit, nothing else
+# in this image imports the removed transformers.file_utils.is_tf_available).
+transformers>=4.57.0
+accelerate
+fastapi
+uvicorn[standard]
+pillow
+numpy
+einops
+av
+scipy
+requests