Upload folder using huggingface_hub

.env.example

@@ -21,3 +21,6 @@ SSH_KNOWN_HOSTS=
 # Optional: Provider CLI auth (prefer env vars, not files)
 GEMINI_API_KEY=
 ANTHROPIC_API_KEY=
+
+# Optional: GitHub indexing (private repos / higher rate limits)
+GITHUB_TOKEN=

PLANS.md

@@ -3,6 +3,7 @@
 This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally opinionated and ordered by risk reduction first, then maintainability, then feature expansion.
 
 ## P0 — Security + correctness (blockers)
+
 - Gate **all dangerous endpoints** server-side (not just UI):
   - `/ws/terminal`
   - `/api/codex*`
@@ -14,6 +15,7 @@ This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally op
 - Add `SECURITY.md` with threat model + safe deployment guidance.
 
 ## P1 — Backend refactor + lifecycle
+
 - Split `main.py` into routers/services:
   - `app/auth.py`, `app/chat.py`, `app/terminal.py`, `app/codex.py`, `app/mcp.py`, `app/settings.py`, `app/admin.py`, `app/indexing.py`
 - Add FastAPI lifespan management:
@@ -23,6 +25,7 @@ This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally op
 - Standardize API error schema (UI should not parse strings to detect failure modes).
 
 ## P2 — UI/UX, settings, admin, landing
+
 - Split `static/dashboard.html` into modules:
   - `static/dashboard.js`, `static/terminal.js`, `static/agent.js`, `static/settings.js`, `static/admin.js`, `static/mcp.js`, `static/rag.js`
   - `static/theme.css`
@@ -38,6 +41,7 @@ This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally op
   - keep `/login` and `/app` as dedicated routes (or similar)
 
 ## P2 — Provider auth parity (Codex/Gemini/Claude)
+
 - Keep provider auth out of git; source from env/HF Secrets.
 - Support “Codex-like” auth file generation when a CLI requires it:
   - Codex: `~/.codex/.auth.json` and `~/.codex/auth.json` from `CODEX_*` (or fallback envs).
@@ -46,21 +50,25 @@ This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally op
   - `SSH_PRIVATE_KEY` (+ optional `SSH_PUBLIC_KEY`, `SSH_KNOWN_HOSTS`)
 
 ## P2 — Codex workspace directory (UI)
+
 - Add a per-user “workspace directory” setting.
 - Enforce an allowlisted root (e.g. `/data/codex/workspace/<user>`), prevent traversal, ensure it exists.
 
 ## P2 — Stream Codex events in Agent mode
+
 - Use `/api/codex/cli/stream` for agent execution.
 - UI: render streaming events progressively (agent text, tool events, final summary + usage).
 - Add stop/reconnect handling.
 
 ## P2/P3 — MCP registry
+
 - Add a first-class MCP registry:
   - per-user servers + optional global templates
   - “test connection”, “list tools”, allow/deny tool lists
   - import/export `mcp.json`
 
 ## P3 — RAG + indexing (docs/web/GitHub) + “password manager”
+
 - Clarify “password manager” scope:
   - secure vault for secrets (high-risk; encryption + audit required), or
   - indexed notes (lower-risk but still private)
@@ -73,6 +81,7 @@ This file is the repo-level roadmap for `autonomy-labs`. It’s intentionally op
 Note: see `docs/PASSWORD_MANAGER_SCOPE.md` for the current (non-vault) stance and recommended path forward.
 
 ## P3 — P2P pubsub chat + account manager
+
 - Implement account manager concepts:
   - identities/devices, room/topic membership, permissions, moderation tools
 - Transport:
@@ -82,7 +91,12 @@ Note: see `docs/PASSWORD_MANAGER_SCOPE.md` for the current (non-vault) stance an
   - rooms, presence, delivery status, network mode indicators
 
 ## Engineering hygiene (ongoing)
+
 - Add `.env.example`, `docs/TROUBLESHOOTING.md`, `docs/ARCHITECTURE.md`, `docs/SECURITY_DEPLOYMENT.md`
 - Add lint/tests + CI:
   - Python: `ruff`, `pytest`
   - basic security smoke tests for endpoint gating
+
+## Feature suggestions (By User)
+
+- Support GitHub token auth via HF Secrets (`GITHUB_TOKEN`/`GITHUB_PAT`) and document it in `.env.example`.

TASKS.md

@@ -43,7 +43,7 @@ Legend:
 ## P2/P3 — MCP registry
 - [~] First-class MCP registry storage (per-user persistence via backend).
 - [~] Admin-managed MCP templates (server-side persisted).
-- [~] “Test connection” (browser/CORS; SSRF-safe server proxy pending).
+- [x] “Test connection” (server-side, SSRF-safe).
 - [x] “List tools” (via `/api/mcp/tools`).
 - [x] Tool allow/deny policy (UI + server-side enforcement).
 - [x] Import/export `mcp.json` via UI with validation.
@@ -52,7 +52,7 @@ Legend:
 - [x] Clarify “password manager” scope and threat model (`docs/PASSWORD_MANAGER_SCOPE.md`).
 - [x] Document upload indexing connector (MVP: text-only, keyword search).
 - [~] Website crawler indexing (MVP: same-origin crawl with depth/pages, basic robots, private-host blocking).
-- [ ] GitHub repo indexing connector (branch/path filters + token support).
+- [~] GitHub repo indexing connector (MVP: owner/repo + ref + path prefix; token via env).
 - [~] Jobs UI (MVP: start/cancel/list crawl jobs).
 
 ## P3 — P2P pubsub chat + account manager

app/indexing_jobs.py

@@ -2,8 +2,8 @@ from __future__ import annotations
 
 import asyncio
 import json
+import os
 import re
-import socket
 import uuid
 from dataclasses import asdict, dataclass, field
 from datetime import UTC, datetime
@@ -15,6 +15,7 @@ from urllib.parse import urljoin, urlparse
 import httpx
 from fastapi import HTTPException
 
+from app.net_safety import is_public_host, validate_public_http_url
 from app.storage import user_data_dir
 
 
@@ -26,54 +27,8 @@ def _now_iso() -> str:
     return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
 
 
-def _is_public_host(hostname: str) -> bool:
-    host = (hostname or "").strip().lower()
-    if not host:
-        return False
-    if host in {"localhost", "localhost.localdomain"}:
-        return False
-    if host.endswith(".local") or host.endswith(".internal"):
-        return False
-
-    try:
-        infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
-    except Exception:
-        return False
-
-    import ipaddress
-
-    for info in infos:
-        addr = info[4][0]
-        try:
-            ip = ipaddress.ip_address(addr)
-        except Exception:
-            return False
-        if (
-            ip.is_private
-            or ip.is_loopback
-            or ip.is_link_local
-            or ip.is_multicast
-            or ip.is_reserved
-            or ip.is_unspecified
-        ):
-            return False
-    return True
-
-
 def _normalize_url(url: str) -> str:
-    u = (url or "").strip()
-    if not u:
-        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Missing URL"})
-    p = urlparse(u)
-    if p.scheme not in {"https", "http"}:
-        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "URL must be http(s)"})
-    if not p.netloc:
-        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "URL must include a host"})
-    if not _is_public_host(p.hostname or ""):
-        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Host is not allowed"})
-    # Normalize: strip fragment, keep query.
-    normalized = p._replace(fragment="").geturl()
-    return normalized
+    return validate_public_http_url(url)
 
 
 def _extract_links(html: str) -> list[str]:
@@ -174,6 +129,68 @@ def add_rag_document(user_id: str, *, name: str, text: str, source: str | None =
     return {"id": doc_id, "chunks": len(chunks)}
 
 
+def _parse_github_repo(repo: str) -> tuple[str, str]:
+    raw = (repo or "").strip()
+    if not raw:
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Missing repo"})
+
+    if raw.startswith(("https://", "http://")):
+        p = urlparse(raw)
+        host = (p.hostname or "").lower()
+        if host != "github.com":
+            raise HTTPException(
+                status_code=400,
+                detail={"code": "invalid_request", "message": "Repo host must be github.com"},
+            )
+        parts = [x for x in (p.path or "").split("/") if x]
+        if len(parts) < 2:
+            raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid repo URL"})
+        owner, name = parts[0], parts[1]
+    else:
+        if "/" not in raw:
+            raise HTTPException(
+                status_code=400,
+                detail={"code": "invalid_request", "message": "Repo must be owner/name"},
+            )
+        owner, name = raw.split("/", 1)
+
+    owner = owner.strip()
+    name = name.strip()
+    if name.endswith(".git"):
+        name = name[:-4]
+
+    if not re.fullmatch(r"[A-Za-z0-9_.-]{1,100}", owner or ""):
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid owner"})
+    if not re.fullmatch(r"[A-Za-z0-9_.-]{1,100}", name or ""):
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid repo name"})
+    return owner, name
+
+
+def _github_headers() -> dict[str, str]:
+    token = (os.environ.get("GITHUB_TOKEN") or os.environ.get("GITHUB_PAT") or "").strip()
+    headers = {
+        "Accept": "application/vnd.github+json",
+        "User-Agent": "autonomy-labs/1.0",
+    }
+    if token:
+        headers["Authorization"] = f"Bearer {token}"
+    return headers
+
+
+def _looks_binary(data: bytes) -> bool:
+    if not data:
+        return False
+    if b"\x00" in data[:4096]:
+        return True
+    return False
+
+
+_GITHUB_TEXT_FILE_RE = re.compile(
+    r"\.(md|markdown|txt|rst|py|js|ts|jsx|tsx|json|yaml|yml|toml|go|rs|java|kt|c|cc|cpp|h|hpp|sh)$",
+    re.IGNORECASE,
+)
+
+
 @dataclass
 class IndexJob:
     id: str
@@ -282,6 +299,43 @@ class IndexJobStore:
         self._task_map(user_id)[job.id] = task
         return job
 
+    async def create_github_repo_job(
+        self,
+        user_id: str,
+        *,
+        repo: str,
+        ref: str | None = None,
+        path_prefix: str | None = None,
+        max_files: int = 60,
+        max_file_bytes: int = 200_000,
+        max_total_bytes: int = 2_000_000,
+    ) -> IndexJob:
+        owner, name = _parse_github_repo(repo)
+        ref_s = (ref or "").strip() or None
+        prefix = (path_prefix or "").strip().lstrip("/")
+        max_files = max(1, min(int(max_files), 400))
+        max_file_bytes = max(1_000, min(int(max_file_bytes), 1_000_000))
+        max_total_bytes = max(10_000, min(int(max_total_bytes), 15_000_000))
+
+        job = IndexJob(
+            id=str(uuid.uuid4()),
+            type="github_repo",
+            createdAt=_now_iso(),
+            params={
+                "owner": owner,
+                "repo": name,
+                "ref": ref_s,
+                "pathPrefix": prefix,
+                "maxFiles": max_files,
+                "maxFileBytes": max_file_bytes,
+                "maxTotalBytes": max_total_bytes,
+            },
+        )
+        await self._update_job(user_id, job)
+        task = asyncio.create_task(self._run_github_repo(user_id, job))
+        self._task_map(user_id)[job.id] = task
+        return job
+
     async def _run_web_crawl(self, user_id: str, job: IndexJob) -> None:
         job.status = "running"
         job.progress = {"visited": 0, "indexedPages": 0, "queued": 0}
@@ -300,7 +354,11 @@ class IndexJobStore:
         robots_disallow_all = False
         if respect_robots:
             try:
-                async with httpx.AsyncClient(timeout=10.0, follow_redirects=False, headers={"User-Agent": "autonomy-labs/1.0"}) as c:
+                async with httpx.AsyncClient(
+                    timeout=10.0,
+                    follow_redirects=False,
+                    headers={"User-Agent": "autonomy-labs/1.0"},
+                ) as c:
                     r = await c.get(f"{base}/robots.txt")
                 if r.status_code == 200:
                     txt = (r.text or "")
@@ -331,7 +389,11 @@ class IndexJobStore:
         visited: set[str] = set()
         pages: list[tuple[str, str]] = []
 
-        async with httpx.AsyncClient(timeout=15.0, follow_redirects=False, headers={"User-Agent": "autonomy-labs/1.0"}) as client:
+        async with httpx.AsyncClient(
+            timeout=15.0,
+            follow_redirects=False,
+            headers={"User-Agent": "autonomy-labs/1.0"},
+        ) as client:
             try:
                 while queue and len(visited) < max_pages:
                     url, depth = queue.pop(0)
@@ -346,7 +408,7 @@ class IndexJobStore:
                         continue
                     if parsed.netloc != allowed_netloc:
                         continue
-                    if not _is_public_host(parsed.hostname or ""):
+                    if not is_public_host(parsed.hostname or ""):
                         continue
 
                     resp = await client.get(url)
@@ -416,3 +478,170 @@ class IndexJobStore:
         job.result = {"pages": len(pages), "ragDoc": result}
         job.progress = {"visited": len(visited), "indexedPages": len(pages), "queued": 0}
         await self._update_job(user_id, job)
+
+    async def _run_github_repo(self, user_id: str, job: IndexJob) -> None:
+        job.status = "running"
+        job.progress = {"files": 0, "indexedFiles": 0, "bytes": 0}
+        await self._update_job(user_id, job)
+
+        owner = str(job.params.get("owner") or "")
+        repo = str(job.params.get("repo") or "")
+        ref = (job.params.get("ref") or "").strip() or None
+        prefix = (job.params.get("pathPrefix") or "").strip().lstrip("/")
+        max_files = int(job.params.get("maxFiles") or 60)
+        max_file_bytes = int(job.params.get("maxFileBytes") or 200_000)
+        max_total_bytes = int(job.params.get("maxTotalBytes") or 2_000_000)
+
+        api_base = "https://api.github.com"
+        headers = _github_headers()
+
+        def api_url(path: str) -> str:
+            return f"{api_base}{path}"
+
+        files_text: list[tuple[str, str]] = []
+        total_bytes = 0
+
+        async with httpx.AsyncClient(timeout=20.0, follow_redirects=False) as client:
+            try:
+                # Resolve default branch if ref is not provided.
+                if not ref:
+                    r = await client.get(api_url(f"/repos/{owner}/{repo}"), headers=headers)
+                    if r.status_code == 404:
+                        raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Repo not found"})
+                    if r.status_code == 401 or r.status_code == 403:
+                        raise HTTPException(
+                            status_code=401,
+                            detail={"code": "unauthorized", "message": "GitHub auth required"},
+                        )
+                    r.raise_for_status()
+                    ref = str(r.json().get("default_branch") or "main")
+
+                # Resolve commit -> tree sha.
+                c = await client.get(api_url(f"/repos/{owner}/{repo}/commits/{ref}"), headers=headers)
+                if c.status_code == 404:
+                    raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Ref not found"})
+                if c.status_code == 401 or c.status_code == 403:
+                    raise HTTPException(
+                        status_code=401,
+                        detail={"code": "unauthorized", "message": "GitHub auth required"},
+                    )
+                c.raise_for_status()
+                commit = c.json()
+                tree_sha = ((commit.get("commit") or {}).get("tree") or {}).get("sha")
+                if not tree_sha:
+                    raise HTTPException(
+                        status_code=500,
+                        detail={"code": "github_error", "message": "Failed to resolve tree"},
+                    )
+
+                t = await client.get(
+                    api_url(f"/repos/{owner}/{repo}/git/trees/{tree_sha}?recursive=1"),
+                    headers=headers,
+                )
+                if t.status_code == 401 or t.status_code == 403:
+                    raise HTTPException(
+                        status_code=401,
+                        detail={"code": "unauthorized", "message": "GitHub auth required"},
+                    )
+                t.raise_for_status()
+                tree = t.json().get("tree")
+                if not isinstance(tree, list):
+                    raise HTTPException(
+                        status_code=500,
+                        detail={"code": "github_error", "message": "Invalid tree response"},
+                    )
+
+                candidates = []
+                for node in tree:
+                    if not isinstance(node, dict):
+                        continue
+                    if node.get("type") != "blob":
+                        continue
+                    path = str(node.get("path") or "")
+                    if not path:
+                        continue
+                    if prefix and not path.startswith(prefix.rstrip("/") + "/") and path != prefix.rstrip("/"):
+                        continue
+                    size = int(node.get("size") or 0)
+                    if size <= 0 or size > max_file_bytes:
+                        continue
+                    # Simple extension filter.
+                    lower = path.lower()
+                    if not _GITHUB_TEXT_FILE_RE.search(lower):
+                        continue
+                    candidates.append((path, size))
+
+                candidates.sort(key=lambda x: x[1])
+                candidates = candidates[:max_files]
+
+                job.progress = {"files": len(candidates), "indexedFiles": 0, "bytes": 0}
+                await self._update_job(user_id, job)
+
+                for i, (path, size) in enumerate(candidates, start=1):
+                    if total_bytes + size > max_total_bytes:
+                        break
+                    # Contents endpoint; request raw bytes.
+                    content_url = api_url(f"/repos/{owner}/{repo}/contents/{path}")
+                    r = await client.get(
+                        content_url,
+                        headers={**headers, "Accept": "application/vnd.github.raw"},
+                        params={"ref": ref},
+                    )
+                    if r.status_code == 404:
+                        continue
+                    if r.status_code == 401 or r.status_code == 403:
+                        raise HTTPException(
+                            status_code=401,
+                            detail={"code": "unauthorized", "message": "GitHub auth required"},
+                        )
+                    r.raise_for_status()
+                    data = r.content[: max_file_bytes + 1]
+                    if len(data) > max_file_bytes:
+                        continue
+                    if _looks_binary(data):
+                        continue
+                    text = data.decode("utf-8", errors="ignore").strip()
+                    if not text:
+                        continue
+                    files_text.append((path, text))
+                    total_bytes += len(data)
+                    job.progress = {"files": len(candidates), "indexedFiles": len(files_text), "bytes": total_bytes}
+                    await self._update_job(user_id, job)
+                    await asyncio.sleep(0.05)
+
+            except asyncio.CancelledError:
+                job.status = "canceled"
+                job.error = None
+                await self._update_job(user_id, job)
+                return
+            except HTTPException as e:
+                job.status = "failed"
+                job.error = str((e.detail or {}).get("message") or e.detail or "GitHub indexing failed")
+                await self._update_job(user_id, job)
+                return
+            except Exception as e:
+                job.status = "failed"
+                job.error = str(e)
+                await self._update_job(user_id, job)
+                return
+
+        if not files_text:
+            job.status = "failed"
+            job.error = "No indexable files found"
+            await self._update_job(user_id, job)
+            return
+
+        combined = []
+        for path, text in files_text:
+            combined.append(f"FILE: {path}\n\n{text}\n\n---\n")
+        combined_text = "\n".join(combined).strip()
+        result = add_rag_document(
+            user_id,
+            name=f"GitHub: {owner}/{repo}@{ref}",
+            text=combined_text,
+            source=f"https://github.com/{owner}/{repo}",
+        )
+        job.status = "succeeded"
+        job.result = {"files": len(files_text), "ragDoc": result}
+        job.progress = {"files": job.progress.get("files", 0), "indexedFiles": len(files_text), "bytes": total_bytes}
+        await self._update_job(user_id, job)

app/net_safety.py

@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+import socket
+from urllib.parse import urlparse
+
+from fastapi import HTTPException
+
+
+def is_public_host(hostname: str) -> bool:
+    host = (hostname or "").strip().lower()
+    if not host:
+        return False
+    if host in {"localhost", "localhost.localdomain"}:
+        return False
+    if host.endswith(".local") or host.endswith(".internal"):
+        return False
+
+    try:
+        infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
+    except Exception:
+        return False
+
+    import ipaddress
+
+    for info in infos:
+        addr = info[4][0]
+        try:
+            ip = ipaddress.ip_address(addr)
+        except Exception:
+            return False
+        if (
+            ip.is_private
+            or ip.is_loopback
+            or ip.is_link_local
+            or ip.is_multicast
+            or ip.is_reserved
+            or ip.is_unspecified
+        ):
+            return False
+    return True
+
+
+def validate_public_http_url(
+    url: str,
+    *,
+    allowed_hosts: set[str] | None = None,
+) -> str:
+    u = (url or "").strip()
+    if not u:
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Missing URL"})
+    p = urlparse(u)
+    if p.scheme not in {"https", "http"}:
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "URL must be http(s)"})
+    if not p.netloc:
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "URL must include a host"})
+    host = (p.hostname or "").strip().lower()
+    if allowed_hosts is not None and host not in allowed_hosts:
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Host is not allowed"})
+    if not is_public_host(host):
+        raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Host is not allowed"})
+    return p._replace(fragment="").geturl()
+

app/routes/indexing.py

@@ -17,6 +17,15 @@ class WebCrawlRequest(BaseModel):
     respectRobots: bool = True
 
 
+class GitHubRepoRequest(BaseModel):
+    repo: str  # owner/name or https://github.com/owner/name
+    ref: str | None = None  # branch, tag, or sha
+    pathPrefix: str | None = None  # optional subdirectory
+    maxFiles: int = 60
+    maxFileBytes: int = 200_000
+    maxTotalBytes: int = 2_000_000
+
+
 @router.get("/api/indexing/jobs")
 async def list_indexing_jobs(http_request: Request):
     if not feature_enabled("indexing"):
@@ -48,6 +57,25 @@ async def start_web_crawl(body: WebCrawlRequest, http_request: Request):
     return {"ok": True, "job": job.__dict__}
 
 
+@router.post("/api/indexing/jobs/github-repo")
+async def start_github_repo(body: GitHubRepoRequest, http_request: Request):
+    if not feature_enabled("indexing"):
+        raise HTTPException(status_code=403, detail={"code": "feature_disabled", "message": "Indexing is disabled"})
+    user = await require_user_from_request(http_request)
+    user_id = str(user.get("id") or "")
+    store = http_request.app.state.index_job_store
+    job = await store.create_github_repo_job(
+        user_id,
+        repo=body.repo,
+        ref=body.ref,
+        path_prefix=body.pathPrefix,
+        max_files=body.maxFiles,
+        max_file_bytes=body.maxFileBytes,
+        max_total_bytes=body.maxTotalBytes,
+    )
+    return {"ok": True, "job": job.__dict__}
+
+
 @router.post("/api/indexing/jobs/{job_id}/cancel")
 async def cancel_job(job_id: str, http_request: Request):
     if not feature_enabled("indexing"):
@@ -57,4 +85,3 @@ async def cancel_job(job_id: str, http_request: Request):
     store = http_request.app.state.index_job_store
     ok = await store.cancel_job(user_id, job_id)
     return {"ok": True, "canceled": ok}
-

app/routes/mcp.py

@@ -1,10 +1,14 @@
 from __future__ import annotations
 
+from time import monotonic
+
+import httpx
 from fastapi import APIRouter, HTTPException, Request
 from pydantic import BaseModel
 
 from app.auth import require_user_from_request
 from app.mcp_policy import load_mcp_policy, tool_allowed
+from app.net_safety import validate_public_http_url
 from app.settings import feature_enabled
 
 router = APIRouter()
@@ -43,6 +47,12 @@ class McpCallRequest(BaseModel):
     arguments: dict
 
 
+class McpTestRequest(BaseModel):
+    url: str
+    headers: dict[str, str] | None = None
+    timeoutSec: float = 3.0
+
+
 @router.post("/api/mcp/call")
 async def mcp_tools_call(request: McpCallRequest, http_request: Request):
     if not feature_enabled("mcp"):
@@ -61,3 +71,57 @@ async def mcp_tools_call(request: McpCallRequest, http_request: Request):
         raise
     except Exception as e:
         raise HTTPException(status_code=500, detail={"code": "mcp_error", "message": str(e)}) from e
+
+
+@router.post("/api/mcp/test")
+async def mcp_test(request: McpTestRequest, http_request: Request):
+    if not feature_enabled("mcp"):
+        raise HTTPException(status_code=403, detail={"code": "feature_disabled", "message": "MCP is disabled"})
+    _ = await require_user_from_request(http_request)
+
+    url = validate_public_http_url(request.url)
+    timeout = float(request.timeoutSec or 3.0)
+    timeout = max(0.5, min(timeout, 8.0))
+
+    raw_headers = request.headers if isinstance(request.headers, dict) else {}
+    headers: dict[str, str] = {}
+    for k, v in raw_headers.items():
+        key = str(k).strip()
+        if not key:
+            continue
+        lk = key.lower()
+        if lk in {"host", "content-length", "connection"}:
+            continue
+        if len(key) > 80:
+            continue
+        val = str(v).strip()
+        if len(val) > 2000:
+            val = val[:2000]
+        headers[key] = val
+
+    start = monotonic()
+    try:
+        async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
+            async with client.stream("GET", url, headers=headers) as resp:
+                limit = 1024
+                buf = bytearray()
+                async for chunk in resp.aiter_bytes():
+                    if not chunk:
+                        continue
+                    buf.extend(chunk)
+                    if len(buf) >= limit:
+                        break
+        elapsed_ms = int((monotonic() - start) * 1000)
+        preview = bytes(buf[:1024]).decode("utf-8", errors="ignore").strip()
+        return {
+            "ok": True,
+            "url": url,
+            "statusCode": resp.status_code,
+            "contentType": resp.headers.get("content-type"),
+            "elapsedMs": elapsed_ms,
+            "preview": preview[:500],
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        raise HTTPException(status_code=500, detail={"code": "mcp_test_failed", "message": str(e)}) from e

docs/TROUBLESHOOTING.md

@@ -34,6 +34,10 @@ Set `ENABLE_INDEXING=1` in your environment and restart the container.
 Website indexing blocks private/localhost targets to reduce SSRF risk.
 Use a public `http(s)` URL and keep indexing within the same origin.
 
+## GitHub indexing fails for private repos
+
+Set `GITHUB_TOKEN` (or `GITHUB_PAT`) as an environment variable / HF Secret, then retry.
+
 ## Terminal shows vertical/1-column text
 
 This usually means the terminal “fit” ran while the terminal view was hidden or at size 0.

static/dashboard.html

@@ -521,6 +521,34 @@
                                     <div id="indexing-status" class="text-xs text-gray-500"></div>
                                     <div id="indexing-jobs" class="space-y-2"></div>
                                 </div>
+                                <div class="bg-gray-900/30 border border-gray-700 rounded-lg p-3 space-y-2">
+                                    <div class="text-xs font-semibold text-gray-300 uppercase">GitHub Repo Indexing</div>
+                                    <div class="text-xs text-gray-400">Indexes repository text files into RAG (uses `GITHUB_TOKEN`/`GITHUB_PAT` for private repos).</div>
+                                    <div class="grid grid-cols-1 md:grid-cols-3 gap-2">
+                                        <input id="gh-repo" type="text" placeholder="owner/repo or https://github.com/owner/repo"
+                                            class="md:col-span-2 bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500">
+                                        <button onclick="startGithubRepoJob()"
+                                            class="bg-blue-600 hover:bg-blue-700 text-white px-3 py-2 rounded text-xs">Start</button>
+                                    </div>
+                                    <div class="grid grid-cols-1 md:grid-cols-3 gap-2">
+                                        <div>
+                                            <label class="block text-xs font-semibold text-gray-400 mb-1 uppercase">Ref</label>
+                                            <input id="gh-ref" type="text" placeholder="main"
+                                                class="w-full bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500">
+                                        </div>
+                                        <div>
+                                            <label class="block text-xs font-semibold text-gray-400 mb-1 uppercase">Path Prefix</label>
+                                            <input id="gh-path" type="text" placeholder="(optional) src/"
+                                                class="w-full bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500">
+                                        </div>
+                                        <div>
+                                            <label class="block text-xs font-semibold text-gray-400 mb-1 uppercase">Max Files</label>
+                                            <input id="gh-max-files" type="number" min="1" max="400" value="60"
+                                                class="w-full bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500">
+                                        </div>
+                                    </div>
+                                    <div id="github-indexing-status" class="text-xs text-gray-500"></div>
+                                </div>
                                 <div>
                                     <div class="text-xs font-semibold text-gray-400 mb-1 uppercase">Documents</div>
                                     <div id="rag-documents" class="space-y-2"></div>

static/dashboard.js

@@ -374,6 +374,11 @@ let supabase;
             if (el) el.textContent = text || '';
         }
 
+        function setGithubIndexingStatus(text) {
+            const el = document.getElementById('github-indexing-status');
+            if (el) el.textContent = text || '';
+        }
+
         function renderIndexingJobs(jobs) {
             const el = document.getElementById('indexing-jobs');
             if (!el) return;
@@ -401,7 +406,22 @@ let supabase;
                 const p = j?.progress || {};
                 const visited = p?.visited ?? 0;
                 const indexed = p?.indexedPages ?? 0;
-                meta.textContent = `${j?.createdAt || ''} • visited ${visited} • indexed ${indexed}`;
+                const params = j?.params || {};
+                const detail = (() => {
+                    if (j?.type === 'web_crawl') return params?.startUrl || '';
+                    if (j?.type === 'github_repo') {
+                        const ref = params?.ref ? `@${params.ref}` : '';
+                        const pref = params?.pathPrefix ? ` • ${params.pathPrefix}` : '';
+                        return `${params?.owner || ''}/${params?.repo || ''}${ref}${pref}`.trim();
+                    }
+                    return '';
+                })();
+                const indexedFiles = p?.indexedFiles ?? 0;
+                const bytes = p?.bytes ?? 0;
+                const progressBits = j?.type === 'github_repo'
+                    ? `files ${indexedFiles} • ${Math.round(bytes / 1024)} KB`
+                    : `visited ${visited} • indexed ${indexed}`;
+                meta.textContent = `${j?.createdAt || ''}${detail ? ' • ' + detail : ''} • ${progressBits}`;
                 left.appendChild(title);
                 left.appendChild(meta);
 
@@ -460,6 +480,31 @@ let supabase;
             }
         }
 
+        async function startGithubRepoJob() {
+            const repo = (document.getElementById('gh-repo')?.value || '').trim();
+            const ref = (document.getElementById('gh-ref')?.value || '').trim() || null;
+            const pathPrefix = (document.getElementById('gh-path')?.value || '').trim() || null;
+            const maxFiles = Number(document.getElementById('gh-max-files')?.value || 60);
+            if (!repo) {
+                setGithubIndexingStatus('Enter a repo (owner/repo).');
+                return;
+            }
+            try {
+                setGithubIndexingStatus('Starting job...');
+                const res = await authFetch('/api/indexing/jobs/github-repo', {
+                    method: 'POST',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify({ repo, ref, pathPrefix, maxFiles }),
+                });
+                if (!res.ok) throw new Error(await res.text());
+                await res.json();
+                setGithubIndexingStatus('Job started.');
+                loadIndexingJobs();
+            } catch (e) {
+                setGithubIndexingStatus(`Failed to start job: ${e?.message || e}`);
+            }
+        }
+
         async function cancelIndexingJob(jobId) {
             const id = (jobId || '').trim();
             if (!id) return;
@@ -2521,8 +2566,16 @@ let supabase;
             if (server.token) headers['Authorization'] = `Bearer ${server.token}`;
 
             try {
-                const res = await fetch(server.url, { method: 'GET', headers });
-                if (statusEl) statusEl.textContent = `HTTP ${res.status}`;
+                const res = await authFetch('/api/mcp/test', {
+                    method: 'POST',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify({ url: server.url, headers, timeoutSec: 3.0 }),
+                });
+                if (!res.ok) throw new Error(await res.text());
+                const data = await res.json();
+                const code = data?.statusCode ?? '—';
+                const ctype = data?.contentType ? ` • ${data.contentType}` : '';
+                if (statusEl) statusEl.textContent = `HTTP ${code}${ctype}`;
             } catch (e) {
                 if (statusEl) statusEl.textContent = `Error: ${e.message}`;
             }