Surface real website-crawl failures instead of silently reporting fake success

Same bug class as the earlier Drive fix: if the homepage itself couldn't be
reached or came back as a bot-challenge/CAPTCHA interstitial (which we
already confirmed ezofis.com's SiteGround WAF does to automated clients),
crawl_website silently returned an empty list, and reindex_website reported
"success, 0 pages crawled" with no indication anything was wrong.

crawl_website now fetches the homepage first, outside the tolerant BFS loop,
and raises a clear RuntimeError if it's unreachable or is a bot-challenge
page. Individual sub-page failures deeper in the crawl are still tolerated
(a normal, acceptable partial result) — only a broken homepage fetch is
treated as a real failure, surfaced via reindex_website's existing error
handling into the Chat Agent status log.

Verified: a simulated WAF-block (the exact SiteGround captcha page observed
from ezofis.com) now marks the log status "error" with an actionable
message; a normal successful crawl still indexes correctly (regression
check).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

backend/app/chat_agent_crawler.py

@@ -221,27 +221,54 @@ def crawl_website(base_url: str, max_pages: int = MAX_PAGES) -> List[Dict[str, s
     so pages that aren't well-linked from nav still get discovered, then supplements via BFS
     link-following from every fetched page.
     Returns list of {"url": ..., "title": ..., "markdown": ...}.
+
+    Raises RuntimeError if the homepage itself can't be reached or comes back as a bot-challenge
+    page — a caller treating that the same as "0 real pages found" would misreport a broken crawl
+    (e.g. blocked by a WAF) as a successful index of an empty site.
     """
     base_url = (base_url or "").strip()
     if not base_url.startswith(("http://", "https://")):
         base_url = "https://" + base_url
     parsed_root = urlparse(base_url)
     if not parsed_root.netloc:
-        return []
+        raise RuntimeError(f"'{base_url}' is not a valid URL")
     root_host = parsed_root.netloc
 
     robots = _load_robots(base_url)
 
-    sitemap_urls = []
+    # Fetch the homepage first, outside the tolerant loop below — if this specific fetch fails
+    # or is a bot-challenge page, the crawl is fundamentally broken, not just "site has 0 pages".
+    home_resp = _fetch(base_url)
+    if home_resp is None:
+        raise RuntimeError(
+            f"Could not reach {base_url} (network error, or a non-2xx/non-HTML response after retries)."
+        )
+    home_title, home_text, home_links = _extract_title_text_links(home_resp.text, base_url)
+    if _looks_like_bot_challenge(home_text):
+        raise RuntimeError(
+            f"{base_url} appears to be behind bot/CAPTCHA protection (Cloudflare, SiteGround, Sucuri, "
+            "etc.) that blocks automated crawling — it returned a bot-challenge page instead of real "
+            "content. Try allowlisting this server, or add this content via the Google Drive "
+            "knowledge base instead."
+        )
+
+    sitemap_urls: List[str] = []
     try:
         sitemap_urls = _discover_sitemap_urls(base_url, root_host)
     except Exception as e:
         logger.warning("Sitemap discovery failed for %s: %s", base_url, e)
 
-    visited: Set[str] = set()
-    queue: List[str] = [base_url] + [u for u in sitemap_urls if u != base_url]
+    visited: Set[str] = {base_url}
+    queue: List[str] = [u for u in sitemap_urls if u != base_url]
     results: List[Dict[str, str]] = []
 
+    if len(home_text) >= 60:
+        markdown = f"# {home_title or base_url}\n\nSource: {base_url}\n\n{home_text[:20000]}"
+        results.append({"url": base_url, "title": home_title or base_url, "markdown": markdown})
+    for link in home_links:
+        if link not in visited:
+            queue.append(link)
+
     while queue and len(results) < max_pages:
         url = queue.pop(0)
         url, _ = urldefrag(url)