wardrobe-os-ml / src /lib /error_scrub.py
fengchen31's picture
Sync from GitHub refs/heads/main @ 3da1b47
b288a25 verified
Raw
History Blame Contribute Delete
7.42 kB
"""Error scrubber——把 raw exception 收斂成 opaque category label。
【為什麼存在】
ml-service 的 sentry_sdk.capture_exception(exc) 會把 exc.args 原樣寫進
Sentry event 的 exception.values[].value 欄位:
- huggingface_hub.HfHubHTTPError 的 401 message 含 token=hf_…
- psycopg.OperationalError / asyncpg.* 的連線失敗 message 含 Neon DSN
- httpx.ConnectError 含完整 hostname + port
Sentry 整合 token 一旦外流(讀權限 → exception detail),attacker 直接拿
到 plaintext credentials,就算 breadcrumbs 全清也擋不住。
【設計】
鏡像 packages/error-scrubber/src/index.ts 的 scrubErrorForAudit:
1. class name 比對先(HfHubHTTPError、TimeoutError、StripeXxx、
PostgresError、DatabaseError、OperationalError、ReplyError)
2. 再 substring fallback(postgres://、rediss://、stripe、timeout、
401+(huggingface|hf_))
3. fall-through 一律 "unknown error"
純函數,不 raise,weird __str__ 用 try/except 收掉。Operator 從
stdout/file logger 撈 unredacted exc 對應 label 做 forensic correlation
(同 trace context / request id),Sentry 只見 label。
【caller】
- apps/ml-service/src/main.py:_on_preload_done (lifespan 預載 task)
- apps/ml-service/src/lib/background.py:_log_unhandled_exception
(fire-and-forget)
【Sibling rule】
TypeScript 端對應:`packages/error-scrubber/scrubErrorForAudit`。
Memory: feedback_python_sentry_capture_needs_scrub.md +
feedback_sentry_breadcrumb_bypasses_message_scrub.md。
"""
from __future__ import annotations
# 不要在這裡 import sentry_sdk / db / huggingface_hub——helper 只看
# class name (str) + message (str),不需要任何第三方型別。隔離 import 也
# 讓 helper 在最早期 (pre-init_sentry) 也能安全使用。
def scrub_exc_for_audit(exc: BaseException | None) -> str:
"""收斂任意 exception 為 opaque category label,永不 leak raw 字串。
【優先序】
class-name 檢查在前,substring 檢查在後——HfHubHTTPError 含 timeout
字眼仍判 hf auth;OperationalError 帶 timeout 仍判 db。class 才反映
故障性質,substring 只是 fallback for libraries that wrap exceptions
without preserving class name.
【return 值】
"hf auth error" | "timeout" | "stripe api error" | "db error"
| "redis error" | "network error" | "unknown error"
"""
# ── 0. None / 非 exception → unknown ────────────────────────────────
if exc is None or not isinstance(exc, BaseException):
return "unknown error"
name = type(exc).__name__
# exc.__str__ 可能 raise(自訂例外的怪 __str__)—收成空字串走 class-only path
try:
message = str(exc)
except Exception:
message = ""
lower = message.lower()
# ── 1. HF auth class(F3 新 bucket)─────────────────────────────────
# HfHubHTTPError class name 一見即判,不用看 message——huggingface_hub
# 401 / 403 / token expired 全部走這個 class。
#
# c-series wave-2 security-review:huggingface_hub.errors 下 4 個常見
# 子類別 class name 不以 "HfHub" 開頭,原 startswith 抓不到 →
# message(含 token / repo URL)落到 unknown error 後配合
# LoggingIntegration breadcrumb 進 Sentry。明列 catch。
if name == "HfHubHTTPError" or name.startswith("HfHub") or name in {
"RepositoryNotFoundError",
"EntryNotFoundError",
"LocalEntryNotFoundError",
"BadRequestError",
}:
return "hf auth error"
# ── 2. Timeout class——builtins.TimeoutError、asyncio.TimeoutError
# (Python 3.11+ 兩者已統一)、httpx.TimeoutException 等。
if name in {"TimeoutError", "TimeoutException", "AsyncTimeoutError"}:
return "timeout"
# ── 3. Stripe class——stripe-python SDK 例外多以 Stripe 為前綴
# (StripeError / CardError / APIConnectionError 屬 stripe.error 模組
# 但 class name 都 Stripe-prefix 或在 stripe namespace)。
if name.startswith("Stripe"):
return "stripe api error"
# ── 4. Postgres / DB class——psycopg、asyncpg、SQLAlchemy 都用以下名稱
#
# c-series wave-2 security-review:asyncpg exception 名稱跟 psycopg 不同;
# ConnectionFailureError / PoolConnectionError / ConnectionDoesNotExistError
# 內常嵌 'could not connect to server "ep-xxx.neon.tech"' 形式,原
# postgres:// substring fallback 抓不到 → 落 unknown error 後 LoggingIntegration
# breadcrumb 還是把 raw hostname 送進 Sentry。明列 catch。
# PoolTimeout / PoolClosedError 也加進來涵蓋 psycopg.pool / asyncpg.pool。
if name in {
"OperationalError",
"PostgresError",
"DatabaseError",
"InterfaceError",
"DataError",
"IntegrityError",
"InternalError",
"ProgrammingError",
"ConnectionFailureError", # asyncpg
"PoolConnectionError", # asyncpg
"ConnectionDoesNotExistError", # asyncpg
"PoolTimeout", # psycopg.pool
"PoolClosedError", # asyncpg / psycopg
}:
return "db error"
# ── 5. Redis class——redis-py 的 ConnectionError 跟 builtins 撞名
# 不能單看 class,要靠 substring;ReplyError 比較明確。
if name == "ReplyError":
return "redis error"
# ── 6. Substring:postgres URL(順序在 stripe substring 前——
# postgres error message 偶爾會帶 stripe webhook metadata 字串)
if "postgres://" in lower or "postgresql://" in lower:
return "db error"
# ── 7. Substring:HF 401 信號——token 字串或 huggingface domain
# 需要兩個訊號("401" + ("huggingface" or "hf_") )才判,避免任意
# 401 都誤判 HF。
if "401" in lower and ("huggingface" in lower or "hf_" in lower):
return "hf auth error"
# ── 8. Substring:stripe(third-party retry wrapper 包過 Stripe 例外
# 後 class name 會 lost)
if "stripe" in lower:
return "stripe api error"
# ── 9. Substring:redis URL
if "redis://" in lower or "rediss://" in lower:
return "redis error"
# ── 10. Substring:timeout 關鍵字(class check 沒抓到的 wrapped timeout)
if "timeout" in lower or "timed out" in lower:
return "timeout"
# ── 11. Substring:fetch / network 失敗信號(httpx ConnectError、
# OSError ECONNREFUSED 等的訊息常見字眼)。需配 class 才升級——
# 避免 user-controlled string 含 "ECONNREFUSED" 誤判。
if name in {"ConnectError", "NetworkError", "ConnectionError"} and (
"econnrefused" in lower
or "enotfound" in lower
or "etimedout" in lower
or "name or service not known" in lower
):
return "network error"
# ── 12. fall-through ───────────────────────────────────────────────
return "unknown error"