Z User commited on
Commit
b93f0d1
·
1 Parent(s): 4c14cbd

feat(jobs): server-driven bulk job runner — survives browser tab closure

Browse files

POST /api/v1/jobs accepts the whole list, validates the JWT + job ownership,
and finalizes the run server-side: per-chunk verifier_checks upserts,
processed_count/valid/risky/bad heartbeats on verifier_jobs, completed/error
finalization and risky+unprocessed refunds via the service-role ledger.

This removes the 'submitting client is gone' failure class (10 of the last 15
failed CSV runs on 2026-10-20/21): the browser no longer orchestrates chunking,
so a closed/throttled tab can no longer orphan a run holding thousands of
credits. classify_engine_result mirrors src/lib/verifier.ts classify() exactly
(NXDOMAIN -> Invalid/bad, dns_timeout/a_fallback_only -> Unknown/risky).

Frontend counterpart: runBulk v2 posts /jobs and polls the DB row only, with a
legacy chunk-loop fallback for engines predating this route (404/405).

Tests: tests/test_verifier_jobs.py (12 cases: classification edges, auth,
ownership, lifecycle finalization, chunk-failure refund path).

Files changed (4) hide show
  1. app/api/jobs.py +346 -0
  2. app/db/repo.py +67 -0
  3. app/main.py +4 -0
  4. tests/test_verifier_jobs.py +278 -0
app/api/jobs.py ADDED
@@ -0,0 +1,346 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Server-driven bulk job runner — the fix for the "submitting client is gone"
2
+ failure class (10 of the last 15 failed CSV runs on 2026-10-20/21).
3
+
4
+ Before this module the BROWSER orchestrated every bulk run: it chunked the
5
+ file, submitted each chunk to /validate/bulk, polled the engine, mirrored rows
6
+ into verifier_checks and finalized the job. If the tab closed, throttled or
7
+ slept mid-run, the job orphaned in 'processing' until the reaper closed it and
8
+ refunded the unprocessed remainder. The engine itself was healthy — its chunk
9
+ task even completed 500/500 while the browser row froze at 0/3000.
10
+
11
+ Contract (v2):
12
+ POST /api/v1/jobs {job_id, emails, deep?, final?}
13
+ - JWT required (productFetch always attaches the Supabase session token).
14
+ - verifier_jobs row must exist, belong to the JWT subject and still be
15
+ open (queued/processing).
16
+ - A background task validates the list in chunks (bulk_sync_chunk_size),
17
+ upserts verifier_checks rows after EVERY chunk, heartbeats verifier_jobs
18
+ (processed_count + counts + updated_at) so the dashboard and the reaper
19
+ both see live progress, and — on the final batch — flips the job to
20
+ 'completed' and refunds risky rows server-side.
21
+ - Any chunk-level failure finalizes the job as 'error' and refunds BOTH
22
+ the risky rows and the unprocessed remainder (same ledger mechanics as
23
+ the client finalize and the pg_cron reaper; idempotent via
24
+ refunded_credits).
25
+ The client polls verifier_jobs directly after accepting 202 — it no longer
26
+ needs to stay alive for the run to finish.
27
+ """
28
+ import asyncio
29
+ import json
30
+ import logging
31
+ import time
32
+ from typing import List, Optional
33
+
34
+ from fastapi import APIRouter, HTTPException, Request
35
+ from pydantic import BaseModel
36
+
37
+ from ..auth import verify_supabase_jwt
38
+ from ..config import settings
39
+ from ..db import repo
40
+ from ..models import ValidationResult
41
+ from ..validator.engine import engine
42
+
43
+ logger = logging.getLogger("emailvalidator.jobs")
44
+
45
+ router = APIRouter()
46
+
47
+ # Strong references so the event loop cannot GC an in-flight runner, and a
48
+ # semaphore so concurrent job submissions cannot spawn unbounded SMTP work.
49
+ _job_tasks: set[asyncio.Task] = set()
50
+ _job_semaphore = asyncio.Semaphore(settings.bulk_concurrency)
51
+
52
+ GROUPS = ("good", "risky", "bad")
53
+
54
+
55
+ class JobRequest(BaseModel):
56
+ job_id: str
57
+ emails: List[str]
58
+ deep: bool = True
59
+ final: bool = True
60
+ user_id: Optional[str] = None # informational; ownership comes from the JWT
61
+
62
+
63
+ # ---- classification (mirror of src/lib/verifier.ts classify()) ------------
64
+
65
+
66
+ def classify_engine_result(r: ValidationResult) -> dict:
67
+ """Map an engine ValidationResult to the verifier_checks row shape the
68
+ dashboard renders. Mirrors the frontend classify() branch order so a row
69
+ never disagrees with a client-side re-classification:
70
+ disposable → bad | dead domain → bad | dns inconclusive → risky |
71
+ valid+smtp → good | engine invalid → bad | catch-all → risky | else risky.
72
+ """
73
+ disposable = bool(getattr(r.disposable, "is_disposable", False))
74
+ catch_all = bool(getattr(r.catch_all, "is_catch_all", False))
75
+ dns = r.dns
76
+ status = (r.status.value if hasattr(r.status, "value") else str(r.status)) or "unknown"
77
+
78
+ checks_run = r.checks_run or []
79
+ dns_error = (dns.error or "") if dns else ""
80
+ dns_failure = dns_error == "dns_timeout" or "timeout" in dns_error.lower()
81
+ a_fallback_only = bool(dns and dns.domain_exists and not dns.has_mx_records)
82
+ smtp_verified = bool(r.smtp and r.smtp.verified is True)
83
+ reason = r.reason_code
84
+
85
+ if disposable:
86
+ out_status, result, grp = "disposable", "Disposable", "bad"
87
+ reason = reason or "disposable"
88
+ elif dns is not None and dns.domain_exists is False:
89
+ # NXDOMAIN is definitive (the stale-build regression that showed a
90
+ # 0% Bad column on a 3000-row list must never resurrect server-side).
91
+ out_status, result, grp = "invalid", "Invalid", "bad"
92
+ reason = reason or "nxdomain"
93
+ elif dns_failure or a_fallback_only:
94
+ out_status, result, grp = "unknown", "Unknown", "risky"
95
+ reason = reason or ("dns_timeout" if dns_failure else "a_fallback_only")
96
+ elif status == "valid" and smtp_verified:
97
+ out_status, result, grp = "valid", "Ok", "good"
98
+ elif status == "invalid":
99
+ out_status, result, grp = "invalid", "Invalid", "bad"
100
+ elif catch_all:
101
+ out_status, result, grp = "catch_all", "Catch-all", "risky"
102
+ reason = reason or "catch_all_detected"
103
+ else:
104
+ out_status, result, grp = "unknown", "Unknown", "risky"
105
+ reason = reason or "smtp_unavailable"
106
+
107
+ score = r.scoring.score if r.scoring is not None else None
108
+ return {
109
+ "email": r.email,
110
+ "status": out_status,
111
+ "result": result,
112
+ "grp": grp,
113
+ "score": score,
114
+ "reason": reason,
115
+ "pass_through": None,
116
+ "checks_run": checks_run,
117
+ "verified_at": r.validated_at,
118
+ "run_id": r.run_id,
119
+ "validator_version": r.validator_version,
120
+ }
121
+
122
+
123
+ # ---- persistence helpers (service-role writes, best-effort like repo) ------
124
+
125
+
126
+ async def _write_checks(rows: List[dict]) -> None:
127
+ if not rows:
128
+ return
129
+ try:
130
+ await asyncio.to_thread(repo.write_verifier_checks, rows)
131
+ except Exception as exc: # never lose a run to one write blip
132
+ logger.warning("job_checks_write_failed: %s", exc)
133
+
134
+
135
+ async def _update_job(job_id: str, **fields) -> None:
136
+ try:
137
+ await asyncio.to_thread(repo.update_verifier_job, job_id, fields)
138
+ except Exception as exc:
139
+ logger.warning("job_update_failed job=%s: %s", job_id, exc)
140
+
141
+
142
+ async def _finalize_failed(job_id: str, error: str, counts: dict, processed: int) -> None:
143
+ """Close a failed job and refund risky + unprocessed server-side.
144
+
145
+ Mirrors the ledger mechanics of public.refund_job_risky() (migration
146
+ 20261020000000) minus its auth.uid() guard — the engine writes with the
147
+ service-role key, under which auth.uid() is null and the SQL function
148
+ would raise. Idempotency comes from verifier_jobs.refunded_credits.
149
+ """
150
+ await _update_job(
151
+ job_id,
152
+ status="error",
153
+ error=error[:500],
154
+ processed_count=processed,
155
+ valid_count=counts.get("good", 0),
156
+ risky_count=counts.get("risky", 0),
157
+ bad_count=counts.get("bad", 0),
158
+ )
159
+ try:
160
+ job = await asyncio.to_thread(repo.get_verifier_job, job_id)
161
+ if not job:
162
+ return
163
+ total = int(job.get("total_count") or 0)
164
+ charged = int(job.get("credits_charged") or 0)
165
+ refunded = int(job.get("refunded_credits") or 0)
166
+ risky = int(job.get("risky_count") or 0)
167
+ unprocessed = max(total - int(job.get("processed_count") or 0), 0)
168
+ due = min(risky + unprocessed, charged) - refunded
169
+ if due > 0:
170
+ await asyncio.to_thread(repo.refund_credits, job["user_id"], due,
171
+ "Auto-refund — risky and unprocessed results are not billed")
172
+ await _update_job(job_id, refunded_credits=refunded + due)
173
+ except Exception as exc:
174
+ logger.warning("job_refund_failed job=%s: %s", job_id, exc)
175
+
176
+
177
+ # ---- the runner ------------------------------------------------------------
178
+
179
+
180
+ async def _run_verifier_job(job_id: str, user_id: str, emails: List[str], deep: bool,
181
+ final: bool) -> None:
182
+ started = time.monotonic()
183
+ counts = {g: 0 for g in GROUPS}
184
+ processed = 0
185
+ # Defensive dedupe (the frontend already deduped; a client bug must not
186
+ # double-write rows — (job_id, email) upsert would silently dedupe them
187
+ # while counts drift).
188
+ seen: set[str] = set()
189
+ ordered: List[str] = []
190
+ for e in emails:
191
+ e = (e or "").strip().lower()
192
+ if e and "@" in e and e not in seen:
193
+ seen.add(e)
194
+ ordered.append(e)
195
+ emails = ordered
196
+ total = len(emails)
197
+ if total == 0:
198
+ await _finalize_failed(job_id, "No valid addresses reached the engine.", counts, 0)
199
+ return
200
+
201
+ chunk_size = max(int(settings.bulk_sync_chunk_size or 50), 10)
202
+ async with _job_semaphore:
203
+ try:
204
+ await _update_job(job_id, status="processing")
205
+ for i in range(0, total, chunk_size):
206
+ chunk = emails[i:i + chunk_size]
207
+ try:
208
+ results = await engine.validate_batch(chunk, deep=deep)
209
+ except Exception as exc: # one bad stage must not kill the job
210
+ logger.warning("job_chunk_failed job=%s: %s", job_id, exc)
211
+ await _finalize_failed(
212
+ job_id,
213
+ f"Engine chunk failed after {processed}/{total} addresses: {exc}",
214
+ counts, processed,
215
+ )
216
+ return
217
+ if len(results) != len(chunk):
218
+ await _finalize_failed(
219
+ job_id,
220
+ f"Engine returned {len(results)} of {len(chunk)} results for one chunk",
221
+ counts, processed,
222
+ )
223
+ return
224
+ rows = [classify_engine_result(r) for r in results]
225
+ await _write_checks(rows)
226
+ for row in rows:
227
+ counts[row["grp"]] = counts.get(row["grp"], 0) + 1
228
+ processed += len(chunk)
229
+ # Heartbeat + live progress in one write; the dashboard polls
230
+ # this row instead of the engine, and the reaper reads it.
231
+ await _update_job(
232
+ job_id,
233
+ processed_count=processed,
234
+ valid_count=counts["good"],
235
+ risky_count=counts["risky"],
236
+ bad_count=counts["bad"],
237
+ )
238
+ if final:
239
+ await _update_job(
240
+ job_id,
241
+ status="completed",
242
+ error=None,
243
+ processed_count=processed,
244
+ valid_count=counts["good"],
245
+ risky_count=counts["risky"],
246
+ bad_count=counts["bad"],
247
+ )
248
+ # Risky rows are never billed; the completed-job refund covers
249
+ # exactly the risky_count (unprocessed term is zero by design).
250
+ try:
251
+ job = await asyncio.to_thread(repo.get_verifier_job, job_id)
252
+ if job:
253
+ charged = int(job.get("credits_charged") or 0)
254
+ refunded = int(job.get("refunded_credits") or 0)
255
+ due = min(counts["risky"], charged) - refunded
256
+ if due > 0:
257
+ await asyncio.to_thread(
258
+ repo.refund_credits, job["user_id"], due,
259
+ "Auto-refund — risky results are not billed")
260
+ await _update_job(job_id, refunded_credits=refunded + due)
261
+ except Exception as exc:
262
+ logger.warning("job_risky_refund_failed job=%s: %s", job_id, exc)
263
+ logger.info("job_batch_done job=%s processed=%s/%s final=%s secs=%.1f",
264
+ job_id, processed, total, final, time.monotonic() - started)
265
+ except Exception as exc: # absolute safety net
266
+ logger.exception("job_runner_crashed job=%s", job_id)
267
+ await _finalize_failed(job_id, f"Runner crashed: {exc}", counts, processed)
268
+
269
+
270
+ # ---- routes ----------------------------------------------------------------
271
+
272
+
273
+ @router.post("/jobs", tags=["Bulk Jobs"])
274
+ async def create_server_job(request: Request, payload: JobRequest):
275
+ """Accept a bulk job for server-side processing (202 + DB polling)."""
276
+ token = request.headers.get("authorization", "")
277
+ token = token[7:].strip() if token.lower().startswith("bearer ") else ""
278
+ if not token:
279
+ raise HTTPException(401, "Missing bearer token — sign in first.")
280
+ try:
281
+ claims = await verify_supabase_jwt(token)
282
+ except Exception as exc:
283
+ raise HTTPException(401, f"Invalid or expired token: {exc}")
284
+ user_id = claims.get("sub", "")
285
+ if not user_id:
286
+ raise HTTPException(401, "Token has no subject claim.")
287
+ if not payload.emails:
288
+ raise HTTPException(400, "emails must not be empty.")
289
+ if not settings.supabase_configured:
290
+ raise HTTPException(503, "Server persistence is not configured (SUPABASE_URL/SERVICE key).")
291
+
292
+ job = await asyncio.to_thread(repo.get_verifier_job, payload.job_id)
293
+ if not job:
294
+ raise HTTPException(404, f"Job {payload.job_id} not found — create it first.")
295
+ if str(job.get("user_id")) != user_id:
296
+ # Ownership mismatch: identical response for missing/foreign jobs.
297
+ raise HTTPException(404, f"Job {payload.job_id} not found for this account.")
298
+ if job.get("status") not in ("queued", "processing"):
299
+ raise HTTPException(409, f"Job {payload.job_id} is already {job.get('status')}.")
300
+ if not payload.emails:
301
+ raise HTTPException(400, "emails must not be empty.")
302
+
303
+ # Cap one submission so a giant file cannot pin the runner forever: the
304
+ # client splits into final-flagged batches (20k ≈ 600 KB JSON).
305
+ MAX_PER_CALL = 20000
306
+ emails = payload.emails[:MAX_PER_CALL]
307
+
308
+ task = asyncio.create_task(
309
+ _run_verifier_job(payload.job_id, user_id, emails, payload.deep, payload.final)
310
+ )
311
+ _job_tasks.add(task)
312
+ task.add_done_callback(_job_tasks.discard)
313
+
314
+ return {
315
+ "accepted": len(emails),
316
+ "job_id": payload.job_id,
317
+ "status": "processing",
318
+ "final": payload.final,
319
+ "message": "Processing server-side — poll verifier_jobs for progress.",
320
+ }
321
+
322
+
323
+ @router.get("/jobs/{job_id}", tags=["Bulk Jobs"])
324
+ async def job_status(job_id: str, request: Request):
325
+ """Lightweight engine-side echo of the job row (debug aid; the dashboard
326
+ polls the DB directly and does not depend on this)."""
327
+ token = request.headers.get("authorization", "")
328
+ token = token[7:].strip() if token.lower().startswith("bearer ") else ""
329
+ if not token:
330
+ raise HTTPException(401, "Missing bearer token — sign in first.")
331
+ try:
332
+ claims = await verify_supabase_jwt(token)
333
+ except Exception as exc:
334
+ raise HTTPException(401, f"Invalid or expired token: {exc}")
335
+ job = await asyncio.to_thread(repo.get_verifier_job, job_id)
336
+ if not job or str(job.get("user_id")) != claims.get("sub", ""):
337
+ raise HTTPException(404, f"Job {job_id} not found for this account.")
338
+ return {
339
+ "job_id": job_id,
340
+ "status": job.get("status"),
341
+ "total": job.get("total_count"),
342
+ "processed": job.get("processed_count"),
343
+ "valid": job.get("valid_count"),
344
+ "risky": job.get("risky_count"),
345
+ "bad": job.get("bad_count"),
346
+ }
app/db/repo.py CHANGED
@@ -220,3 +220,70 @@ def save_domain_record(domain: str, dns_result: Any) -> Optional[Dict[str, Any]]
220
  "has_dkim": bool(dns_result and dns_result.has_dkim),
221
  }
222
  return _upsert("domain_records", payload, on_conflict="domain")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
220
  "has_dkim": bool(dns_result and dns_result.has_dkim),
221
  }
222
  return _upsert("domain_records", payload, on_conflict="domain")
223
+
224
+
225
+ # ---------------------------------------------------------------------------
226
+ # Verifier jobs — server-driven bulk runs (app/api/jobs.py)
227
+ # The engine writes the same tables the browser used to write, so a job now
228
+ # finishes even when the submitting tab is gone.
229
+ # ---------------------------------------------------------------------------
230
+ def get_verifier_job(job_id: str) -> Optional[Dict[str, Any]]:
231
+ rows = _select("verifier_jobs", "*", limit=1, eq={"id": job_id})
232
+ return rows[0] if rows else None
233
+
234
+
235
+ def update_verifier_job(job_id: str, fields: Dict[str, Any]) -> None:
236
+ """Patch the job row. Callers pass business fields only; updated_at is
237
+ always touched so the stale-job reaper sees live heartbeats."""
238
+ payload = {**fields, "updated_at": "now()"}
239
+ client = _client()
240
+ if client is None:
241
+ return
242
+ try:
243
+ client.table("verifier_jobs").update(payload).eq("id", job_id).execute()
244
+ except Exception as exc:
245
+ logger.warning("supabase_verifier_job_update_failed", job_id=job_id, error=str(exc))
246
+
247
+
248
+ def write_verifier_checks(rows: List[Dict[str, Any]]) -> None:
249
+ """Upsert verifier_checks rows, idempotent on (job_id, email)."""
250
+ if not rows:
251
+ return
252
+ client = _client()
253
+ if client is None:
254
+ return
255
+ try:
256
+ client.table("verifier_checks").upsert(rows, on_conflict="job_id,email").execute()
257
+ except Exception as exc:
258
+ logger.warning("supabase_verifier_checks_upsert_failed", count=len(rows), error=str(exc))
259
+
260
+
261
+ def refund_credits(user_id: str, amount: int, description: str) -> None:
262
+ """Credit balance + ledger row (service-role twin of the SQL refund path).
263
+ Idempotency is the caller's duty via verifier_jobs.refunded_credits."""
264
+ if amount <= 0 or not user_id:
265
+ return
266
+ client = _client()
267
+ if client is None:
268
+ return
269
+ try:
270
+ # Atomic increment via an RPC is unavailable server-side without a
271
+ # dedicated function; PostgREST PATCH with a computed value races, so
272
+ # read-modify-write under a single-writer assumption (one runner task
273
+ # per job). The refunded_credits ledger column keeps each refund
274
+ # one-shot even if a retry replays the same due amount.
275
+ resp = client.table("user_credits").select("balance").eq("user_id", user_id).limit(1).execute()
276
+ rows = resp.data or []
277
+ balance = int(rows[0]["balance"]) if rows else 0
278
+ client.table("user_credits").update({
279
+ "balance": balance + int(amount),
280
+ "last_updated": "now()",
281
+ }).eq("user_id", user_id).execute()
282
+ client.table("credit_transactions").insert({
283
+ "user_id": user_id,
284
+ "type": "refund",
285
+ "amount": int(amount),
286
+ "description": description,
287
+ }).execute()
288
+ except Exception as exc:
289
+ logger.warning("supabase_refund_credits_failed", user_id=user_id, error=str(exc))
app/main.py CHANGED
@@ -10,6 +10,7 @@ import time
10
 
11
  from .config import settings
12
  from .api.routes import router
 
13
  from .auth import license_middleware
14
  from .utils.logger import setup_logging
15
  from .db import repo
@@ -176,6 +177,9 @@ async def global_exception_handler(request: Request, exc: Exception):
176
 
177
  # ---- Include Routes -------------------------------------------------
178
  app.include_router(router, prefix="/api/v1")
 
 
 
179
 
180
 
181
  # ---- Health alias for the website productApi probe -------------------
 
10
 
11
  from .config import settings
12
  from .api.routes import router
13
+ from .api.jobs import router as jobs_router
14
  from .auth import license_middleware
15
  from .utils.logger import setup_logging
16
  from .db import repo
 
177
 
178
  # ---- Include Routes -------------------------------------------------
179
  app.include_router(router, prefix="/api/v1")
180
+ # Server-driven bulk jobs (POST /api/v1/jobs): the engine finalizes the run
181
+ # with DB heartbeats so a closed browser tab can no longer orphan it.
182
+ app.include_router(jobs_router, prefix="/api/v1")
183
 
184
 
185
  # ---- Health alias for the website productApi probe -------------------
tests/test_verifier_jobs.py ADDED
@@ -0,0 +1,278 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Server-driven bulk job runner (app/api/jobs.py).
2
+
3
+ Regression class this locks down (2026-10-20/21 incidents, 10 of the last 15
4
+ failed CSV runs): the browser used to orchestrate chunking, so a closed tab
5
+ orphaned verifier_jobs rows in 'processing' holding thousands of credits. The
6
+ engine now finalizes the run server-side with per-chunk DB writes and
7
+ heartbeats, and refunds risky + unprocessed rows on failure.
8
+ """
9
+ import asyncio
10
+ import unittest
11
+ from unittest.mock import AsyncMock, MagicMock, patch
12
+
13
+ from fastapi.testclient import TestClient
14
+
15
+ from app.api import jobs as jobs_mod
16
+ from app.api.jobs import classify_engine_result
17
+ from app.main import app
18
+ from app.models import (
19
+ CatchAllResult,
20
+ DNSResult,
21
+ DisposableResult,
22
+ ValidationResult,
23
+ ScoringResult,
24
+ SMTPResult,
25
+ SyntaxResult,
26
+ ValidationStatus,
27
+ )
28
+
29
+ client = TestClient(app)
30
+
31
+
32
+ def _result(email: str, *, status=ValidationStatus.UNKNOWN, domain_exists=True,
33
+ has_mx=True, dns_error=None, disposable=False, catch_all=False,
34
+ smtp_verified=None, smtp_code=None, score=70, reason_code=None) -> ValidationResult:
35
+ dns = DNSResult(
36
+ domain_exists=domain_exists,
37
+ has_mx_records=has_mx,
38
+ error=dns_error,
39
+ error_code=dns_error,
40
+ )
41
+ syntax = SyntaxResult(passed=True, normalized_email=email, domain=email.split("@")[-1])
42
+ return ValidationResult(
43
+ email=email,
44
+ status=status,
45
+ syntax=syntax,
46
+ dns=dns,
47
+ disposable=DisposableResult(is_disposable=disposable, is_free_provider=False,
48
+ is_role_based=False, confidence=0.9) if disposable else None,
49
+ catch_all=CatchAllResult(is_catch_all=catch_all, confidence=0.9) if catch_all else None,
50
+ smtp=SMTPResult(verified=smtp_verified, smtp_code=smtp_code) if smtp_code or smtp_verified is not None else None,
51
+ scoring=ScoringResult(score=score, risk_level="medium", breakdown={}, warnings=[]),
52
+ processing_time_ms=1.0,
53
+ validated_at="2026-10-21T00:00:00+00:00",
54
+ checks_run=["syntax", "dns"],
55
+ reason_code=reason_code,
56
+ run_id="testrun",
57
+ validator_version="3.1.0",
58
+ )
59
+
60
+
61
+ class ClassifyTests(unittest.TestCase):
62
+ """classify_engine_result must mirror src/lib/verifier.ts classify()."""
63
+
64
+ def test_nxdomain_is_invalid_bad(self):
65
+ # 185/3000 rows in the 2026-10-21 run were dead domains mis-classified
66
+ # as Unknown/risky by the stale frontend build. Server-side rows must
67
+ # be Invalid/bad — a domain that does not exist cannot accept mail.
68
+ row = classify_engine_result(
69
+ _result("a@dead.test", status=ValidationStatus.INVALID,
70
+ domain_exists=False, has_mx=False, dns_error="nxdomain",
71
+ reason_code="nxdomain")
72
+ )
73
+ self.assertEqual(row["grp"], "bad")
74
+ self.assertEqual(row["status"], "invalid")
75
+ self.assertEqual(row["result"], "Invalid")
76
+
77
+ def test_smtp_unavailable_is_risky(self):
78
+ row = classify_engine_result(
79
+ _result("b@gmail.test", status=ValidationStatus.RISKY,
80
+ smtp_verified=None, reason_code="smtp_unavailable")
81
+ )
82
+ self.assertEqual(row["grp"], "risky")
83
+ self.assertEqual(row["result"], "Unknown")
84
+
85
+ def test_valid_with_smtp_is_good(self):
86
+ row = classify_engine_result(
87
+ _result("c@corp.test", status=ValidationStatus.VALID,
88
+ smtp_verified=True, smtp_code=250, score=95)
89
+ )
90
+ self.assertEqual(row["grp"], "good")
91
+ self.assertEqual(row["result"], "Ok")
92
+
93
+ def test_disposable_is_bad(self):
94
+ row = classify_engine_result(_result("x@mailinator.test", disposable=True))
95
+ self.assertEqual(row["grp"], "bad")
96
+ self.assertEqual(row["result"], "Disposable")
97
+
98
+ def test_catch_all_is_risky(self):
99
+ row = classify_engine_result(
100
+ _result("d@ca.test", status=ValidationStatus.RISKY, catch_all=True)
101
+ )
102
+ self.assertEqual(row["grp"], "risky")
103
+ self.assertEqual(row["result"], "Catch-all")
104
+
105
+ def test_dns_timeout_is_risky_not_bad(self):
106
+ # Transient DNS states are genuinely undecidable — golden invariant.
107
+ row = classify_engine_result(
108
+ _result("e@corp.test", status=ValidationStatus.UNKNOWN,
109
+ dns_error="timeout", reason_code="dns_timeout")
110
+ )
111
+ self.assertEqual(row["grp"], "risky")
112
+ self.assertEqual(row["reason"], "dns_timeout")
113
+
114
+
115
+ class JobEndpointTests(unittest.TestCase):
116
+ """POST /api/v1/jobs auth, ownership and lifecycle."""
117
+
118
+ def setUp(self):
119
+ self.job_row = {
120
+ "id": "11111111-1111-1111-1111-111111111111",
121
+ "user_id": "22222222-2222-2222-2222-222222222222",
122
+ "name": "t.csv",
123
+ "status": "queued",
124
+ "total_count": 3,
125
+ "processed_count": 0,
126
+ "valid_count": 0, "risky_count": 0, "bad_count": 0,
127
+ "credits_charged": 3, "refunded_credits": 0,
128
+ }
129
+ patcher = patch("app.api.jobs.repo", autospec=True)
130
+ self.repo_mock = patcher.start()
131
+ self.addCleanup(patcher.stop)
132
+ # The route 503s without persistence config; tests run without a real
133
+ # Supabase, so present the route with a configured project.
134
+ for field, value in (("supabase_url", "https://test.supabase.co"),
135
+ ("supabase_service_role_key", "svc-key-test")):
136
+ p = patch.object(jobs_mod.settings, field, value)
137
+ p.start()
138
+ self.addCleanup(p.stop)
139
+ self.repo_mock.get_verifier_job.return_value = self.job_row
140
+ self.repo_mock.update_verifier_job.return_value = None
141
+ self.repo_mock.write_verifier_checks.return_value = None
142
+ self.repo_mock.refund_credits.return_value = None
143
+
144
+ def _auth(self, sub="22222222-2222-2222-2222-222222222222"):
145
+ return patch(
146
+ "app.api.jobs.verify_supabase_jwt",
147
+ new=AsyncMock(return_value={"sub": sub, "email": "t@t.test"}),
148
+ )
149
+
150
+ def test_401_without_token(self):
151
+ res = client.post("/api/v1/jobs", json={"job_id": "x", "emails": ["a@b.test"]})
152
+ self.assertEqual(res.status_code, 401)
153
+
154
+ def test_404_foreign_job(self):
155
+ # Same-shaped id, DIFFERENT owner: the token subject does not match the
156
+ # row's user_id, and the route must answer 404 (never 403/200).
157
+ with self._auth(sub="99999999-9999-9999-9999-999999999999"):
158
+ res = client.post(
159
+ "/api/v1/jobs",
160
+ json={"job_id": "11111111-1111-1111-1111-111111111111",
161
+ "emails": ["a@b.test"]},
162
+ headers={"Authorization": "Bearer faketoken"},
163
+ )
164
+ self.assertEqual(res.status_code, 404)
165
+
166
+ def test_409_when_job_closed(self):
167
+ self.job_row["status"] = "completed"
168
+ with self._auth():
169
+ res = client.post(
170
+ "/api/v1/jobs",
171
+ json={"job_id": "11111111-1111-1111-1111-111111111111",
172
+ "emails": ["a@b.test"]},
173
+ headers={"Authorization": "Bearer faketoken"},
174
+ )
175
+ self.assertEqual(res.status_code, 409)
176
+
177
+ def test_400_empty_emails(self):
178
+ with self._auth():
179
+ res = client.post(
180
+ "/api/v1/jobs",
181
+ json={"job_id": "11111111-1111-1111-1111-111111111111", "emails": []},
182
+ headers={"Authorization": "Bearer faketoken"},
183
+ )
184
+ self.assertEqual(res.status_code, 400)
185
+
186
+ def test_run_finalizes_completed_and_refunds_risky(self):
187
+ """Happy path: 3 emails, 1 risky (smtp unavailable) → job completed,
188
+ risky row refunded once (refunded_credits ledger updated).
189
+ The endpoint wiring is asserted on the POST response; the lifecycle is
190
+ asserted by driving the runner coroutine to completion on a real loop
191
+ (TestClient does not guarantee background-task scheduling after the
192
+ response returns)."""
193
+
194
+ async def fake_batch(emails, deep=True):
195
+ return [
196
+ _result(emails[0], status=ValidationStatus.VALID, smtp_verified=True,
197
+ smtp_code=250, score=95),
198
+ _result(emails[1], status=ValidationStatus.RISKY, reason_code="smtp_unavailable"),
199
+ _result(emails[2], status=ValidationStatus.INVALID, domain_exists=False,
200
+ has_mx=False, dns_error="nxdomain", reason_code="nxdomain"),
201
+ ]
202
+
203
+ self.job_row["refunded_credits"] = 0
204
+ emails = ["a@x.test", "b@x.test", "c@x.test"]
205
+ # The endpoint spawns the runner as a fire-and-forget task; in tests
206
+ # that task would outlive the patch context and run the REAL engine on
207
+ # .test domains (all NXDOMAIN). Swallow the spawn — the lifecycle is
208
+ # driven deterministically below on a real loop.
209
+ def _no_task(coro):
210
+ coro.close()
211
+ return MagicMock()
212
+ with patch("app.api.jobs.engine.validate_batch", new=AsyncMock(side_effect=fake_batch)), \
213
+ patch("app.api.jobs.settings.bulk_sync_chunk_size", 2), \
214
+ patch("app.api.jobs.asyncio.create_task", side_effect=_no_task), \
215
+ self._auth():
216
+ res = client.post(
217
+ "/api/v1/jobs",
218
+ json={"job_id": "11111111-1111-1111-1111-111111111111",
219
+ "emails": emails},
220
+ headers={"Authorization": "Bearer faketoken"},
221
+ )
222
+ self.assertEqual(res.status_code, 200)
223
+ self.assertEqual(res.json()["accepted"], 3)
224
+ # Deterministic lifecycle: drive the same runner invocation to
225
+ # completion INSIDE the patch context (a test-client background
226
+ # task would outlive it and run the real engine on .test domains).
227
+ asyncio.run(jobs_mod._run_verifier_job(
228
+ "11111111-1111-1111-1111-111111111111",
229
+ "22222222-2222-2222-2222-222222222222", emails, True, True))
230
+ updates = [c.args[1] for c in self.repo_mock.update_verifier_job.call_args_list
231
+ if len(c.args) >= 2 and isinstance(c.args[1], dict)]
232
+ final = [u for u in updates if u.get("status") == "completed"]
233
+ self.assertTrue(final, f"runner never completed; updates={updates}")
234
+ last = final[-1]
235
+ self.assertEqual(last["bad_count"], 1) # nxdomain → bad
236
+ self.assertEqual(last["risky_count"], 1) # smtp unavailable → risky
237
+ self.assertEqual(last["valid_count"], 1)
238
+ # risky refund path executed server-side
239
+ self.repo_mock.refund_credits.assert_called_once()
240
+ self.assertEqual(self.repo_mock.refund_credits.call_args[0][1], 1)
241
+
242
+ def test_chunk_failure_finalizes_error_and_refunds(self):
243
+ """Engine crash mid-run → job error + risky + unprocessed refund."""
244
+
245
+ async def boom(emails, deep=True):
246
+ raise RuntimeError("smtp stage exploded")
247
+
248
+ self.job_row["total_count"] = 5
249
+ self.job_row["credits_charged"] = 5
250
+ def _no_task(coro):
251
+ coro.close()
252
+ return MagicMock()
253
+ with patch("app.api.jobs.engine.validate_batch", new=AsyncMock(side_effect=boom)), \
254
+ patch("app.api.jobs.asyncio.create_task", side_effect=_no_task), \
255
+ self._auth():
256
+ res = client.post(
257
+ "/api/v1/jobs",
258
+ json={"job_id": "11111111-1111-1111-1111-111111111111",
259
+ "emails": ["a@x.test"] * 5},
260
+ headers={"Authorization": "Bearer faketoken"},
261
+ )
262
+ self.assertEqual(res.status_code, 200)
263
+ # Deterministic lifecycle inside the patch context.
264
+ asyncio.run(jobs_mod._run_verifier_job(
265
+ "11111111-1111-1111-1111-111111111111",
266
+ "22222222-2222-2222-2222-222222222222", ["a@x.test"] * 5, True, True))
267
+ updates = [c.args[1] for c in self.repo_mock.update_verifier_job.call_args_list
268
+ if len(c.args) >= 2 and isinstance(c.args[1], dict)]
269
+ err_calls = [u for u in updates if u.get("status") == "error"]
270
+ self.assertTrue(err_calls, f"runner never finalized the failed job; updates={updates}")
271
+ self.assertIn("Engine chunk failed", err_calls[-1]["error"])
272
+ # refund = risky(0) + unprocessed(5) clamped to charged(5)
273
+ self.repo_mock.refund_credits.assert_called_once()
274
+ self.assertEqual(self.repo_mock.refund_credits.call_args[0][1], 5)
275
+
276
+
277
+ if __name__ == "__main__":
278
+ unittest.main()