feat(api): async job queue with comprehensive test coverage (#36)
Browse files* docs(bugs): add gateway timeout audit and deployment checklist
- Document Bug 003: HF Spaces ~60s proxy timeout risk for ML inference
- Add comprehensive deployment checklist with verification status
- Include E2E flow audit diagram for debugging reference
- Verify existing bug fixes (001, 002) are correct and complete
* feat(api): async job queue to eliminate gateway timeout
Implement async job queue pattern to handle HuggingFace Spaces' ~60s
gateway timeout for long-running ML inference (30-60s typical).
## Problem
- HF Spaces proxy has hard ~60s timeout
- DeepISLES inference takes 30-60s
- Intermittent 504 Gateway Timeout errors
## Solution
POST /api/segment now returns 202 Accepted immediately with job ID.
Frontend polls GET /api/jobs/{id} every 2s for status/progress/results.
No single request exceeds the timeout - completely eliminates the issue.
### Backend Changes
- Add job_store.py: Thread-safe in-memory job storage with TTL cleanup
- Update routes.py: Async job creation + background task execution
- Update schemas.py: CreateJobResponse, JobStatusResponse types
- Update main.py: Lifespan handler for job store initialization
### Frontend Changes
- Add ProgressIndicator component with animated progress bar
- Update useSegmentation hook with polling logic
- Update api/client.ts with createSegmentJob/getJobStatus methods
- Update App.tsx to show progress and cancel button
- Update types/index.ts with JobStatus types
- Update mock handlers for testing
### Documentation
- Add docs/specs/async-job-queue.md (full spec)
- Update docs/bugs/003-gateway-timeout-long-inference.md (FIXED)
- Update docs/bugs/README.md (checklist, E2E flow v2.0)
Performance: Initial response <1s (was 30-60s), zero timeout risk
* fix(test): comprehensive test fixes for async job queue
Frontend Test Fixes:
- Remove fake timers from App.test.tsx (incompatible with MSW polling)
- Use real timers with configurable mock job duration (500ms)
- Add setMockJobDuration() to handlers.ts for test configuration
- Fix ambiguous element queries (multiple elements with same text)
- Add proper timeout for multi-run test (15s for two job cycles)
- Use behavior-based assertions instead of implementation details
Backend Tests:
- Add comprehensive unit tests for job_store.py (20 tests)
- Job dataclass tests: elapsed time, to_dict(), status handling
- JobStore tests: create, get, start, update, complete, fail
- Cleanup tests: TTL expiration, file removal
- Global store tests: init/get patterns
- Update test_endpoints.py for async API (11 tests)
- POST /api/segment returns 202 with job ID
- GET /api/jobs/{id} returns status/progress/result
- Error handling tests
All tests now pass:
- Frontend: 63 tests
- Backend: 31 tests (API only)
* fix(lint): remove unused imports and fix type annotation
- Remove unused `time` and `field` imports from job_store.py
- Import AsyncIterator from collections.abc instead of typing
- Prefix unused `app` parameter with underscore in lifespan function
* style: format schemas.py to pass CI format check
* fix(e2e): update fixtures for async job queue API pattern
The e2e fixtures were mocking the old sync API (POST /api/segment returns
result directly). Updated to match the new async job queue pattern:
- POST /api/segment returns 202 with jobId
- GET /api/jobs/:jobId returns job status with progress
- Jobs progress over ~1 second from pending → running → completed
- Fixed processingText locator to use button role (avoid strict mode violation)
* fix(security): apply CodeRabbit security and quality fixes
Security fixes (CRITICAL):
- Add path traversal protection in cleanup_old_jobs()
- Validate job_id format to allow only alphanumeric, hyphens, underscores
- Use full UUID hex instead of truncated 8-char (prevents collisions)
- Sanitize error messages - don't expose raw exceptions to clients
Code quality improvements:
- Use 'is not None' checks in Job.to_dict() instead of truthiness
- Ensure started_at is set before computing elapsed time
- Prevent job_id overwrites with KeyError on duplicate
- Don't log case_id (potentially sensitive medical data)
- Fix misleading comment about lifespan vs import order
Documentation:
- Add 'text' language specifier to code fence blocks
- Note multi-worker would require shared store (Redis/DB)
---------
Co-authored-by: Claude <noreply@anthropic.com>
- docs/bugs/003-gateway-timeout-long-inference.md +143 -0
- docs/bugs/README.md +84 -0
- docs/specs/async-job-queue.md +600 -0
- frontend/e2e/fixtures.ts +126 -27
- frontend/e2e/pages/HomePage.ts +1 -1
- frontend/src/App.test.tsx +102 -32
- frontend/src/App.tsx +48 -5
- frontend/src/api/__tests__/client.test.ts +35 -20
- frontend/src/api/client.ts +52 -4
- frontend/src/components/ProgressIndicator.tsx +84 -0
- frontend/src/components/index.ts +1 -0
- frontend/src/hooks/__tests__/useSegmentation.test.tsx +95 -29
- frontend/src/hooks/useSegmentation.ts +191 -49
- frontend/src/mocks/handlers.ts +169 -15
- frontend/src/types/index.ts +25 -0
- src/stroke_deepisles_demo/api/job_store.py +380 -0
- src/stroke_deepisles_demo/api/main.py +76 -6
- src/stroke_deepisles_demo/api/routes.py +199 -30
- src/stroke_deepisles_demo/api/schemas.py +45 -2
- tests/api/test_endpoints.py +120 -72
- tests/api/test_job_store.py +314 -0
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Bug 003: Gateway Timeout Risk for Long ML Inference
|
| 2 |
+
|
| 3 |
+
**Status**: FIXED
|
| 4 |
+
**Date Found**: 2025-12-12
|
| 5 |
+
**Date Fixed**: 2025-12-12
|
| 6 |
+
**Severity**: Medium (was causing intermittent failures)
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
## Summary
|
| 11 |
+
|
| 12 |
+
HuggingFace Spaces has an approximately 60-second proxy/gateway timeout. The DeepISLES
|
| 13 |
+
ML inference typically takes 30-60 seconds in fast mode, which was causing intermittent
|
| 14 |
+
504 Gateway Timeout errors.
|
| 15 |
+
|
| 16 |
+
**Solution**: Implemented async job queue pattern with client-side polling.
|
| 17 |
+
|
| 18 |
+
## Original Problem
|
| 19 |
+
|
| 20 |
+
### HF Spaces Timeout Behavior
|
| 21 |
+
|
| 22 |
+
From HuggingFace community forums:
|
| 23 |
+
- "When requests take longer than a minute, users get a 504 timeout error"
|
| 24 |
+
- "After the POST request, the inference is run, but the API does not get the result
|
| 25 |
+
since it's long timed out by then"
|
| 26 |
+
|
| 27 |
+
### Symptoms
|
| 28 |
+
|
| 29 |
+
When this issue occurred:
|
| 30 |
+
1. User clicks "Run Segmentation"
|
| 31 |
+
2. UI shows "Processing..." for ~60 seconds
|
| 32 |
+
3. Browser receives 504 Gateway Timeout
|
| 33 |
+
4. Error displayed: "Segmentation failed: Gateway Timeout"
|
| 34 |
+
5. Backend may still complete the inference (results exist but response lost)
|
| 35 |
+
|
| 36 |
+
## Solution: Async Job Queue Pattern
|
| 37 |
+
|
| 38 |
+
### Architecture
|
| 39 |
+
|
| 40 |
+
```text
|
| 41 |
+
BEFORE (Synchronous - Timeout Risk):
|
| 42 |
+
Frontend Backend
|
| 43 |
+
|--POST /api/segment------->|
|
| 44 |
+
| (30-60s wait) |
|
| 45 |
+
|<--200 OK + results--------| # TIMEOUT!
|
| 46 |
+
|
| 47 |
+
AFTER (Async with Polling - No Timeout):
|
| 48 |
+
Frontend Backend
|
| 49 |
+
|--POST /api/segment------->|
|
| 50 |
+
|<--202 + jobId (<1s)-------|
|
| 51 |
+
|--GET /api/jobs/{id}------>|
|
| 52 |
+
|<--200 {progress: 30%}----|
|
| 53 |
+
|--GET /api/jobs/{id}------>|
|
| 54 |
+
|<--200 {progress: 70%}----|
|
| 55 |
+
|--GET /api/jobs/{id}------>|
|
| 56 |
+
|<--200 {complete, result}--|
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
### Implementation
|
| 60 |
+
|
| 61 |
+
#### Backend Changes
|
| 62 |
+
|
| 63 |
+
1. **Job Store** (`src/stroke_deepisles_demo/api/job_store.py`)
|
| 64 |
+
- In-memory job storage with thread-safe operations
|
| 65 |
+
- Automatic cleanup of old jobs (1 hour TTL)
|
| 66 |
+
- Progress tracking with status updates
|
| 67 |
+
|
| 68 |
+
2. **Routes** (`src/stroke_deepisles_demo/api/routes.py`)
|
| 69 |
+
- `POST /api/segment` returns 202 with job ID immediately
|
| 70 |
+
- `GET /api/jobs/{job_id}` returns current status/progress/results
|
| 71 |
+
- Background task executes inference
|
| 72 |
+
|
| 73 |
+
3. **Schemas** (`src/stroke_deepisles_demo/api/schemas.py`)
|
| 74 |
+
- `CreateJobResponse` for job creation
|
| 75 |
+
- `JobStatusResponse` for polling
|
| 76 |
+
|
| 77 |
+
#### Frontend Changes
|
| 78 |
+
|
| 79 |
+
1. **Types** (`frontend/src/types/index.ts`)
|
| 80 |
+
- `JobStatus`, `CreateJobResponse`, `JobStatusResponse`
|
| 81 |
+
|
| 82 |
+
2. **API Client** (`frontend/src/api/client.ts`)
|
| 83 |
+
- `createSegmentJob()` - creates job
|
| 84 |
+
- `getJobStatus()` - polls for status
|
| 85 |
+
|
| 86 |
+
3. **Hook** (`frontend/src/hooks/useSegmentation.ts`)
|
| 87 |
+
- Polls every 2 seconds
|
| 88 |
+
- Tracks progress, status, elapsed time
|
| 89 |
+
- Handles completion and errors
|
| 90 |
+
|
| 91 |
+
4. **Components**
|
| 92 |
+
- `ProgressIndicator` - shows progress bar and status
|
| 93 |
+
- `App` - integrates progress display and cancel button
|
| 94 |
+
|
| 95 |
+
### Spec Document
|
| 96 |
+
|
| 97 |
+
Full specification: `docs/specs/async-job-queue.md`
|
| 98 |
+
|
| 99 |
+
## Performance Impact
|
| 100 |
+
|
| 101 |
+
| Metric | Before (Sync) | After (Async) |
|
| 102 |
+
|--------|--------------|---------------|
|
| 103 |
+
| Initial response time | 30-60s | <1s |
|
| 104 |
+
| Total request count | 1 | ~15-30 (polling) |
|
| 105 |
+
| Timeout risk | HIGH | NONE |
|
| 106 |
+
| User feedback | None during wait | Real-time progress |
|
| 107 |
+
|
| 108 |
+
## Files Changed
|
| 109 |
+
|
| 110 |
+
### Backend
|
| 111 |
+
- `src/stroke_deepisles_demo/api/job_store.py` (NEW)
|
| 112 |
+
- `src/stroke_deepisles_demo/api/schemas.py`
|
| 113 |
+
- `src/stroke_deepisles_demo/api/routes.py`
|
| 114 |
+
- `src/stroke_deepisles_demo/api/main.py`
|
| 115 |
+
|
| 116 |
+
### Frontend
|
| 117 |
+
- `frontend/src/types/index.ts`
|
| 118 |
+
- `frontend/src/api/client.ts`
|
| 119 |
+
- `frontend/src/hooks/useSegmentation.ts`
|
| 120 |
+
- `frontend/src/components/ProgressIndicator.tsx` (NEW)
|
| 121 |
+
- `frontend/src/App.tsx`
|
| 122 |
+
- `frontend/src/mocks/handlers.ts`
|
| 123 |
+
|
| 124 |
+
### Tests
|
| 125 |
+
- `frontend/src/api/__tests__/client.test.ts`
|
| 126 |
+
- `frontend/src/hooks/__tests__/useSegmentation.test.tsx`
|
| 127 |
+
- `frontend/src/App.test.tsx`
|
| 128 |
+
|
| 129 |
+
## Verification
|
| 130 |
+
|
| 131 |
+
After fix:
|
| 132 |
+
1. Deploy backend to HF Spaces
|
| 133 |
+
2. Refresh frontend
|
| 134 |
+
3. Run segmentation on any case
|
| 135 |
+
4. Observe progress bar updating in real-time
|
| 136 |
+
5. Results display after completion - NO timeout errors
|
| 137 |
+
|
| 138 |
+
## References
|
| 139 |
+
|
| 140 |
+
- [FastAPI Background Tasks](https://fastapi.tiangolo.com/tutorial/background-tasks/)
|
| 141 |
+
- [FastAPI Polling Strategy](https://openillumi.com/en/en-fastapi-long-task-progress-polling/)
|
| 142 |
+
- [504 Gateway Timeout - HF Forums](https://discuss.huggingface.co/t/504-gateway-timeout-with-http-request/24018)
|
| 143 |
+
- [Real Time Polling in React Query 2025](https://samwithcode.in/tutorial/react-js/real-time-polling-in-react-query-2025)
|
|
@@ -12,6 +12,24 @@ None currently.
|
|
| 12 |
|----|-------|----------|--------|
|
| 13 |
| [001](./001-cors-static-files-hf-spaces.md) | CORS regex blocking static file requests | Critical | FIXED |
|
| 14 |
| [002](./002-http-vs-https-proxy-headers.md) | HTTP vs HTTPS URL mismatch behind proxy | High | FIXED |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
## Common HuggingFace Spaces Pitfalls
|
| 17 |
|
|
@@ -43,9 +61,75 @@ Based on research and experience, here are common issues to watch for:
|
|
| 43 |
- `SPACE_ID` contains the space identifier
|
| 44 |
- Use these to detect production environment
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
## Sources
|
| 47 |
|
| 48 |
- [Deploying FastAPI on HuggingFace Spaces](https://huggingface.co/blog/HemanthSai7/deploy-applications-on-huggingface-spaces)
|
| 49 |
- [HF Spaces Restrictions](https://medium.com/@na.mazaheri/deploying-a-fastapi-app-on-hugging-face-spaces-and-handling-all-its-restrictions-d494d97a78fa)
|
| 50 |
- [FastAPI HTTPS Discussion](https://github.com/fastapi/fastapi/discussions/6670)
|
| 51 |
- [HF Docker Spaces Docs](https://huggingface.co/docs/hub/en/spaces-sdks-docker)
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|----|-------|----------|--------|
|
| 13 |
| [001](./001-cors-static-files-hf-spaces.md) | CORS regex blocking static file requests | Critical | FIXED |
|
| 14 |
| [002](./002-http-vs-https-proxy-headers.md) | HTTP vs HTTPS URL mismatch behind proxy | High | FIXED |
|
| 15 |
+
| [003](./003-gateway-timeout-long-inference.md) | Gateway timeout for long ML inference | Medium | FIXED |
|
| 16 |
+
|
| 17 |
+
## HF Spaces Deployment Checklist
|
| 18 |
+
|
| 19 |
+
Last audit: 2025-12-12
|
| 20 |
+
|
| 21 |
+
| Check | Status | Notes |
|
| 22 |
+
|-------|--------|-------|
|
| 23 |
+
| CORS regex matches both URL formats | PASS | `r"https://.*stroke-viewer-frontend.*\.hf\.space"` |
|
| 24 |
+
| All URLs use HTTPS | PASS | `--proxy-headers` flag in Dockerfile |
|
| 25 |
+
| File outputs to /tmp/ | PASS | Uses `/tmp/stroke-results/` |
|
| 26 |
+
| Static files mounted after dir exists | PASS | `mkdir()` before `app.mount()` in main.py |
|
| 27 |
+
| HF_SPACES env var set | PASS | Set in Dockerfile |
|
| 28 |
+
| Using port 7860 | PASS | Configured in Dockerfile CMD |
|
| 29 |
+
| Inference timeout handled | PASS | Async job queue pattern (no timeout risk) |
|
| 30 |
+
| Error responses return JSON | PASS | HTTPException with detail |
|
| 31 |
+
| CORS preflight (OPTIONS) handled | PASS | CORSMiddleware handles automatically |
|
| 32 |
+
| Progress updates for long tasks | PASS | Polling with ProgressIndicator component |
|
| 33 |
|
| 34 |
## Common HuggingFace Spaces Pitfalls
|
| 35 |
|
|
|
|
| 61 |
- `SPACE_ID` contains the space identifier
|
| 62 |
- Use these to detect production environment
|
| 63 |
|
| 64 |
+
### 6. Gateway Timeouts (SOLVED)
|
| 65 |
+
- HF Spaces proxy has ~60 second timeout
|
| 66 |
+
- Solution: Async job queue pattern with polling
|
| 67 |
+
- POST returns immediately with job ID
|
| 68 |
+
- Frontend polls GET /api/jobs/{id} for progress
|
| 69 |
+
- See [Bug 003](./003-gateway-timeout-long-inference.md) and [Spec](../specs/async-job-queue.md)
|
| 70 |
+
|
| 71 |
+
## E2E Flow (v2.0 - Async Job Pattern)
|
| 72 |
+
|
| 73 |
+
The complete flow from frontend to backend and back:
|
| 74 |
+
|
| 75 |
+
```text
|
| 76 |
+
1. Frontend loads
|
| 77 |
+
├── CaseSelector fetches GET /api/cases
|
| 78 |
+
├── CORS: origin regex must match frontend URL
|
| 79 |
+
└── Response: JSON list of case IDs
|
| 80 |
+
|
| 81 |
+
2. User runs segmentation
|
| 82 |
+
├── App calls POST /api/segment {case_id, fast_mode}
|
| 83 |
+
├── Backend creates job record
|
| 84 |
+
└── Response: 202 Accepted + {jobId, status: "pending"}
|
| 85 |
+
|
| 86 |
+
3. Frontend polls for status
|
| 87 |
+
├── GET /api/jobs/{jobId} every 2 seconds
|
| 88 |
+
├── Response: {status, progress, progressMessage}
|
| 89 |
+
└── ProgressIndicator shows real-time updates
|
| 90 |
+
|
| 91 |
+
4. Backend processes (in background thread)
|
| 92 |
+
├── Job status: "running"
|
| 93 |
+
├── Progress updates: 10% → 30% → 85% → 95%
|
| 94 |
+
├── Runs DeepISLES inference
|
| 95 |
+
└── Writes results to /tmp/stroke-results/{jobId}/
|
| 96 |
+
|
| 97 |
+
5. Job completes
|
| 98 |
+
├── Status: "completed"
|
| 99 |
+
├── Result includes file URLs
|
| 100 |
+
└── Frontend stops polling
|
| 101 |
+
|
| 102 |
+
6. Frontend receives result
|
| 103 |
+
├── Updates state with URLs
|
| 104 |
+
├── Passes URLs to NiiVueViewer
|
| 105 |
+
└── Shows metrics in MetricsPanel
|
| 106 |
+
|
| 107 |
+
7. NiiVue fetches static files
|
| 108 |
+
├── Cross-origin fetch to backend /files/...
|
| 109 |
+
├── CORS headers on static file response
|
| 110 |
+
└── Binary NIfTI files download
|
| 111 |
+
|
| 112 |
+
8. Viewer displays
|
| 113 |
+
└── NIfTI volumes rendered in WebGL canvas
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
## API Endpoints (v2.0)
|
| 117 |
+
|
| 118 |
+
| Method | Endpoint | Description |
|
| 119 |
+
|--------|----------|-------------|
|
| 120 |
+
| GET | /api/cases | List available cases |
|
| 121 |
+
| POST | /api/segment | Create segmentation job (202 Accepted) |
|
| 122 |
+
| GET | /api/jobs/{id} | Get job status/progress/results |
|
| 123 |
+
| GET | /files/{jobId}/{caseId}/* | Static NIfTI files |
|
| 124 |
+
| GET | / | Health check |
|
| 125 |
+
| GET | /health | Detailed health with job count |
|
| 126 |
+
|
| 127 |
## Sources
|
| 128 |
|
| 129 |
- [Deploying FastAPI on HuggingFace Spaces](https://huggingface.co/blog/HemanthSai7/deploy-applications-on-huggingface-spaces)
|
| 130 |
- [HF Spaces Restrictions](https://medium.com/@na.mazaheri/deploying-a-fastapi-app-on-hugging-face-spaces-and-handling-all-its-restrictions-d494d97a78fa)
|
| 131 |
- [FastAPI HTTPS Discussion](https://github.com/fastapi/fastapi/discussions/6670)
|
| 132 |
- [HF Docker Spaces Docs](https://huggingface.co/docs/hub/en/spaces-sdks-docker)
|
| 133 |
+
- [504 Gateway Timeout - HF Forums](https://discuss.huggingface.co/t/504-gateway-timeout-with-http-request/24018)
|
| 134 |
+
- [FastAPI Background Tasks](https://fastapi.tiangolo.com/tutorial/background-tasks/)
|
| 135 |
+
- [FastAPI Polling Strategy](https://openillumi.com/en/en-fastapi-long-task-progress-polling/)
|
|
@@ -0,0 +1,600 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Async Job Queue for Long-Running ML Inference
|
| 2 |
+
|
| 3 |
+
**Status**: APPROVED
|
| 4 |
+
**Created**: 2025-12-12
|
| 5 |
+
**Author**: Claude Code Audit
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## Executive Summary
|
| 10 |
+
|
| 11 |
+
HuggingFace Spaces has a ~60-second gateway timeout that cannot be bypassed through
|
| 12 |
+
configuration. DeepISLES ML inference typically takes 30-60 seconds, creating
|
| 13 |
+
intermittent 504 Gateway Timeout errors. This spec defines a robust async job queue
|
| 14 |
+
system that eliminates timeout issues by immediately returning a job ID and using
|
| 15 |
+
client-side polling for status/results.
|
| 16 |
+
|
| 17 |
+
## Problem Statement
|
| 18 |
+
|
| 19 |
+
### Current Architecture (Synchronous)
|
| 20 |
+
|
| 21 |
+
```
|
| 22 |
+
Frontend Backend ML Inference
|
| 23 |
+
| | |
|
| 24 |
+
|--POST /api/segment------->| |
|
| 25 |
+
| |--run_pipeline_on_case()--->|
|
| 26 |
+
| | |
|
| 27 |
+
| (30-60s wait) | (processing) |
|
| 28 |
+
| | |
|
| 29 |
+
| |<---result------------------|
|
| 30 |
+
|<--200 OK + JSON-----------| |
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
**Problem**: HF Spaces proxy times out at ~60s, killing the connection before
|
| 34 |
+
the ML inference completes. The response is lost even though processing succeeds.
|
| 35 |
+
|
| 36 |
+
### Target Architecture (Async with Polling)
|
| 37 |
+
|
| 38 |
+
```
|
| 39 |
+
Frontend Backend ML Inference
|
| 40 |
+
| | |
|
| 41 |
+
|--POST /api/segment------->| |
|
| 42 |
+
|<--202 Accepted + job_id---| |
|
| 43 |
+
| |--BackgroundTask----------->|
|
| 44 |
+
| | |
|
| 45 |
+
|--GET /api/jobs/{id}------>| (processing) |
|
| 46 |
+
|<--200 {status: running}---| |
|
| 47 |
+
| | |
|
| 48 |
+
|--GET /api/jobs/{id}------>| |
|
| 49 |
+
|<--200 {status: running}---| |
|
| 50 |
+
| |<---result------------------|
|
| 51 |
+
|--GET /api/jobs/{id}------>| |
|
| 52 |
+
|<--200 {status: completed, | |
|
| 53 |
+
| result: {...}}-----| |
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
**Solution**: Initial request returns in <1s. Polling requests are fast (<100ms).
|
| 57 |
+
No single request exceeds the proxy timeout.
|
| 58 |
+
|
| 59 |
+
## Technical Design
|
| 60 |
+
|
| 61 |
+
### 1. Backend Job Store
|
| 62 |
+
|
| 63 |
+
In-memory dictionary storing job state. This is appropriate because:
|
| 64 |
+
- HF Spaces runs a single uvicorn worker (no multi-worker sync needed)
|
| 65 |
+
- Jobs are ephemeral (results cached, cleanup after 1 hour)
|
| 66 |
+
- No external dependencies (Redis, DB) required
|
| 67 |
+
|
| 68 |
+
```python
|
| 69 |
+
from dataclasses import dataclass, field
|
| 70 |
+
from datetime import datetime
|
| 71 |
+
from enum import Enum
|
| 72 |
+
from typing import Any
|
| 73 |
+
|
| 74 |
+
class JobStatus(str, Enum):
|
| 75 |
+
PENDING = "pending" # Job created, not started
|
| 76 |
+
RUNNING = "running" # Inference in progress
|
| 77 |
+
COMPLETED = "completed" # Success, results available
|
| 78 |
+
FAILED = "failed" # Error occurred
|
| 79 |
+
|
| 80 |
+
@dataclass
|
| 81 |
+
class Job:
|
| 82 |
+
id: str
|
| 83 |
+
status: JobStatus
|
| 84 |
+
case_id: str
|
| 85 |
+
fast_mode: bool
|
| 86 |
+
created_at: datetime
|
| 87 |
+
started_at: datetime | None = None
|
| 88 |
+
completed_at: datetime | None = None
|
| 89 |
+
progress: int = 0 # 0-100 percentage
|
| 90 |
+
progress_message: str = ""
|
| 91 |
+
result: dict[str, Any] | None = None
|
| 92 |
+
error: str | None = None
|
| 93 |
+
|
| 94 |
+
# Thread-safe job store (single writer pattern)
|
| 95 |
+
jobs: dict[str, Job] = {}
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
### 2. API Endpoints
|
| 99 |
+
|
| 100 |
+
#### POST /api/segment (Modified)
|
| 101 |
+
Returns immediately with job ID.
|
| 102 |
+
|
| 103 |
+
**Request**: Same as before
|
| 104 |
+
```json
|
| 105 |
+
{
|
| 106 |
+
"case_id": "sub-strokecase0001",
|
| 107 |
+
"fast_mode": true
|
| 108 |
+
}
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
**Response**: 202 Accepted
|
| 112 |
+
```json
|
| 113 |
+
{
|
| 114 |
+
"jobId": "a1b2c3d4",
|
| 115 |
+
"status": "pending",
|
| 116 |
+
"message": "Segmentation job queued"
|
| 117 |
+
}
|
| 118 |
+
```
|
| 119 |
+
|
| 120 |
+
#### GET /api/jobs/{job_id}
|
| 121 |
+
Poll for job status and results.
|
| 122 |
+
|
| 123 |
+
**Response (Running)**:
|
| 124 |
+
```json
|
| 125 |
+
{
|
| 126 |
+
"jobId": "a1b2c3d4",
|
| 127 |
+
"status": "running",
|
| 128 |
+
"progress": 45,
|
| 129 |
+
"progressMessage": "Running DeepISLES inference...",
|
| 130 |
+
"elapsedSeconds": 23.5
|
| 131 |
+
}
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
**Response (Completed)**:
|
| 135 |
+
```json
|
| 136 |
+
{
|
| 137 |
+
"jobId": "a1b2c3d4",
|
| 138 |
+
"status": "completed",
|
| 139 |
+
"progress": 100,
|
| 140 |
+
"progressMessage": "Segmentation complete",
|
| 141 |
+
"elapsedSeconds": 42.3,
|
| 142 |
+
"result": {
|
| 143 |
+
"caseId": "sub-strokecase0001",
|
| 144 |
+
"diceScore": 0.847,
|
| 145 |
+
"volumeMl": 12.34,
|
| 146 |
+
"dwiUrl": "https://...hf.space/files/a1b2c3d4/...",
|
| 147 |
+
"predictionUrl": "https://...hf.space/files/a1b2c3d4/..."
|
| 148 |
+
}
|
| 149 |
+
}
|
| 150 |
+
```
|
| 151 |
+
|
| 152 |
+
**Response (Failed)**:
|
| 153 |
+
```json
|
| 154 |
+
{
|
| 155 |
+
"jobId": "a1b2c3d4",
|
| 156 |
+
"status": "failed",
|
| 157 |
+
"progress": 0,
|
| 158 |
+
"progressMessage": "Error occurred",
|
| 159 |
+
"elapsedSeconds": 5.2,
|
| 160 |
+
"error": "Case not found: sub-invalid"
|
| 161 |
+
}
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
**Response (Not Found)**: 404
|
| 165 |
+
```json
|
| 166 |
+
{
|
| 167 |
+
"detail": "Job not found: xyz123"
|
| 168 |
+
}
|
| 169 |
+
```
|
| 170 |
+
|
| 171 |
+
### 3. Background Task Execution
|
| 172 |
+
|
| 173 |
+
```python
|
| 174 |
+
from fastapi import BackgroundTasks
|
| 175 |
+
|
| 176 |
+
@router.post("/segment", response_model=SegmentJobResponse, status_code=202)
|
| 177 |
+
def create_segment_job(
|
| 178 |
+
request: Request,
|
| 179 |
+
body: SegmentRequest,
|
| 180 |
+
background_tasks: BackgroundTasks
|
| 181 |
+
) -> SegmentJobResponse:
|
| 182 |
+
"""Create a segmentation job and return immediately."""
|
| 183 |
+
job_id = str(uuid.uuid4())[:8]
|
| 184 |
+
|
| 185 |
+
# Create job record
|
| 186 |
+
job = Job(
|
| 187 |
+
id=job_id,
|
| 188 |
+
status=JobStatus.PENDING,
|
| 189 |
+
case_id=body.case_id,
|
| 190 |
+
fast_mode=body.fast_mode,
|
| 191 |
+
created_at=datetime.now(),
|
| 192 |
+
)
|
| 193 |
+
jobs[job_id] = job
|
| 194 |
+
|
| 195 |
+
# Queue background task
|
| 196 |
+
background_tasks.add_task(
|
| 197 |
+
run_segmentation_job,
|
| 198 |
+
job_id=job_id,
|
| 199 |
+
case_id=body.case_id,
|
| 200 |
+
fast_mode=body.fast_mode,
|
| 201 |
+
backend_url=get_backend_base_url(request),
|
| 202 |
+
)
|
| 203 |
+
|
| 204 |
+
return SegmentJobResponse(
|
| 205 |
+
jobId=job_id,
|
| 206 |
+
status=JobStatus.PENDING,
|
| 207 |
+
message="Segmentation job queued",
|
| 208 |
+
)
|
| 209 |
+
```
|
| 210 |
+
|
| 211 |
+
### 4. Job Execution with Progress Updates
|
| 212 |
+
|
| 213 |
+
```python
|
| 214 |
+
def run_segmentation_job(
|
| 215 |
+
job_id: str,
|
| 216 |
+
case_id: str,
|
| 217 |
+
fast_mode: bool,
|
| 218 |
+
backend_url: str,
|
| 219 |
+
) -> None:
|
| 220 |
+
"""Execute segmentation in background thread."""
|
| 221 |
+
job = jobs.get(job_id)
|
| 222 |
+
if not job:
|
| 223 |
+
return
|
| 224 |
+
|
| 225 |
+
try:
|
| 226 |
+
# Mark as running
|
| 227 |
+
job.status = JobStatus.RUNNING
|
| 228 |
+
job.started_at = datetime.now()
|
| 229 |
+
job.progress = 10
|
| 230 |
+
job.progress_message = "Loading case data..."
|
| 231 |
+
|
| 232 |
+
# Run inference with progress callbacks
|
| 233 |
+
output_dir = RESULTS_BASE / job_id
|
| 234 |
+
|
| 235 |
+
job.progress = 20
|
| 236 |
+
job.progress_message = "Staging files for DeepISLES..."
|
| 237 |
+
|
| 238 |
+
result = run_pipeline_on_case(
|
| 239 |
+
case_id,
|
| 240 |
+
output_dir=output_dir,
|
| 241 |
+
fast=fast_mode,
|
| 242 |
+
compute_dice=True,
|
| 243 |
+
cleanup_staging=True,
|
| 244 |
+
# Future: pass progress_callback for finer updates
|
| 245 |
+
)
|
| 246 |
+
|
| 247 |
+
job.progress = 90
|
| 248 |
+
job.progress_message = "Computing metrics..."
|
| 249 |
+
|
| 250 |
+
# Compute volume
|
| 251 |
+
volume_ml = None
|
| 252 |
+
with contextlib.suppress(Exception):
|
| 253 |
+
volume_ml = round(compute_volume_ml(result.prediction_mask, threshold=0.5), 2)
|
| 254 |
+
|
| 255 |
+
# Build result
|
| 256 |
+
job.progress = 100
|
| 257 |
+
job.progress_message = "Segmentation complete"
|
| 258 |
+
job.status = JobStatus.COMPLETED
|
| 259 |
+
job.completed_at = datetime.now()
|
| 260 |
+
job.result = {
|
| 261 |
+
"caseId": result.case_id,
|
| 262 |
+
"diceScore": result.dice_score,
|
| 263 |
+
"volumeMl": volume_ml,
|
| 264 |
+
"elapsedSeconds": round(result.elapsed_seconds, 2),
|
| 265 |
+
"dwiUrl": f"{backend_url}/files/{job_id}/{result.case_id}/{result.input_files['dwi'].name}",
|
| 266 |
+
"predictionUrl": f"{backend_url}/files/{job_id}/{result.case_id}/{result.prediction_mask.name}",
|
| 267 |
+
}
|
| 268 |
+
|
| 269 |
+
except Exception as e:
|
| 270 |
+
job.status = JobStatus.FAILED
|
| 271 |
+
job.completed_at = datetime.now()
|
| 272 |
+
job.error = str(e)
|
| 273 |
+
job.progress_message = "Error occurred"
|
| 274 |
+
```
|
| 275 |
+
|
| 276 |
+
### 5. Job Cleanup (Memory Management)
|
| 277 |
+
|
| 278 |
+
```python
|
| 279 |
+
import threading
|
| 280 |
+
from datetime import timedelta
|
| 281 |
+
|
| 282 |
+
JOB_TTL = timedelta(hours=1) # Keep completed jobs for 1 hour
|
| 283 |
+
|
| 284 |
+
def cleanup_old_jobs() -> None:
|
| 285 |
+
"""Remove jobs older than TTL to prevent memory leaks."""
|
| 286 |
+
now = datetime.now()
|
| 287 |
+
expired = [
|
| 288 |
+
job_id for job_id, job in jobs.items()
|
| 289 |
+
if job.completed_at and (now - job.completed_at) > JOB_TTL
|
| 290 |
+
]
|
| 291 |
+
for job_id in expired:
|
| 292 |
+
# Also cleanup result files
|
| 293 |
+
result_dir = RESULTS_BASE / job_id
|
| 294 |
+
if result_dir.exists():
|
| 295 |
+
shutil.rmtree(result_dir, ignore_errors=True)
|
| 296 |
+
del jobs[job_id]
|
| 297 |
+
|
| 298 |
+
# Run cleanup every 10 minutes
|
| 299 |
+
def start_cleanup_scheduler():
|
| 300 |
+
def run():
|
| 301 |
+
while True:
|
| 302 |
+
time.sleep(600) # 10 minutes
|
| 303 |
+
cleanup_old_jobs()
|
| 304 |
+
|
| 305 |
+
thread = threading.Thread(target=run, daemon=True)
|
| 306 |
+
thread.start()
|
| 307 |
+
```
|
| 308 |
+
|
| 309 |
+
### 6. Frontend Polling Hook
|
| 310 |
+
|
| 311 |
+
```typescript
|
| 312 |
+
// hooks/useJobPolling.ts
|
| 313 |
+
import { useState, useEffect, useCallback, useRef } from 'react'
|
| 314 |
+
import { apiClient, JobStatus, JobStatusResponse } from '../api/client'
|
| 315 |
+
|
| 316 |
+
interface UseJobPollingOptions {
|
| 317 |
+
pollingInterval?: number // ms, default 2000
|
| 318 |
+
onComplete?: (result: SegmentationResult) => void
|
| 319 |
+
onError?: (error: string) => void
|
| 320 |
+
}
|
| 321 |
+
|
| 322 |
+
export function useJobPolling(options: UseJobPollingOptions = {}) {
|
| 323 |
+
const { pollingInterval = 2000, onComplete, onError } = options
|
| 324 |
+
|
| 325 |
+
const [jobId, setJobId] = useState<string | null>(null)
|
| 326 |
+
const [status, setStatus] = useState<JobStatus | null>(null)
|
| 327 |
+
const [progress, setProgress] = useState(0)
|
| 328 |
+
const [progressMessage, setProgressMessage] = useState('')
|
| 329 |
+
const [error, setError] = useState<string | null>(null)
|
| 330 |
+
const [isPolling, setIsPolling] = useState(false)
|
| 331 |
+
|
| 332 |
+
const intervalRef = useRef<number | null>(null)
|
| 333 |
+
const onCompleteRef = useRef(onComplete)
|
| 334 |
+
const onErrorRef = useRef(onError)
|
| 335 |
+
|
| 336 |
+
// Keep callbacks current
|
| 337 |
+
useEffect(() => {
|
| 338 |
+
onCompleteRef.current = onComplete
|
| 339 |
+
onErrorRef.current = onError
|
| 340 |
+
})
|
| 341 |
+
|
| 342 |
+
const stopPolling = useCallback(() => {
|
| 343 |
+
if (intervalRef.current) {
|
| 344 |
+
clearInterval(intervalRef.current)
|
| 345 |
+
intervalRef.current = null
|
| 346 |
+
}
|
| 347 |
+
setIsPolling(false)
|
| 348 |
+
}, [])
|
| 349 |
+
|
| 350 |
+
const pollJobStatus = useCallback(async (id: string) => {
|
| 351 |
+
try {
|
| 352 |
+
const response = await apiClient.getJobStatus(id)
|
| 353 |
+
|
| 354 |
+
setStatus(response.status)
|
| 355 |
+
setProgress(response.progress)
|
| 356 |
+
setProgressMessage(response.progressMessage)
|
| 357 |
+
|
| 358 |
+
if (response.status === 'completed' && response.result) {
|
| 359 |
+
stopPolling()
|
| 360 |
+
onCompleteRef.current?.(response.result)
|
| 361 |
+
} else if (response.status === 'failed') {
|
| 362 |
+
stopPolling()
|
| 363 |
+
setError(response.error || 'Job failed')
|
| 364 |
+
onErrorRef.current?.(response.error || 'Job failed')
|
| 365 |
+
}
|
| 366 |
+
} catch (err) {
|
| 367 |
+
// Don't stop polling on network errors - might be transient
|
| 368 |
+
console.warn('Polling error:', err)
|
| 369 |
+
}
|
| 370 |
+
}, [stopPolling])
|
| 371 |
+
|
| 372 |
+
const startJob = useCallback(async (caseId: string, fastMode = true) => {
|
| 373 |
+
// Reset state
|
| 374 |
+
setError(null)
|
| 375 |
+
setProgress(0)
|
| 376 |
+
setProgressMessage('Starting...')
|
| 377 |
+
setStatus('pending')
|
| 378 |
+
|
| 379 |
+
try {
|
| 380 |
+
// Create job
|
| 381 |
+
const response = await apiClient.createSegmentJob(caseId, fastMode)
|
| 382 |
+
setJobId(response.jobId)
|
| 383 |
+
setStatus(response.status)
|
| 384 |
+
|
| 385 |
+
// Start polling
|
| 386 |
+
setIsPolling(true)
|
| 387 |
+
intervalRef.current = window.setInterval(
|
| 388 |
+
() => pollJobStatus(response.jobId),
|
| 389 |
+
pollingInterval
|
| 390 |
+
)
|
| 391 |
+
|
| 392 |
+
// Initial poll
|
| 393 |
+
await pollJobStatus(response.jobId)
|
| 394 |
+
|
| 395 |
+
} catch (err) {
|
| 396 |
+
const message = err instanceof Error ? err.message : 'Failed to start job'
|
| 397 |
+
setError(message)
|
| 398 |
+
onErrorRef.current?.(message)
|
| 399 |
+
}
|
| 400 |
+
}, [pollingInterval, pollJobStatus])
|
| 401 |
+
|
| 402 |
+
// Cleanup on unmount
|
| 403 |
+
useEffect(() => {
|
| 404 |
+
return () => {
|
| 405 |
+
if (intervalRef.current) {
|
| 406 |
+
clearInterval(intervalRef.current)
|
| 407 |
+
}
|
| 408 |
+
}
|
| 409 |
+
}, [])
|
| 410 |
+
|
| 411 |
+
return {
|
| 412 |
+
jobId,
|
| 413 |
+
status,
|
| 414 |
+
progress,
|
| 415 |
+
progressMessage,
|
| 416 |
+
error,
|
| 417 |
+
isPolling,
|
| 418 |
+
startJob,
|
| 419 |
+
stopPolling,
|
| 420 |
+
}
|
| 421 |
+
}
|
| 422 |
+
```
|
| 423 |
+
|
| 424 |
+
### 7. Frontend API Client Extensions
|
| 425 |
+
|
| 426 |
+
```typescript
|
| 427 |
+
// api/client.ts additions
|
| 428 |
+
|
| 429 |
+
export type JobStatus = 'pending' | 'running' | 'completed' | 'failed'
|
| 430 |
+
|
| 431 |
+
export interface CreateJobResponse {
|
| 432 |
+
jobId: string
|
| 433 |
+
status: JobStatus
|
| 434 |
+
message: string
|
| 435 |
+
}
|
| 436 |
+
|
| 437 |
+
export interface JobStatusResponse {
|
| 438 |
+
jobId: string
|
| 439 |
+
status: JobStatus
|
| 440 |
+
progress: number
|
| 441 |
+
progressMessage: string
|
| 442 |
+
elapsedSeconds?: number
|
| 443 |
+
result?: SegmentResponse
|
| 444 |
+
error?: string
|
| 445 |
+
}
|
| 446 |
+
|
| 447 |
+
class ApiClient {
|
| 448 |
+
// ... existing methods ...
|
| 449 |
+
|
| 450 |
+
async createSegmentJob(
|
| 451 |
+
caseId: string,
|
| 452 |
+
fastMode: boolean = true,
|
| 453 |
+
signal?: AbortSignal
|
| 454 |
+
): Promise<CreateJobResponse> {
|
| 455 |
+
const response = await fetch(`${this.baseUrl}/api/segment`, {
|
| 456 |
+
method: 'POST',
|
| 457 |
+
headers: { 'Content-Type': 'application/json' },
|
| 458 |
+
body: JSON.stringify({ case_id: caseId, fast_mode: fastMode }),
|
| 459 |
+
signal,
|
| 460 |
+
})
|
| 461 |
+
|
| 462 |
+
if (!response.ok) {
|
| 463 |
+
const error = await response.json().catch(() => ({}))
|
| 464 |
+
throw new ApiError(
|
| 465 |
+
`Failed to create job: ${error.detail || response.statusText}`,
|
| 466 |
+
response.status,
|
| 467 |
+
error.detail
|
| 468 |
+
)
|
| 469 |
+
}
|
| 470 |
+
|
| 471 |
+
return response.json()
|
| 472 |
+
}
|
| 473 |
+
|
| 474 |
+
async getJobStatus(jobId: string, signal?: AbortSignal): Promise<JobStatusResponse> {
|
| 475 |
+
const response = await fetch(`${this.baseUrl}/api/jobs/${jobId}`, { signal })
|
| 476 |
+
|
| 477 |
+
if (response.status === 404) {
|
| 478 |
+
throw new ApiError('Job not found', 404)
|
| 479 |
+
}
|
| 480 |
+
|
| 481 |
+
if (!response.ok) {
|
| 482 |
+
const error = await response.json().catch(() => ({}))
|
| 483 |
+
throw new ApiError(
|
| 484 |
+
`Failed to get job status: ${error.detail || response.statusText}`,
|
| 485 |
+
response.status,
|
| 486 |
+
error.detail
|
| 487 |
+
)
|
| 488 |
+
}
|
| 489 |
+
|
| 490 |
+
return response.json()
|
| 491 |
+
}
|
| 492 |
+
}
|
| 493 |
+
```
|
| 494 |
+
|
| 495 |
+
### 8. UI Progress Display
|
| 496 |
+
|
| 497 |
+
```tsx
|
| 498 |
+
// components/ProgressIndicator.tsx
|
| 499 |
+
interface ProgressIndicatorProps {
|
| 500 |
+
progress: number
|
| 501 |
+
message: string
|
| 502 |
+
status: JobStatus
|
| 503 |
+
}
|
| 504 |
+
|
| 505 |
+
export function ProgressIndicator({ progress, message, status }: ProgressIndicatorProps) {
|
| 506 |
+
return (
|
| 507 |
+
<div className="bg-gray-800 rounded-lg p-4 space-y-3">
|
| 508 |
+
<div className="flex justify-between text-sm">
|
| 509 |
+
<span className="text-gray-400">{message}</span>
|
| 510 |
+
<span className="text-gray-300">{progress}%</span>
|
| 511 |
+
</div>
|
| 512 |
+
<div className="w-full bg-gray-700 rounded-full h-2">
|
| 513 |
+
<div
|
| 514 |
+
className={`h-2 rounded-full transition-all duration-300 ${
|
| 515 |
+
status === 'failed' ? 'bg-red-500' : 'bg-blue-500'
|
| 516 |
+
}`}
|
| 517 |
+
style={{ width: `${progress}%` }}
|
| 518 |
+
/>
|
| 519 |
+
</div>
|
| 520 |
+
</div>
|
| 521 |
+
)
|
| 522 |
+
}
|
| 523 |
+
```
|
| 524 |
+
|
| 525 |
+
## Implementation Checklist
|
| 526 |
+
|
| 527 |
+
### Backend
|
| 528 |
+
- [ ] Create `job_store.py` with Job dataclass and jobs dict
|
| 529 |
+
- [ ] Create Pydantic schemas for job responses
|
| 530 |
+
- [ ] Modify POST /api/segment to return 202 with job ID
|
| 531 |
+
- [ ] Add GET /api/jobs/{job_id} endpoint
|
| 532 |
+
- [ ] Implement background task execution with progress updates
|
| 533 |
+
- [ ] Add job cleanup scheduler
|
| 534 |
+
- [ ] Update CORS if needed for new endpoint
|
| 535 |
+
|
| 536 |
+
### Frontend
|
| 537 |
+
- [ ] Add job-related types to `types/index.ts`
|
| 538 |
+
- [ ] Add API client methods for job creation and polling
|
| 539 |
+
- [ ] Create `useJobPolling` hook
|
| 540 |
+
- [ ] Create `ProgressIndicator` component
|
| 541 |
+
- [ ] Update `useSegmentation` to use job polling
|
| 542 |
+
- [ ] Update `App.tsx` to show progress during processing
|
| 543 |
+
|
| 544 |
+
### Testing
|
| 545 |
+
- [ ] Unit tests for job store
|
| 546 |
+
- [ ] Unit tests for job endpoints
|
| 547 |
+
- [ ] Unit tests for useJobPolling hook
|
| 548 |
+
- [ ] E2E test for full job flow
|
| 549 |
+
- [ ] Manual test on HF Spaces deployment
|
| 550 |
+
|
| 551 |
+
### Documentation
|
| 552 |
+
- [ ] Update API documentation
|
| 553 |
+
- [ ] Update bug tracker with resolution
|
| 554 |
+
- [ ] Add architecture diagram
|
| 555 |
+
|
| 556 |
+
## Migration Strategy
|
| 557 |
+
|
| 558 |
+
1. **Backend**: Add new endpoints alongside existing. Keep old `/api/segment`
|
| 559 |
+
temporarily for backwards compatibility (marked deprecated).
|
| 560 |
+
|
| 561 |
+
2. **Frontend**: Update to use new job polling system. Old sync behavior removed.
|
| 562 |
+
|
| 563 |
+
3. **Testing**: Verify on HF Spaces before removing deprecated endpoint.
|
| 564 |
+
|
| 565 |
+
4. **Cleanup**: Remove deprecated sync endpoint after validation.
|
| 566 |
+
|
| 567 |
+
## Performance Considerations
|
| 568 |
+
|
| 569 |
+
| Metric | Before (Sync) | After (Async) |
|
| 570 |
+
|--------|--------------|---------------|
|
| 571 |
+
| Initial response time | 30-60s | <1s |
|
| 572 |
+
| Total request count | 1 | ~15-30 (polling) |
|
| 573 |
+
| Timeout risk | HIGH | NONE |
|
| 574 |
+
| User feedback | None during wait | Progress updates |
|
| 575 |
+
| Network efficiency | 1 large response | Many small responses |
|
| 576 |
+
|
| 577 |
+
## Alternatives Considered
|
| 578 |
+
|
| 579 |
+
### 1. SSE (Server-Sent Events)
|
| 580 |
+
- **Pros**: Real-time updates, single connection
|
| 581 |
+
- **Cons**: Connection stays open (could still timeout), HF proxy issues possible
|
| 582 |
+
- **Decision**: Polling is more robust for HF Spaces constraints
|
| 583 |
+
|
| 584 |
+
### 2. WebSockets
|
| 585 |
+
- **Pros**: Bi-directional, real-time
|
| 586 |
+
- **Cons**: Known 404 issues on HF Spaces, complex
|
| 587 |
+
- **Decision**: Not viable on HF Spaces
|
| 588 |
+
|
| 589 |
+
### 3. Redis/Celery
|
| 590 |
+
- **Pros**: Production-grade, multi-worker support
|
| 591 |
+
- **Cons**: Not available on HF Spaces Docker
|
| 592 |
+
- **Decision**: In-memory sufficient for single-worker
|
| 593 |
+
|
| 594 |
+
## References
|
| 595 |
+
|
| 596 |
+
- [FastAPI Background Tasks](https://fastapi.tiangolo.com/tutorial/background-tasks/)
|
| 597 |
+
- [FastAPI Polling Strategy for Long-Running Tasks](https://openillumi.com/en/en-fastapi-long-task-progress-polling/)
|
| 598 |
+
- [Managing Background Tasks in FastAPI](https://leapcell.io/blog/managing-background-tasks-and-long-running-operations-in-fastapi)
|
| 599 |
+
- [Real Time Polling in React Query 2025](https://samwithcode.in/tutorial/react-js/real-time-polling-in-react-query-2025)
|
| 600 |
+
- [504 Gateway Timeout - HF Forums](https://discuss.huggingface.co/t/504-gateway-timeout-with-http-request/24018)
|
|
@@ -1,48 +1,147 @@
|
|
| 1 |
-
import { test as base, expect } from '@playwright/test'
|
| 2 |
|
| 3 |
-
// API response mocks matching
|
| 4 |
const MOCK_CASES = ['sub-stroke0001', 'sub-stroke0002', 'sub-stroke0003']
|
| 5 |
-
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
diceScore: 0.847,
|
| 8 |
volumeMl: 15.32,
|
| 9 |
elapsedSeconds: 12.5,
|
| 10 |
// Use real public NIfTI for visual testing (NiiVue demo image)
|
| 11 |
dwiUrl: 'https://niivue.github.io/niivue-demo-images/mni152.nii.gz',
|
| 12 |
predictionUrl: 'https://niivue.github.io/niivue-demo-images/mni152.nii.gz',
|
| 13 |
-
}
|
| 14 |
|
| 15 |
-
//
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
})
|
| 26 |
})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
-
//
|
| 29 |
-
|
| 30 |
-
const request = route.request()
|
| 31 |
-
const body = JSON.parse(request.postData() || '{}') as { case_id?: string }
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
route.fulfill({
|
| 37 |
-
status:
|
| 38 |
contentType: 'application/json',
|
| 39 |
-
body: JSON.stringify({
|
| 40 |
-
...MOCK_SEGMENT_RESPONSE,
|
| 41 |
-
caseId: body.case_id || 'sub-stroke0001',
|
| 42 |
-
}),
|
| 43 |
})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
})
|
|
|
|
|
|
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
await use(page)
|
| 47 |
},
|
| 48 |
})
|
|
|
|
| 1 |
+
import { test as base, expect, Page } from '@playwright/test'
|
| 2 |
|
| 3 |
+
// API response mocks matching the async job queue pattern
|
| 4 |
const MOCK_CASES = ['sub-stroke0001', 'sub-stroke0002', 'sub-stroke0003']
|
| 5 |
+
|
| 6 |
+
// Track jobs for the async pattern
|
| 7 |
+
interface MockJob {
|
| 8 |
+
id: string
|
| 9 |
+
caseId: string
|
| 10 |
+
status: 'pending' | 'running' | 'completed' | 'failed'
|
| 11 |
+
progress: number
|
| 12 |
+
progressMessage: string
|
| 13 |
+
createdAt: number
|
| 14 |
+
}
|
| 15 |
+
|
| 16 |
+
// Job store per test (reset for each test)
|
| 17 |
+
const createJobStore = () => {
|
| 18 |
+
const jobs = new Map<string, MockJob>()
|
| 19 |
+
let jobCounter = 0
|
| 20 |
+
|
| 21 |
+
return {
|
| 22 |
+
createJob(caseId: string): MockJob {
|
| 23 |
+
const jobId = `e2e-job-${++jobCounter}`
|
| 24 |
+
const job: MockJob = {
|
| 25 |
+
id: jobId,
|
| 26 |
+
caseId,
|
| 27 |
+
status: 'pending',
|
| 28 |
+
progress: 0,
|
| 29 |
+
progressMessage: 'Job queued',
|
| 30 |
+
createdAt: Date.now(),
|
| 31 |
+
}
|
| 32 |
+
jobs.set(jobId, job)
|
| 33 |
+
return job
|
| 34 |
+
},
|
| 35 |
+
getJob(jobId: string): MockJob | undefined {
|
| 36 |
+
return jobs.get(jobId)
|
| 37 |
+
},
|
| 38 |
+
updateJobProgress(job: MockJob): MockJob {
|
| 39 |
+
// Simulate job progression over 1 second
|
| 40 |
+
const elapsed = Date.now() - job.createdAt
|
| 41 |
+
if (elapsed < 200) {
|
| 42 |
+
return { ...job, status: 'running', progress: 25, progressMessage: 'Loading case data...' }
|
| 43 |
+
} else if (elapsed < 500) {
|
| 44 |
+
return { ...job, status: 'running', progress: 50, progressMessage: 'Running inference...' }
|
| 45 |
+
} else if (elapsed < 800) {
|
| 46 |
+
return { ...job, status: 'running', progress: 75, progressMessage: 'Processing results...' }
|
| 47 |
+
} else {
|
| 48 |
+
return { ...job, status: 'completed', progress: 100, progressMessage: 'Segmentation complete' }
|
| 49 |
+
}
|
| 50 |
+
},
|
| 51 |
+
}
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
// Mock completed job result
|
| 55 |
+
const createMockResult = (caseId: string) => ({
|
| 56 |
+
caseId,
|
| 57 |
diceScore: 0.847,
|
| 58 |
volumeMl: 15.32,
|
| 59 |
elapsedSeconds: 12.5,
|
| 60 |
// Use real public NIfTI for visual testing (NiiVue demo image)
|
| 61 |
dwiUrl: 'https://niivue.github.io/niivue-demo-images/mni152.nii.gz',
|
| 62 |
predictionUrl: 'https://niivue.github.io/niivue-demo-images/mni152.nii.gz',
|
| 63 |
+
})
|
| 64 |
|
| 65 |
+
// Setup API mocking for async job queue pattern
|
| 66 |
+
async function setupApiMocks(page: Page) {
|
| 67 |
+
const jobStore = createJobStore()
|
| 68 |
+
|
| 69 |
+
// Mock GET /api/cases
|
| 70 |
+
await page.route('**/api/cases', (route) => {
|
| 71 |
+
route.fulfill({
|
| 72 |
+
status: 200,
|
| 73 |
+
contentType: 'application/json',
|
| 74 |
+
body: JSON.stringify({ cases: MOCK_CASES }),
|
|
|
|
| 75 |
})
|
| 76 |
+
})
|
| 77 |
+
|
| 78 |
+
// Mock POST /api/segment - returns 202 with job ID (async pattern)
|
| 79 |
+
await page.route('**/api/segment', async (route) => {
|
| 80 |
+
const request = route.request()
|
| 81 |
+
const body = JSON.parse(request.postData() || '{}') as { case_id?: string }
|
| 82 |
+
const caseId = body.case_id || 'sub-stroke0001'
|
| 83 |
|
| 84 |
+
// Create a new job
|
| 85 |
+
const job = jobStore.createJob(caseId)
|
|
|
|
|
|
|
| 86 |
|
| 87 |
+
// Small delay to simulate network
|
| 88 |
+
await new Promise((r) => setTimeout(r, 50))
|
| 89 |
+
|
| 90 |
+
route.fulfill({
|
| 91 |
+
status: 202,
|
| 92 |
+
contentType: 'application/json',
|
| 93 |
+
body: JSON.stringify({
|
| 94 |
+
jobId: job.id,
|
| 95 |
+
status: 'pending',
|
| 96 |
+
message: `Segmentation job queued for ${caseId}`,
|
| 97 |
+
}),
|
| 98 |
+
})
|
| 99 |
+
})
|
| 100 |
|
| 101 |
+
// Mock GET /api/jobs/:jobId - returns job status (for polling)
|
| 102 |
+
await page.route('**/api/jobs/*', async (route) => {
|
| 103 |
+
const url = route.request().url()
|
| 104 |
+
const jobId = url.split('/api/jobs/')[1]
|
| 105 |
+
|
| 106 |
+
const job = jobStore.getJob(jobId)
|
| 107 |
+
if (!job) {
|
| 108 |
route.fulfill({
|
| 109 |
+
status: 404,
|
| 110 |
contentType: 'application/json',
|
| 111 |
+
body: JSON.stringify({ detail: `Job not found: ${jobId}` }),
|
|
|
|
|
|
|
|
|
|
| 112 |
})
|
| 113 |
+
return
|
| 114 |
+
}
|
| 115 |
+
|
| 116 |
+
// Update job progress based on elapsed time
|
| 117 |
+
const updatedJob = jobStore.updateJobProgress(job)
|
| 118 |
+
|
| 119 |
+
const response: Record<string, unknown> = {
|
| 120 |
+
jobId: updatedJob.id,
|
| 121 |
+
status: updatedJob.status,
|
| 122 |
+
progress: updatedJob.progress,
|
| 123 |
+
progressMessage: updatedJob.progressMessage,
|
| 124 |
+
elapsedSeconds: (Date.now() - updatedJob.createdAt) / 1000,
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
// Include result when completed
|
| 128 |
+
if (updatedJob.status === 'completed') {
|
| 129 |
+
response.result = createMockResult(updatedJob.caseId)
|
| 130 |
+
}
|
| 131 |
+
|
| 132 |
+
route.fulfill({
|
| 133 |
+
status: 200,
|
| 134 |
+
contentType: 'application/json',
|
| 135 |
+
body: JSON.stringify(response),
|
| 136 |
})
|
| 137 |
+
})
|
| 138 |
+
}
|
| 139 |
|
| 140 |
+
// Extend base test to include API mocking
|
| 141 |
+
export const test = base.extend({
|
| 142 |
+
// Auto-mock API routes for every test
|
| 143 |
+
page: async ({ page }, use) => {
|
| 144 |
+
await setupApiMocks(page)
|
| 145 |
await use(page)
|
| 146 |
},
|
| 147 |
})
|
|
@@ -19,7 +19,7 @@ export class HomePage {
|
|
| 19 |
})
|
| 20 |
this.caseSelector = page.getByRole('combobox')
|
| 21 |
this.runButton = page.getByRole('button', { name: /run segmentation/i })
|
| 22 |
-
this.processingText = page.
|
| 23 |
this.metricsPanel = page.getByRole('heading', { name: /results/i })
|
| 24 |
this.diceScore = page.getByText(/0\.\d{3}/)
|
| 25 |
this.viewer = page.locator('canvas')
|
|
|
|
| 19 |
})
|
| 20 |
this.caseSelector = page.getByRole('combobox')
|
| 21 |
this.runButton = page.getByRole('button', { name: /run segmentation/i })
|
| 22 |
+
this.processingText = page.getByRole('button', { name: /processing/i })
|
| 23 |
this.metricsPanel = page.getByRole('heading', { name: /results/i })
|
| 24 |
this.diceScore = page.getByText(/0\.\d{3}/)
|
| 25 |
this.viewer = page.locator('canvas')
|
|
@@ -1,8 +1,8 @@
|
|
| 1 |
-
import { describe, it, expect, vi } from 'vitest'
|
| 2 |
import { render, screen, waitFor } from '@testing-library/react'
|
| 3 |
import userEvent from '@testing-library/user-event'
|
| 4 |
import { server } from './mocks/server'
|
| 5 |
-
import { errorHandlers } from './mocks/handlers'
|
| 6 |
import App from './App'
|
| 7 |
|
| 8 |
// Mock NiiVue to avoid WebGL in tests
|
|
@@ -19,6 +19,17 @@ vi.mock('@niivue/niivue', () => ({
|
|
| 19 |
}))
|
| 20 |
|
| 21 |
describe('App Integration', () => {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
describe('Initial Render', () => {
|
| 23 |
it('renders main heading', () => {
|
| 24 |
render(<App />)
|
|
@@ -94,10 +105,11 @@ describe('App Integration', () => {
|
|
| 94 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 95 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 96 |
|
| 97 |
-
|
|
|
|
| 98 |
})
|
| 99 |
|
| 100 |
-
it('
|
| 101 |
const user = userEvent.setup()
|
| 102 |
render(<App />)
|
| 103 |
|
|
@@ -108,12 +120,33 @@ describe('App Integration', () => {
|
|
| 108 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 109 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 110 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
await waitFor(() => {
|
| 112 |
-
expect(screen.
|
| 113 |
})
|
| 114 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
expect(screen.getByText('15.32 mL')).toBeInTheDocument()
|
| 116 |
-
expect(screen.getByText(/12\.5s/)).toBeInTheDocument()
|
| 117 |
})
|
| 118 |
|
| 119 |
it('displays viewer after successful segmentation', async () => {
|
|
@@ -127,9 +160,13 @@ describe('App Integration', () => {
|
|
| 127 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 128 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 129 |
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
})
|
| 134 |
|
| 135 |
it('hides placeholder after successful segmentation', async () => {
|
|
@@ -143,19 +180,37 @@ describe('App Integration', () => {
|
|
| 143 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 144 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 145 |
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
expect(
|
| 151 |
screen.queryByText(/select a case and run segmentation/i)
|
| 152 |
).not.toBeInTheDocument()
|
| 153 |
})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
})
|
| 155 |
|
| 156 |
describe('Error Handling', () => {
|
| 157 |
-
it('shows error when
|
| 158 |
-
server.use(errorHandlers.
|
| 159 |
const user = userEvent.setup()
|
| 160 |
|
| 161 |
render(<App />)
|
|
@@ -171,11 +226,11 @@ describe('App Integration', () => {
|
|
| 171 |
expect(screen.getByRole('alert')).toBeInTheDocument()
|
| 172 |
})
|
| 173 |
|
| 174 |
-
expect(screen.getByText(/
|
| 175 |
})
|
| 176 |
|
| 177 |
it('allows retry after error', async () => {
|
| 178 |
-
server.use(errorHandlers.
|
| 179 |
const user = userEvent.setup()
|
| 180 |
|
| 181 |
render(<App />)
|
|
@@ -197,16 +252,20 @@ describe('App Integration', () => {
|
|
| 197 |
// Retry
|
| 198 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 199 |
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 203 |
|
| 204 |
expect(screen.queryByRole('alert')).not.toBeInTheDocument()
|
| 205 |
})
|
| 206 |
})
|
| 207 |
|
| 208 |
describe('Multiple Runs', () => {
|
| 209 |
-
it('allows running segmentation on different cases', async () => {
|
| 210 |
const user = userEvent.setup()
|
| 211 |
render(<App />)
|
| 212 |
|
|
@@ -218,23 +277,34 @@ describe('App Integration', () => {
|
|
| 218 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 219 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 220 |
|
| 221 |
-
// Wait for first segmentation to complete
|
| 222 |
-
await waitFor(
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
|
| 231 |
// Second case
|
| 232 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0002')
|
| 233 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 234 |
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
})
|
| 239 |
})
|
| 240 |
})
|
|
|
|
| 1 |
+
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
| 2 |
import { render, screen, waitFor } from '@testing-library/react'
|
| 3 |
import userEvent from '@testing-library/user-event'
|
| 4 |
import { server } from './mocks/server'
|
| 5 |
+
import { errorHandlers, setMockJobDuration } from './mocks/handlers'
|
| 6 |
import App from './App'
|
| 7 |
|
| 8 |
// Mock NiiVue to avoid WebGL in tests
|
|
|
|
| 19 |
}))
|
| 20 |
|
| 21 |
describe('App Integration', () => {
|
| 22 |
+
// Use real timers for integration tests - fake timers don't sync well
|
| 23 |
+
// with MSW's async handlers and polling intervals
|
| 24 |
+
beforeEach(() => {
|
| 25 |
+
// Reset mock job duration to fast for tests
|
| 26 |
+
setMockJobDuration(500) // Jobs complete in 500ms
|
| 27 |
+
})
|
| 28 |
+
|
| 29 |
+
afterEach(() => {
|
| 30 |
+
setMockJobDuration(500) // Reset to default
|
| 31 |
+
})
|
| 32 |
+
|
| 33 |
describe('Initial Render', () => {
|
| 34 |
it('renders main heading', () => {
|
| 35 |
render(<App />)
|
|
|
|
| 105 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 106 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 107 |
|
| 108 |
+
// Button should show "Processing..." while job is running
|
| 109 |
+
expect(screen.getByRole('button', { name: /processing/i })).toBeInTheDocument()
|
| 110 |
})
|
| 111 |
|
| 112 |
+
it('shows progress indicator during job execution', async () => {
|
| 113 |
const user = userEvent.setup()
|
| 114 |
render(<App />)
|
| 115 |
|
|
|
|
| 120 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 121 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 122 |
|
| 123 |
+
// Progress indicator should appear during processing
|
| 124 |
+
await waitFor(() => {
|
| 125 |
+
expect(screen.getByRole('progressbar')).toBeInTheDocument()
|
| 126 |
+
})
|
| 127 |
+
})
|
| 128 |
+
|
| 129 |
+
it('displays metrics after successful segmentation', async () => {
|
| 130 |
+
const user = userEvent.setup()
|
| 131 |
+
render(<App />)
|
| 132 |
+
|
| 133 |
await waitFor(() => {
|
| 134 |
+
expect(screen.getByRole('combobox')).toBeInTheDocument()
|
| 135 |
})
|
| 136 |
|
| 137 |
+
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 138 |
+
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 139 |
+
|
| 140 |
+
// Wait for job to complete (mock duration is 500ms, polling is 2s)
|
| 141 |
+
// Use 5s timeout to account for polling interval
|
| 142 |
+
await waitFor(
|
| 143 |
+
() => {
|
| 144 |
+
expect(screen.getByText('0.847')).toBeInTheDocument()
|
| 145 |
+
},
|
| 146 |
+
{ timeout: 5000 }
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
expect(screen.getByText('15.32 mL')).toBeInTheDocument()
|
|
|
|
| 150 |
})
|
| 151 |
|
| 152 |
it('displays viewer after successful segmentation', async () => {
|
|
|
|
| 160 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 161 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 162 |
|
| 163 |
+
// Wait for job to complete and canvas to render
|
| 164 |
+
await waitFor(
|
| 165 |
+
() => {
|
| 166 |
+
expect(document.querySelector('canvas')).toBeInTheDocument()
|
| 167 |
+
},
|
| 168 |
+
{ timeout: 5000 }
|
| 169 |
+
)
|
| 170 |
})
|
| 171 |
|
| 172 |
it('hides placeholder after successful segmentation', async () => {
|
|
|
|
| 180 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 181 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 182 |
|
| 183 |
+
// Wait for job to complete
|
| 184 |
+
await waitFor(
|
| 185 |
+
() => {
|
| 186 |
+
expect(screen.getByText('0.847')).toBeInTheDocument()
|
| 187 |
+
},
|
| 188 |
+
{ timeout: 5000 }
|
| 189 |
+
)
|
| 190 |
|
| 191 |
expect(
|
| 192 |
screen.queryByText(/select a case and run segmentation/i)
|
| 193 |
).not.toBeInTheDocument()
|
| 194 |
})
|
| 195 |
+
|
| 196 |
+
it('shows cancel button during processing', async () => {
|
| 197 |
+
const user = userEvent.setup()
|
| 198 |
+
render(<App />)
|
| 199 |
+
|
| 200 |
+
await waitFor(() => {
|
| 201 |
+
expect(screen.getByRole('combobox')).toBeInTheDocument()
|
| 202 |
+
})
|
| 203 |
+
|
| 204 |
+
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 205 |
+
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 206 |
+
|
| 207 |
+
expect(screen.getByRole('button', { name: /cancel/i })).toBeInTheDocument()
|
| 208 |
+
})
|
| 209 |
})
|
| 210 |
|
| 211 |
describe('Error Handling', () => {
|
| 212 |
+
it('shows error when job creation fails', async () => {
|
| 213 |
+
server.use(errorHandlers.segmentCreateError)
|
| 214 |
const user = userEvent.setup()
|
| 215 |
|
| 216 |
render(<App />)
|
|
|
|
| 226 |
expect(screen.getByRole('alert')).toBeInTheDocument()
|
| 227 |
})
|
| 228 |
|
| 229 |
+
expect(screen.getByText(/failed to create job/i)).toBeInTheDocument()
|
| 230 |
})
|
| 231 |
|
| 232 |
it('allows retry after error', async () => {
|
| 233 |
+
server.use(errorHandlers.segmentCreateError)
|
| 234 |
const user = userEvent.setup()
|
| 235 |
|
| 236 |
render(<App />)
|
|
|
|
| 252 |
// Retry
|
| 253 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 254 |
|
| 255 |
+
// Wait for job to complete (real timer now)
|
| 256 |
+
await waitFor(
|
| 257 |
+
() => {
|
| 258 |
+
expect(screen.getByText('0.847')).toBeInTheDocument()
|
| 259 |
+
},
|
| 260 |
+
{ timeout: 5000 }
|
| 261 |
+
)
|
| 262 |
|
| 263 |
expect(screen.queryByRole('alert')).not.toBeInTheDocument()
|
| 264 |
})
|
| 265 |
})
|
| 266 |
|
| 267 |
describe('Multiple Runs', () => {
|
| 268 |
+
it('allows running segmentation on different cases', { timeout: 15000 }, async () => {
|
| 269 |
const user = userEvent.setup()
|
| 270 |
render(<App />)
|
| 271 |
|
|
|
|
| 277 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0001')
|
| 278 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 279 |
|
| 280 |
+
// Wait for first segmentation to complete - check metrics (Dice Score proves completion)
|
| 281 |
+
await waitFor(
|
| 282 |
+
() => {
|
| 283 |
+
expect(screen.getByText('0.847')).toBeInTheDocument()
|
| 284 |
+
// Button should no longer say "Processing..." after completion
|
| 285 |
+
expect(screen.queryByRole('button', { name: /processing/i })).not.toBeInTheDocument()
|
| 286 |
+
},
|
| 287 |
+
{ timeout: 5000 }
|
| 288 |
+
)
|
| 289 |
|
| 290 |
// Second case
|
| 291 |
await user.selectOptions(screen.getByRole('combobox'), 'sub-stroke0002')
|
| 292 |
await user.click(screen.getByRole('button', { name: /run segmentation/i }))
|
| 293 |
|
| 294 |
+
// Wait for second job to complete - check that case ID changed in metrics
|
| 295 |
+
// Note: We look within the metrics container for the case ID to avoid matching dropdown
|
| 296 |
+
await waitFor(
|
| 297 |
+
() => {
|
| 298 |
+
// The metrics panel shows case ID in a span with class "ml-2 font-mono"
|
| 299 |
+
// after the "Case:" label
|
| 300 |
+
const caseLabels = screen.getAllByText(/Case:/i)
|
| 301 |
+
expect(caseLabels.length).toBeGreaterThan(0)
|
| 302 |
+
// The second run should show sub-stroke0002 in the metrics
|
| 303 |
+
const metricsContainer = screen.getByText('Results').closest('div')
|
| 304 |
+
expect(metricsContainer).toHaveTextContent('sub-stroke0002')
|
| 305 |
+
},
|
| 306 |
+
{ timeout: 5000 }
|
| 307 |
+
)
|
| 308 |
})
|
| 309 |
})
|
| 310 |
})
|
|
@@ -3,11 +3,22 @@ import { Layout } from './components/Layout'
|
|
| 3 |
import { CaseSelector } from './components/CaseSelector'
|
| 4 |
import { NiiVueViewer } from './components/NiiVueViewer'
|
| 5 |
import { MetricsPanel } from './components/MetricsPanel'
|
|
|
|
| 6 |
import { useSegmentation } from './hooks/useSegmentation'
|
| 7 |
|
| 8 |
export default function App() {
|
| 9 |
const [selectedCase, setSelectedCase] = useState<string | null>(null)
|
| 10 |
-
const {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
const handleRunSegmentation = async () => {
|
| 13 |
if (selectedCase) {
|
|
@@ -15,6 +26,9 @@ export default function App() {
|
|
| 15 |
}
|
| 16 |
}
|
| 17 |
|
|
|
|
|
|
|
|
|
|
| 18 |
return (
|
| 19 |
<Layout>
|
| 20 |
<div className="grid grid-cols-1 lg:grid-cols-3 gap-6">
|
|
@@ -35,12 +49,39 @@ export default function App() {
|
|
| 35 |
{isLoading ? 'Processing...' : 'Run Segmentation'}
|
| 36 |
</button>
|
| 37 |
|
| 38 |
-
{
|
| 39 |
-
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
</div>
|
| 42 |
)}
|
| 43 |
|
|
|
|
| 44 |
{result && <MetricsPanel metrics={result.metrics} />}
|
| 45 |
</div>
|
| 46 |
|
|
@@ -54,7 +95,9 @@ export default function App() {
|
|
| 54 |
) : (
|
| 55 |
<div className="bg-gray-900 rounded-lg h-[500px] flex items-center justify-center">
|
| 56 |
<p className="text-gray-400">
|
| 57 |
-
|
|
|
|
|
|
|
| 58 |
</p>
|
| 59 |
</div>
|
| 60 |
)}
|
|
|
|
| 3 |
import { CaseSelector } from './components/CaseSelector'
|
| 4 |
import { NiiVueViewer } from './components/NiiVueViewer'
|
| 5 |
import { MetricsPanel } from './components/MetricsPanel'
|
| 6 |
+
import { ProgressIndicator } from './components/ProgressIndicator'
|
| 7 |
import { useSegmentation } from './hooks/useSegmentation'
|
| 8 |
|
| 9 |
export default function App() {
|
| 10 |
const [selectedCase, setSelectedCase] = useState<string | null>(null)
|
| 11 |
+
const {
|
| 12 |
+
result,
|
| 13 |
+
isLoading,
|
| 14 |
+
error,
|
| 15 |
+
jobStatus,
|
| 16 |
+
progress,
|
| 17 |
+
progressMessage,
|
| 18 |
+
elapsedSeconds,
|
| 19 |
+
runSegmentation,
|
| 20 |
+
cancelJob,
|
| 21 |
+
} = useSegmentation()
|
| 22 |
|
| 23 |
const handleRunSegmentation = async () => {
|
| 24 |
if (selectedCase) {
|
|
|
|
| 26 |
}
|
| 27 |
}
|
| 28 |
|
| 29 |
+
// Show progress indicator when job is active
|
| 30 |
+
const showProgress = isLoading && jobStatus && jobStatus !== 'completed'
|
| 31 |
+
|
| 32 |
return (
|
| 33 |
<Layout>
|
| 34 |
<div className="grid grid-cols-1 lg:grid-cols-3 gap-6">
|
|
|
|
| 49 |
{isLoading ? 'Processing...' : 'Run Segmentation'}
|
| 50 |
</button>
|
| 51 |
|
| 52 |
+
{/* Cancel button when processing */}
|
| 53 |
+
{isLoading && (
|
| 54 |
+
<button
|
| 55 |
+
onClick={cancelJob}
|
| 56 |
+
className="w-full bg-gray-700 hover:bg-gray-600 text-gray-300
|
| 57 |
+
font-medium py-2 px-4 rounded-lg transition-colors text-sm"
|
| 58 |
+
>
|
| 59 |
+
Cancel
|
| 60 |
+
</button>
|
| 61 |
+
)}
|
| 62 |
+
|
| 63 |
+
{/* Progress indicator */}
|
| 64 |
+
{showProgress && (
|
| 65 |
+
<ProgressIndicator
|
| 66 |
+
progress={progress}
|
| 67 |
+
message={progressMessage}
|
| 68 |
+
status={jobStatus}
|
| 69 |
+
elapsedSeconds={elapsedSeconds}
|
| 70 |
+
/>
|
| 71 |
+
)}
|
| 72 |
+
|
| 73 |
+
{/* Error display */}
|
| 74 |
+
{error && !isLoading && (
|
| 75 |
+
<div
|
| 76 |
+
role="alert"
|
| 77 |
+
className="bg-red-900/50 text-red-300 p-3 rounded-lg text-sm"
|
| 78 |
+
>
|
| 79 |
+
<p className="font-medium">Error</p>
|
| 80 |
+
<p className="mt-1">{error}</p>
|
| 81 |
</div>
|
| 82 |
)}
|
| 83 |
|
| 84 |
+
{/* Results metrics */}
|
| 85 |
{result && <MetricsPanel metrics={result.metrics} />}
|
| 86 |
</div>
|
| 87 |
|
|
|
|
| 95 |
) : (
|
| 96 |
<div className="bg-gray-900 rounded-lg h-[500px] flex items-center justify-center">
|
| 97 |
<p className="text-gray-400">
|
| 98 |
+
{isLoading
|
| 99 |
+
? 'Processing segmentation...'
|
| 100 |
+
: 'Select a case and run segmentation to view results'}
|
| 101 |
</p>
|
| 102 |
</div>
|
| 103 |
)}
|
|
@@ -25,37 +25,52 @@ describe('apiClient', () => {
|
|
| 25 |
})
|
| 26 |
})
|
| 27 |
|
| 28 |
-
describe('
|
| 29 |
-
it('returns
|
| 30 |
-
const result = await apiClient.
|
| 31 |
|
| 32 |
-
expect(result.
|
| 33 |
-
expect(result.
|
| 34 |
-
expect(result.
|
| 35 |
-
expect(result.dwiUrl).toContain('dwi.nii.gz')
|
| 36 |
-
expect(result.predictionUrl).toContain('prediction.nii.gz')
|
| 37 |
})
|
| 38 |
|
| 39 |
-
it('sends fast_mode
|
| 40 |
-
const result = await apiClient.
|
| 41 |
|
| 42 |
-
|
| 43 |
-
expect(result.
|
| 44 |
})
|
| 45 |
|
| 46 |
-
it('
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
|
|
|
| 51 |
})
|
|
|
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
|
| 56 |
await expect(
|
| 57 |
-
apiClient.
|
| 58 |
-
).rejects.toThrow(/
|
| 59 |
})
|
| 60 |
})
|
| 61 |
})
|
|
|
|
| 25 |
})
|
| 26 |
})
|
| 27 |
|
| 28 |
+
describe('createSegmentJob', () => {
|
| 29 |
+
it('returns job ID and pending status', async () => {
|
| 30 |
+
const result = await apiClient.createSegmentJob('sub-stroke0001')
|
| 31 |
|
| 32 |
+
expect(result.jobId).toBeDefined()
|
| 33 |
+
expect(result.status).toBe('pending')
|
| 34 |
+
expect(result.message).toContain('sub-stroke0001')
|
|
|
|
|
|
|
| 35 |
})
|
| 36 |
|
| 37 |
+
it('sends fast_mode parameter', async () => {
|
| 38 |
+
const result = await apiClient.createSegmentJob('sub-stroke0001', false)
|
| 39 |
|
| 40 |
+
expect(result.jobId).toBeDefined()
|
| 41 |
+
expect(result.status).toBe('pending')
|
| 42 |
})
|
| 43 |
|
| 44 |
+
it('throws ApiError on server error', async () => {
|
| 45 |
+
server.use(errorHandlers.segmentCreateError)
|
| 46 |
|
| 47 |
+
await expect(
|
| 48 |
+
apiClient.createSegmentJob('sub-stroke0001')
|
| 49 |
+
).rejects.toThrow(/failed to create job/i)
|
| 50 |
})
|
| 51 |
+
})
|
| 52 |
|
| 53 |
+
describe('getJobStatus', () => {
|
| 54 |
+
it('returns job status with progress', async () => {
|
| 55 |
+
// First create a job
|
| 56 |
+
const createResult = await apiClient.createSegmentJob('sub-stroke0001')
|
| 57 |
+
|
| 58 |
+
// Then get its status
|
| 59 |
+
const status = await apiClient.getJobStatus(createResult.jobId)
|
| 60 |
+
|
| 61 |
+
expect(status.jobId).toBe(createResult.jobId)
|
| 62 |
+
expect(['pending', 'running', 'completed']).toContain(status.status)
|
| 63 |
+
expect(status.progress).toBeGreaterThanOrEqual(0)
|
| 64 |
+
expect(status.progress).toBeLessThanOrEqual(100)
|
| 65 |
+
expect(status.progressMessage).toBeDefined()
|
| 66 |
+
})
|
| 67 |
+
|
| 68 |
+
it('throws ApiError when job not found', async () => {
|
| 69 |
+
server.use(errorHandlers.jobNotFound)
|
| 70 |
|
| 71 |
await expect(
|
| 72 |
+
apiClient.getJobStatus('nonexistent-job')
|
| 73 |
+
).rejects.toThrow(/not found/i)
|
| 74 |
})
|
| 75 |
})
|
| 76 |
})
|
|
@@ -1,4 +1,8 @@
|
|
| 1 |
-
import type {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
function getApiBase(): string {
|
| 4 |
const url = import.meta.env.VITE_API_URL
|
|
@@ -36,6 +40,9 @@ class ApiClient {
|
|
| 36 |
this.baseUrl = baseUrl
|
| 37 |
}
|
| 38 |
|
|
|
|
|
|
|
|
|
|
| 39 |
async getCases(signal?: AbortSignal): Promise<CasesResponse> {
|
| 40 |
const response = await fetch(`${this.baseUrl}/api/cases`, { signal })
|
| 41 |
|
|
@@ -51,11 +58,17 @@ class ApiClient {
|
|
| 51 |
return response.json()
|
| 52 |
}
|
| 53 |
|
| 54 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
caseId: string,
|
| 56 |
fastMode: boolean = true,
|
| 57 |
signal?: AbortSignal
|
| 58 |
-
): Promise<
|
| 59 |
const response = await fetch(`${this.baseUrl}/api/segment`, {
|
| 60 |
method: 'POST',
|
| 61 |
headers: {
|
|
@@ -71,7 +84,42 @@ class ApiClient {
|
|
| 71 |
if (!response.ok) {
|
| 72 |
const error = await response.json().catch(() => ({}))
|
| 73 |
throw new ApiError(
|
| 74 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
response.status,
|
| 76 |
error.detail
|
| 77 |
)
|
|
|
|
| 1 |
+
import type {
|
| 2 |
+
CasesResponse,
|
| 3 |
+
CreateJobResponse,
|
| 4 |
+
JobStatusResponse,
|
| 5 |
+
} from '../types'
|
| 6 |
|
| 7 |
function getApiBase(): string {
|
| 8 |
const url = import.meta.env.VITE_API_URL
|
|
|
|
| 40 |
this.baseUrl = baseUrl
|
| 41 |
}
|
| 42 |
|
| 43 |
+
/**
|
| 44 |
+
* Get list of available cases
|
| 45 |
+
*/
|
| 46 |
async getCases(signal?: AbortSignal): Promise<CasesResponse> {
|
| 47 |
const response = await fetch(`${this.baseUrl}/api/cases`, { signal })
|
| 48 |
|
|
|
|
| 58 |
return response.json()
|
| 59 |
}
|
| 60 |
|
| 61 |
+
/**
|
| 62 |
+
* Create a segmentation job (async - returns immediately with job ID)
|
| 63 |
+
*
|
| 64 |
+
* The actual ML inference runs in the background. Poll getJobStatus()
|
| 65 |
+
* to track progress and retrieve results when complete.
|
| 66 |
+
*/
|
| 67 |
+
async createSegmentJob(
|
| 68 |
caseId: string,
|
| 69 |
fastMode: boolean = true,
|
| 70 |
signal?: AbortSignal
|
| 71 |
+
): Promise<CreateJobResponse> {
|
| 72 |
const response = await fetch(`${this.baseUrl}/api/segment`, {
|
| 73 |
method: 'POST',
|
| 74 |
headers: {
|
|
|
|
| 84 |
if (!response.ok) {
|
| 85 |
const error = await response.json().catch(() => ({}))
|
| 86 |
throw new ApiError(
|
| 87 |
+
`Failed to create job: ${error.detail || response.statusText}`,
|
| 88 |
+
response.status,
|
| 89 |
+
error.detail
|
| 90 |
+
)
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
return response.json()
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
/**
|
| 97 |
+
* Get the status of a segmentation job
|
| 98 |
+
*
|
| 99 |
+
* Poll this endpoint to track progress and retrieve results.
|
| 100 |
+
* When status is 'completed', the result field contains segmentation data.
|
| 101 |
+
* When status is 'failed', the error field contains the error message.
|
| 102 |
+
*/
|
| 103 |
+
async getJobStatus(
|
| 104 |
+
jobId: string,
|
| 105 |
+
signal?: AbortSignal
|
| 106 |
+
): Promise<JobStatusResponse> {
|
| 107 |
+
const response = await fetch(`${this.baseUrl}/api/jobs/${jobId}`, {
|
| 108 |
+
signal,
|
| 109 |
+
})
|
| 110 |
+
|
| 111 |
+
if (response.status === 404) {
|
| 112 |
+
throw new ApiError(
|
| 113 |
+
'Job not found or expired',
|
| 114 |
+
404,
|
| 115 |
+
'Jobs expire after 1 hour'
|
| 116 |
+
)
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
if (!response.ok) {
|
| 120 |
+
const error = await response.json().catch(() => ({}))
|
| 121 |
+
throw new ApiError(
|
| 122 |
+
`Failed to get job status: ${error.detail || response.statusText}`,
|
| 123 |
response.status,
|
| 124 |
error.detail
|
| 125 |
)
|
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import type { JobStatus } from '../types'
|
| 2 |
+
|
| 3 |
+
interface ProgressIndicatorProps {
|
| 4 |
+
progress: number
|
| 5 |
+
message: string
|
| 6 |
+
status: JobStatus
|
| 7 |
+
elapsedSeconds?: number
|
| 8 |
+
}
|
| 9 |
+
|
| 10 |
+
/**
|
| 11 |
+
* Visual progress indicator for long-running ML inference jobs.
|
| 12 |
+
*
|
| 13 |
+
* Shows:
|
| 14 |
+
* - Progress bar with percentage
|
| 15 |
+
* - Current operation message
|
| 16 |
+
* - Elapsed time
|
| 17 |
+
* - Status-appropriate coloring (blue for running, red for failed)
|
| 18 |
+
*/
|
| 19 |
+
export function ProgressIndicator({
|
| 20 |
+
progress,
|
| 21 |
+
message,
|
| 22 |
+
status,
|
| 23 |
+
elapsedSeconds,
|
| 24 |
+
}: ProgressIndicatorProps) {
|
| 25 |
+
const isError = status === 'failed'
|
| 26 |
+
const isComplete = status === 'completed'
|
| 27 |
+
|
| 28 |
+
// Determine bar color based on status
|
| 29 |
+
const barColorClass = isError
|
| 30 |
+
? 'bg-red-500'
|
| 31 |
+
: isComplete
|
| 32 |
+
? 'bg-green-500'
|
| 33 |
+
: 'bg-blue-500'
|
| 34 |
+
|
| 35 |
+
// Animate the bar while running
|
| 36 |
+
const animationClass =
|
| 37 |
+
status === 'running' || status === 'pending' ? 'animate-pulse' : ''
|
| 38 |
+
|
| 39 |
+
return (
|
| 40 |
+
<div className="bg-gray-800 rounded-lg p-4 space-y-3">
|
| 41 |
+
{/* Header with message and percentage */}
|
| 42 |
+
<div className="flex justify-between items-center text-sm">
|
| 43 |
+
<span className="text-gray-300 font-medium">{message}</span>
|
| 44 |
+
<span className="text-gray-400 tabular-nums">{progress}%</span>
|
| 45 |
+
</div>
|
| 46 |
+
|
| 47 |
+
{/* Progress bar */}
|
| 48 |
+
<div className="w-full bg-gray-700 rounded-full h-2.5 overflow-hidden">
|
| 49 |
+
<div
|
| 50 |
+
className={`h-full rounded-full transition-all duration-500 ease-out ${barColorClass} ${animationClass}`}
|
| 51 |
+
style={{ width: `${progress}%` }}
|
| 52 |
+
role="progressbar"
|
| 53 |
+
aria-valuenow={progress}
|
| 54 |
+
aria-valuemin={0}
|
| 55 |
+
aria-valuemax={100}
|
| 56 |
+
aria-label={message}
|
| 57 |
+
/>
|
| 58 |
+
</div>
|
| 59 |
+
|
| 60 |
+
{/* Footer with elapsed time and status */}
|
| 61 |
+
<div className="flex justify-between items-center text-xs text-gray-500">
|
| 62 |
+
{elapsedSeconds !== undefined ? (
|
| 63 |
+
<span className="tabular-nums">
|
| 64 |
+
Elapsed: {elapsedSeconds.toFixed(1)}s
|
| 65 |
+
</span>
|
| 66 |
+
) : (
|
| 67 |
+
<span>Starting...</span>
|
| 68 |
+
)}
|
| 69 |
+
|
| 70 |
+
<span
|
| 71 |
+
className={`capitalize ${
|
| 72 |
+
isError
|
| 73 |
+
? 'text-red-400'
|
| 74 |
+
: isComplete
|
| 75 |
+
? 'text-green-400'
|
| 76 |
+
: 'text-blue-400'
|
| 77 |
+
}`}
|
| 78 |
+
>
|
| 79 |
+
{status}
|
| 80 |
+
</span>
|
| 81 |
+
</div>
|
| 82 |
+
</div>
|
| 83 |
+
)
|
| 84 |
+
}
|
|
@@ -2,3 +2,4 @@ export { Layout } from './Layout'
|
|
| 2 |
export { MetricsPanel } from './MetricsPanel'
|
| 3 |
export { CaseSelector } from './CaseSelector'
|
| 4 |
export { NiiVueViewer } from './NiiVueViewer'
|
|
|
|
|
|
| 2 |
export { MetricsPanel } from './MetricsPanel'
|
| 3 |
export { CaseSelector } from './CaseSelector'
|
| 4 |
export { NiiVueViewer } from './NiiVueViewer'
|
| 5 |
+
export { ProgressIndicator } from './ProgressIndicator'
|
|
@@ -1,19 +1,28 @@
|
|
| 1 |
-
import { describe, it, expect } from 'vitest'
|
| 2 |
import { renderHook, waitFor, act } from '@testing-library/react'
|
| 3 |
import { server } from '../../mocks/server'
|
| 4 |
import { errorHandlers } from '../../mocks/handlers'
|
| 5 |
import { useSegmentation } from '../useSegmentation'
|
| 6 |
|
| 7 |
describe('useSegmentation', () => {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
it('starts with null result and not loading', () => {
|
| 9 |
const { result } = renderHook(() => useSegmentation())
|
| 10 |
|
| 11 |
expect(result.current.result).toBeNull()
|
| 12 |
expect(result.current.isLoading).toBe(false)
|
| 13 |
expect(result.current.error).toBeNull()
|
|
|
|
| 14 |
})
|
| 15 |
|
| 16 |
-
it('sets loading state during segmentation', async () => {
|
| 17 |
const { result } = renderHook(() => useSegmentation())
|
| 18 |
|
| 19 |
act(() => {
|
|
@@ -22,75 +31,132 @@ describe('useSegmentation', () => {
|
|
| 22 |
|
| 23 |
expect(result.current.isLoading).toBe(true)
|
| 24 |
|
|
|
|
| 25 |
await waitFor(() => {
|
| 26 |
-
expect(result.current.
|
| 27 |
})
|
|
|
|
|
|
|
| 28 |
})
|
| 29 |
|
| 30 |
-
it('returns result on
|
| 31 |
const { result } = renderHook(() => useSegmentation())
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
await act(async () => {
|
| 34 |
-
await
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
})
|
| 36 |
|
| 37 |
-
expect(result.current.result).not.toBeNull()
|
| 38 |
expect(result.current.result?.metrics.caseId).toBe('sub-stroke0001')
|
| 39 |
expect(result.current.result?.metrics.diceScore).toBe(0.847)
|
| 40 |
expect(result.current.result?.dwiUrl).toContain('dwi.nii.gz')
|
| 41 |
})
|
| 42 |
|
| 43 |
-
it('
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
const { result } = renderHook(() => useSegmentation())
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
| 50 |
})
|
| 51 |
|
| 52 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
expect(result.current.result).toBeNull()
|
| 54 |
})
|
| 55 |
|
| 56 |
it('clears previous error on new request', async () => {
|
| 57 |
-
server.use(errorHandlers.
|
| 58 |
const { result } = renderHook(() => useSegmentation())
|
| 59 |
|
| 60 |
// First request fails
|
| 61 |
-
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
})
|
| 64 |
-
expect(result.current.error).not.toBeNull()
|
| 65 |
|
| 66 |
// Reset to success handler
|
| 67 |
server.resetHandlers()
|
| 68 |
|
| 69 |
-
// Second request
|
| 70 |
-
|
| 71 |
-
|
| 72 |
})
|
| 73 |
|
| 74 |
expect(result.current.error).toBeNull()
|
| 75 |
-
expect(result.current.
|
| 76 |
})
|
| 77 |
|
| 78 |
-
it('
|
| 79 |
const { result } = renderHook(() => useSegmentation())
|
| 80 |
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
await result.current.runSegmentation('sub-stroke0001')
|
| 84 |
})
|
| 85 |
-
expect(result.current.result).not.toBeNull()
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
act(() => {
|
| 89 |
-
result.current.
|
| 90 |
})
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
})
|
| 96 |
})
|
|
|
|
| 1 |
+
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
| 2 |
import { renderHook, waitFor, act } from '@testing-library/react'
|
| 3 |
import { server } from '../../mocks/server'
|
| 4 |
import { errorHandlers } from '../../mocks/handlers'
|
| 5 |
import { useSegmentation } from '../useSegmentation'
|
| 6 |
|
| 7 |
describe('useSegmentation', () => {
|
| 8 |
+
beforeEach(() => {
|
| 9 |
+
vi.useFakeTimers({ shouldAdvanceTime: true })
|
| 10 |
+
})
|
| 11 |
+
|
| 12 |
+
afterEach(() => {
|
| 13 |
+
vi.useRealTimers()
|
| 14 |
+
})
|
| 15 |
+
|
| 16 |
it('starts with null result and not loading', () => {
|
| 17 |
const { result } = renderHook(() => useSegmentation())
|
| 18 |
|
| 19 |
expect(result.current.result).toBeNull()
|
| 20 |
expect(result.current.isLoading).toBe(false)
|
| 21 |
expect(result.current.error).toBeNull()
|
| 22 |
+
expect(result.current.jobStatus).toBeNull()
|
| 23 |
})
|
| 24 |
|
| 25 |
+
it('sets loading state and job status during segmentation', async () => {
|
| 26 |
const { result } = renderHook(() => useSegmentation())
|
| 27 |
|
| 28 |
act(() => {
|
|
|
|
| 31 |
|
| 32 |
expect(result.current.isLoading).toBe(true)
|
| 33 |
|
| 34 |
+
// Wait for job to be created
|
| 35 |
await waitFor(() => {
|
| 36 |
+
expect(result.current.jobId).toBeDefined()
|
| 37 |
})
|
| 38 |
+
|
| 39 |
+
expect(result.current.jobStatus).toBeDefined()
|
| 40 |
})
|
| 41 |
|
| 42 |
+
it('returns result on job completion', async () => {
|
| 43 |
const { result } = renderHook(() => useSegmentation())
|
| 44 |
|
| 45 |
+
act(() => {
|
| 46 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 47 |
+
})
|
| 48 |
+
|
| 49 |
+
// Wait for job creation
|
| 50 |
+
await waitFor(() => {
|
| 51 |
+
expect(result.current.jobId).toBeDefined()
|
| 52 |
+
})
|
| 53 |
+
|
| 54 |
+
// Advance time to allow job to complete (mock jobs complete in ~3s)
|
| 55 |
await act(async () => {
|
| 56 |
+
await vi.advanceTimersByTimeAsync(5000)
|
| 57 |
+
})
|
| 58 |
+
|
| 59 |
+
await waitFor(() => {
|
| 60 |
+
expect(result.current.isLoading).toBe(false)
|
| 61 |
+
expect(result.current.result).not.toBeNull()
|
| 62 |
})
|
| 63 |
|
|
|
|
| 64 |
expect(result.current.result?.metrics.caseId).toBe('sub-stroke0001')
|
| 65 |
expect(result.current.result?.metrics.diceScore).toBe(0.847)
|
| 66 |
expect(result.current.result?.dwiUrl).toContain('dwi.nii.gz')
|
| 67 |
})
|
| 68 |
|
| 69 |
+
it('shows progress updates during job execution', async () => {
|
| 70 |
+
const { result } = renderHook(() => useSegmentation())
|
| 71 |
+
|
| 72 |
+
act(() => {
|
| 73 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 74 |
+
})
|
| 75 |
+
|
| 76 |
+
// Wait for job to start
|
| 77 |
+
await waitFor(() => {
|
| 78 |
+
expect(result.current.jobId).toBeDefined()
|
| 79 |
+
})
|
| 80 |
+
|
| 81 |
+
// Progress should be tracked
|
| 82 |
+
expect(result.current.progress).toBeGreaterThanOrEqual(0)
|
| 83 |
+
expect(result.current.progressMessage).toBeDefined()
|
| 84 |
+
})
|
| 85 |
+
|
| 86 |
+
it('sets error on job creation failure', async () => {
|
| 87 |
+
server.use(errorHandlers.segmentCreateError)
|
| 88 |
|
| 89 |
const { result } = renderHook(() => useSegmentation())
|
| 90 |
|
| 91 |
+
act(() => {
|
| 92 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 93 |
})
|
| 94 |
|
| 95 |
+
await waitFor(() => {
|
| 96 |
+
expect(result.current.isLoading).toBe(false)
|
| 97 |
+
})
|
| 98 |
+
|
| 99 |
+
expect(result.current.error).toMatch(/failed to create job/i)
|
| 100 |
expect(result.current.result).toBeNull()
|
| 101 |
})
|
| 102 |
|
| 103 |
it('clears previous error on new request', async () => {
|
| 104 |
+
server.use(errorHandlers.segmentCreateError)
|
| 105 |
const { result } = renderHook(() => useSegmentation())
|
| 106 |
|
| 107 |
// First request fails
|
| 108 |
+
act(() => {
|
| 109 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 110 |
+
})
|
| 111 |
+
|
| 112 |
+
await waitFor(() => {
|
| 113 |
+
expect(result.current.error).not.toBeNull()
|
| 114 |
})
|
|
|
|
| 115 |
|
| 116 |
// Reset to success handler
|
| 117 |
server.resetHandlers()
|
| 118 |
|
| 119 |
+
// Second request should clear error
|
| 120 |
+
act(() => {
|
| 121 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 122 |
})
|
| 123 |
|
| 124 |
expect(result.current.error).toBeNull()
|
| 125 |
+
expect(result.current.isLoading).toBe(true)
|
| 126 |
})
|
| 127 |
|
| 128 |
+
it('can cancel a running job', async () => {
|
| 129 |
const { result } = renderHook(() => useSegmentation())
|
| 130 |
|
| 131 |
+
act(() => {
|
| 132 |
+
result.current.runSegmentation('sub-stroke0001')
|
|
|
|
| 133 |
})
|
|
|
|
| 134 |
|
| 135 |
+
await waitFor(() => {
|
| 136 |
+
expect(result.current.isLoading).toBe(true)
|
| 137 |
+
})
|
| 138 |
+
|
| 139 |
+
// Cancel the job
|
| 140 |
act(() => {
|
| 141 |
+
result.current.cancelJob()
|
| 142 |
})
|
| 143 |
|
| 144 |
+
expect(result.current.isLoading).toBe(false)
|
| 145 |
+
expect(result.current.jobStatus).toBeNull()
|
| 146 |
+
})
|
| 147 |
+
|
| 148 |
+
it('cleans up polling on unmount', async () => {
|
| 149 |
+
const { result, unmount } = renderHook(() => useSegmentation())
|
| 150 |
+
|
| 151 |
+
act(() => {
|
| 152 |
+
result.current.runSegmentation('sub-stroke0001')
|
| 153 |
+
})
|
| 154 |
+
|
| 155 |
+
await waitFor(() => {
|
| 156 |
+
expect(result.current.isLoading).toBe(true)
|
| 157 |
+
})
|
| 158 |
+
|
| 159 |
+
// Unmount should not throw
|
| 160 |
+
unmount()
|
| 161 |
})
|
| 162 |
})
|
|
@@ -1,63 +1,205 @@
|
|
| 1 |
-
import { useState, useCallback, useRef } from 'react'
|
| 2 |
import { apiClient } from '../api/client'
|
| 3 |
-
import type { SegmentationResult } from '../types'
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
export function useSegmentation() {
|
|
|
|
| 6 |
const [result, setResult] = useState<SegmentationResult | null>(null)
|
| 7 |
-
const [isLoading, setIsLoading] = useState(false)
|
| 8 |
const [error, setError] = useState<string | null>(null)
|
| 9 |
|
| 10 |
-
//
|
| 11 |
-
|
| 12 |
-
const
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
const abortControllerRef = useRef<AbortController | null>(null)
|
| 14 |
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
//
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
setResult(null)
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
setIsLoading(false)
|
|
|
|
| 58 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
}
|
| 60 |
-
}, [])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
| 63 |
}
|
|
|
|
| 1 |
+
import { useState, useCallback, useRef, useEffect } from 'react'
|
| 2 |
import { apiClient } from '../api/client'
|
| 3 |
+
import type { SegmentationResult, JobStatus } from '../types'
|
| 4 |
|
| 5 |
+
// Polling interval in milliseconds
|
| 6 |
+
const POLLING_INTERVAL = 2000
|
| 7 |
+
|
| 8 |
+
/**
|
| 9 |
+
* Hook for running segmentation with async job polling.
|
| 10 |
+
*
|
| 11 |
+
* Instead of waiting for the full inference to complete (which can timeout
|
| 12 |
+
* on HuggingFace Spaces), this hook:
|
| 13 |
+
* 1. Creates a job that returns immediately with a job ID
|
| 14 |
+
* 2. Polls for job status/progress every 2 seconds
|
| 15 |
+
* 3. Returns results when the job completes
|
| 16 |
+
*
|
| 17 |
+
* This avoids the ~60s gateway timeout on HF Spaces while providing
|
| 18 |
+
* real-time progress updates to the user.
|
| 19 |
+
*/
|
| 20 |
export function useSegmentation() {
|
| 21 |
+
// Result state
|
| 22 |
const [result, setResult] = useState<SegmentationResult | null>(null)
|
|
|
|
| 23 |
const [error, setError] = useState<string | null>(null)
|
| 24 |
|
| 25 |
+
// Job tracking state
|
| 26 |
+
const [jobId, setJobId] = useState<string | null>(null)
|
| 27 |
+
const [jobStatus, setJobStatus] = useState<JobStatus | null>(null)
|
| 28 |
+
const [progress, setProgress] = useState(0)
|
| 29 |
+
const [progressMessage, setProgressMessage] = useState('')
|
| 30 |
+
const [elapsedSeconds, setElapsedSeconds] = useState<number | undefined>(
|
| 31 |
+
undefined
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
// Loading state - true from job creation until completion/failure
|
| 35 |
+
const [isLoading, setIsLoading] = useState(false)
|
| 36 |
+
|
| 37 |
+
// Refs for managing async operations
|
| 38 |
+
const currentJobRef = useRef<string | null>(null)
|
| 39 |
+
const pollingIntervalRef = useRef<number | null>(null)
|
| 40 |
const abortControllerRef = useRef<AbortController | null>(null)
|
| 41 |
|
| 42 |
+
/**
|
| 43 |
+
* Stop polling for job status
|
| 44 |
+
*/
|
| 45 |
+
const stopPolling = useCallback(() => {
|
| 46 |
+
if (pollingIntervalRef.current) {
|
| 47 |
+
clearInterval(pollingIntervalRef.current)
|
| 48 |
+
pollingIntervalRef.current = null
|
| 49 |
+
}
|
| 50 |
+
}, [])
|
| 51 |
+
|
| 52 |
+
/**
|
| 53 |
+
* Poll for job status and update state
|
| 54 |
+
*/
|
| 55 |
+
const pollJobStatus = useCallback(
|
| 56 |
+
async (id: string, signal: AbortSignal) => {
|
| 57 |
+
// Don't poll if this isn't the current job
|
| 58 |
+
if (id !== currentJobRef.current) {
|
| 59 |
+
stopPolling()
|
| 60 |
+
return
|
| 61 |
+
}
|
| 62 |
+
|
| 63 |
+
try {
|
| 64 |
+
const response = await apiClient.getJobStatus(id, signal)
|
| 65 |
+
|
| 66 |
+
// Ignore results if job changed
|
| 67 |
+
if (id !== currentJobRef.current) return
|
| 68 |
+
|
| 69 |
+
// Update progress state
|
| 70 |
+
setJobStatus(response.status)
|
| 71 |
+
setProgress(response.progress)
|
| 72 |
+
setProgressMessage(response.progressMessage)
|
| 73 |
+
setElapsedSeconds(response.elapsedSeconds)
|
| 74 |
+
|
| 75 |
+
// Handle completion
|
| 76 |
+
if (response.status === 'completed' && response.result) {
|
| 77 |
+
stopPolling()
|
| 78 |
+
setIsLoading(false)
|
| 79 |
+
setResult({
|
| 80 |
+
dwiUrl: response.result.dwiUrl,
|
| 81 |
+
predictionUrl: response.result.predictionUrl,
|
| 82 |
+
metrics: {
|
| 83 |
+
caseId: response.result.caseId,
|
| 84 |
+
diceScore: response.result.diceScore,
|
| 85 |
+
volumeMl: response.result.volumeMl,
|
| 86 |
+
elapsedSeconds: response.result.elapsedSeconds,
|
| 87 |
+
},
|
| 88 |
+
})
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
// Handle failure
|
| 92 |
+
if (response.status === 'failed') {
|
| 93 |
+
stopPolling()
|
| 94 |
+
setIsLoading(false)
|
| 95 |
+
setError(response.error || 'Job failed')
|
| 96 |
+
setResult(null)
|
| 97 |
+
}
|
| 98 |
+
} catch (err) {
|
| 99 |
+
// Ignore abort errors
|
| 100 |
+
if (err instanceof Error && err.name === 'AbortError') return
|
| 101 |
+
|
| 102 |
+
// Don't stop polling on transient network errors - retry next interval
|
| 103 |
+
console.warn('Polling error (will retry):', err)
|
| 104 |
+
}
|
| 105 |
+
},
|
| 106 |
+
[stopPolling]
|
| 107 |
+
)
|
| 108 |
+
|
| 109 |
+
/**
|
| 110 |
+
* Start segmentation job and begin polling
|
| 111 |
+
*/
|
| 112 |
+
const runSegmentation = useCallback(
|
| 113 |
+
async (caseId: string, fastMode = true) => {
|
| 114 |
+
// Cancel any existing job/polling
|
| 115 |
+
stopPolling()
|
| 116 |
+
abortControllerRef.current?.abort()
|
| 117 |
+
|
| 118 |
+
const abortController = new AbortController()
|
| 119 |
+
abortControllerRef.current = abortController
|
| 120 |
+
|
| 121 |
+
// Reset state
|
| 122 |
+
setError(null)
|
| 123 |
setResult(null)
|
| 124 |
+
setProgress(0)
|
| 125 |
+
setProgressMessage('Creating job...')
|
| 126 |
+
setJobStatus('pending')
|
| 127 |
+
setElapsedSeconds(undefined)
|
| 128 |
+
setIsLoading(true)
|
| 129 |
+
|
| 130 |
+
try {
|
| 131 |
+
// Create the job
|
| 132 |
+
const response = await apiClient.createSegmentJob(
|
| 133 |
+
caseId,
|
| 134 |
+
fastMode,
|
| 135 |
+
abortController.signal
|
| 136 |
+
)
|
| 137 |
+
|
| 138 |
+
// Store job reference
|
| 139 |
+
const newJobId = response.jobId
|
| 140 |
+
setJobId(newJobId)
|
| 141 |
+
currentJobRef.current = newJobId
|
| 142 |
+
setJobStatus(response.status)
|
| 143 |
+
setProgressMessage(response.message)
|
| 144 |
+
|
| 145 |
+
// Start polling
|
| 146 |
+
pollingIntervalRef.current = window.setInterval(() => {
|
| 147 |
+
pollJobStatus(newJobId, abortController.signal)
|
| 148 |
+
}, POLLING_INTERVAL)
|
| 149 |
+
|
| 150 |
+
// Do an initial poll immediately
|
| 151 |
+
await pollJobStatus(newJobId, abortController.signal)
|
| 152 |
+
} catch (err) {
|
| 153 |
+
// Ignore abort errors
|
| 154 |
+
if (err instanceof Error && err.name === 'AbortError') return
|
| 155 |
+
|
| 156 |
+
const message = err instanceof Error ? err.message : 'Failed to start job'
|
| 157 |
+
setError(message)
|
| 158 |
setIsLoading(false)
|
| 159 |
+
setJobStatus('failed')
|
| 160 |
}
|
| 161 |
+
},
|
| 162 |
+
[pollJobStatus, stopPolling]
|
| 163 |
+
)
|
| 164 |
+
|
| 165 |
+
/**
|
| 166 |
+
* Cancel the current job (stops polling, clears loading state)
|
| 167 |
+
*/
|
| 168 |
+
const cancelJob = useCallback(() => {
|
| 169 |
+
stopPolling()
|
| 170 |
+
abortControllerRef.current?.abort()
|
| 171 |
+
currentJobRef.current = null
|
| 172 |
+
setIsLoading(false)
|
| 173 |
+
setJobStatus(null)
|
| 174 |
+
setProgress(0)
|
| 175 |
+
setProgressMessage('')
|
| 176 |
+
}, [stopPolling])
|
| 177 |
+
|
| 178 |
+
// Cleanup on unmount
|
| 179 |
+
useEffect(() => {
|
| 180 |
+
return () => {
|
| 181 |
+
stopPolling()
|
| 182 |
+
abortControllerRef.current?.abort()
|
| 183 |
}
|
| 184 |
+
}, [stopPolling])
|
| 185 |
+
|
| 186 |
+
return {
|
| 187 |
+
// Result data
|
| 188 |
+
result,
|
| 189 |
+
error,
|
| 190 |
+
|
| 191 |
+
// Job status
|
| 192 |
+
jobId,
|
| 193 |
+
jobStatus,
|
| 194 |
+
progress,
|
| 195 |
+
progressMessage,
|
| 196 |
+
elapsedSeconds,
|
| 197 |
+
|
| 198 |
+
// Loading state
|
| 199 |
+
isLoading,
|
| 200 |
|
| 201 |
+
// Actions
|
| 202 |
+
runSegmentation,
|
| 203 |
+
cancelJob,
|
| 204 |
+
}
|
| 205 |
}
|
|
@@ -1,8 +1,66 @@
|
|
| 1 |
import { http, HttpResponse, delay } from 'msw'
|
|
|
|
| 2 |
|
| 3 |
const API_BASE = import.meta.env.VITE_API_URL || 'http://localhost:7860'
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
export const handlers = [
|
|
|
|
| 6 |
http.get(`${API_BASE}/api/cases`, async () => {
|
| 7 |
await delay(100)
|
| 8 |
return HttpResponse.json({
|
|
@@ -10,18 +68,75 @@ export const handlers = [
|
|
| 10 |
})
|
| 11 |
}),
|
| 12 |
|
|
|
|
| 13 |
http.post(`${API_BASE}/api/segment`, async ({ request }) => {
|
| 14 |
const body = (await request.json()) as { case_id: string; fast_mode?: boolean }
|
| 15 |
-
await delay(
|
| 16 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
caseId: body.case_id,
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
elapsedSeconds:
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
}),
|
| 26 |
]
|
| 27 |
|
|
@@ -38,15 +153,54 @@ export const errorHandlers = {
|
|
| 38 |
return HttpResponse.error()
|
| 39 |
}),
|
| 40 |
|
| 41 |
-
|
| 42 |
return HttpResponse.json(
|
| 43 |
-
{ detail: '
|
| 44 |
-
{ status:
|
| 45 |
)
|
| 46 |
}),
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
| 51 |
}),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
}
|
|
|
|
| 1 |
import { http, HttpResponse, delay } from 'msw'
|
| 2 |
+
import type { JobStatus } from '../types'
|
| 3 |
|
| 4 |
const API_BASE = import.meta.env.VITE_API_URL || 'http://localhost:7860'
|
| 5 |
|
| 6 |
+
// In-memory job store for mocking
|
| 7 |
+
interface MockJob {
|
| 8 |
+
id: string
|
| 9 |
+
caseId: string
|
| 10 |
+
status: JobStatus
|
| 11 |
+
progress: number
|
| 12 |
+
progressMessage: string
|
| 13 |
+
elapsedSeconds: number
|
| 14 |
+
fastMode: boolean
|
| 15 |
+
createdAt: number
|
| 16 |
+
}
|
| 17 |
+
|
| 18 |
+
const mockJobs = new Map<string, MockJob>()
|
| 19 |
+
let jobCounter = 0
|
| 20 |
+
|
| 21 |
+
// Configurable job duration for tests (ms)
|
| 22 |
+
// Default: 500ms for fast tests
|
| 23 |
+
let mockJobDurationMs = 500
|
| 24 |
+
|
| 25 |
+
/**
|
| 26 |
+
* Set the mock job duration for tests.
|
| 27 |
+
* Jobs will complete after this many milliseconds.
|
| 28 |
+
*/
|
| 29 |
+
export function setMockJobDuration(durationMs: number): void {
|
| 30 |
+
mockJobDurationMs = durationMs
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
// Simulate job progression over time
|
| 34 |
+
function getJobProgress(job: MockJob): MockJob {
|
| 35 |
+
const elapsed = (Date.now() - job.createdAt) / 1000
|
| 36 |
+
const duration = mockJobDurationMs / 1000 // Convert to seconds
|
| 37 |
+
|
| 38 |
+
if (job.status === 'completed' || job.status === 'failed') {
|
| 39 |
+
return job
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
// Progress through stages based on elapsed time relative to configured duration
|
| 43 |
+
// Stages: 20% loading, 40% inference, 30% processing, 10% finalizing
|
| 44 |
+
const progress20 = duration * 0.2
|
| 45 |
+
const progress60 = duration * 0.6
|
| 46 |
+
const progress90 = duration * 0.9
|
| 47 |
+
|
| 48 |
+
if (elapsed < progress20) {
|
| 49 |
+
return { ...job, status: 'running', progress: 10, progressMessage: 'Loading case data...', elapsedSeconds: elapsed }
|
| 50 |
+
} else if (elapsed < progress60) {
|
| 51 |
+
return { ...job, status: 'running', progress: 30, progressMessage: 'Running DeepISLES inference...', elapsedSeconds: elapsed }
|
| 52 |
+
} else if (elapsed < progress90) {
|
| 53 |
+
return { ...job, status: 'running', progress: 70, progressMessage: 'Processing results...', elapsedSeconds: elapsed }
|
| 54 |
+
} else if (elapsed < duration) {
|
| 55 |
+
return { ...job, status: 'running', progress: 90, progressMessage: 'Computing metrics...', elapsedSeconds: elapsed }
|
| 56 |
+
} else {
|
| 57 |
+
// Job complete
|
| 58 |
+
return { ...job, status: 'completed', progress: 100, progressMessage: 'Segmentation complete', elapsedSeconds: elapsed }
|
| 59 |
+
}
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
export const handlers = [
|
| 63 |
+
// GET /api/cases - List available cases
|
| 64 |
http.get(`${API_BASE}/api/cases`, async () => {
|
| 65 |
await delay(100)
|
| 66 |
return HttpResponse.json({
|
|
|
|
| 68 |
})
|
| 69 |
}),
|
| 70 |
|
| 71 |
+
// POST /api/segment - Create segmentation job (returns immediately)
|
| 72 |
http.post(`${API_BASE}/api/segment`, async ({ request }) => {
|
| 73 |
const body = (await request.json()) as { case_id: string; fast_mode?: boolean }
|
| 74 |
+
await delay(50) // Small delay to simulate network
|
| 75 |
+
|
| 76 |
+
// Create a new job
|
| 77 |
+
const jobId = `mock-${++jobCounter}`
|
| 78 |
+
const job: MockJob = {
|
| 79 |
+
id: jobId,
|
| 80 |
caseId: body.case_id,
|
| 81 |
+
status: 'pending',
|
| 82 |
+
progress: 0,
|
| 83 |
+
progressMessage: 'Job queued',
|
| 84 |
+
elapsedSeconds: 0,
|
| 85 |
+
fastMode: body.fast_mode !== false,
|
| 86 |
+
createdAt: Date.now(),
|
| 87 |
+
}
|
| 88 |
+
mockJobs.set(jobId, job)
|
| 89 |
+
|
| 90 |
+
// Return 202 Accepted with job ID
|
| 91 |
+
return HttpResponse.json(
|
| 92 |
+
{
|
| 93 |
+
jobId: jobId,
|
| 94 |
+
status: 'pending',
|
| 95 |
+
message: `Segmentation job queued for ${body.case_id}`,
|
| 96 |
+
},
|
| 97 |
+
{ status: 202 }
|
| 98 |
+
)
|
| 99 |
+
}),
|
| 100 |
+
|
| 101 |
+
// GET /api/jobs/:jobId - Get job status
|
| 102 |
+
http.get(`${API_BASE}/api/jobs/:jobId`, async ({ params }) => {
|
| 103 |
+
const jobId = params.jobId as string
|
| 104 |
+
await delay(50) // Small delay to simulate network
|
| 105 |
+
|
| 106 |
+
const job = mockJobs.get(jobId)
|
| 107 |
+
if (!job) {
|
| 108 |
+
return HttpResponse.json(
|
| 109 |
+
{ detail: `Job not found: ${jobId}. Jobs expire after 1 hour.` },
|
| 110 |
+
{ status: 404 }
|
| 111 |
+
)
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
// Update job progress based on elapsed time
|
| 115 |
+
const updatedJob = getJobProgress(job)
|
| 116 |
+
mockJobs.set(jobId, updatedJob)
|
| 117 |
+
|
| 118 |
+
// Build response
|
| 119 |
+
const response: Record<string, unknown> = {
|
| 120 |
+
jobId: updatedJob.id,
|
| 121 |
+
status: updatedJob.status,
|
| 122 |
+
progress: updatedJob.progress,
|
| 123 |
+
progressMessage: updatedJob.progressMessage,
|
| 124 |
+
elapsedSeconds: Math.round(updatedJob.elapsedSeconds * 100) / 100,
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
// Include result if completed
|
| 128 |
+
if (updatedJob.status === 'completed') {
|
| 129 |
+
response.result = {
|
| 130 |
+
caseId: updatedJob.caseId,
|
| 131 |
+
diceScore: 0.847,
|
| 132 |
+
volumeMl: 15.32,
|
| 133 |
+
elapsedSeconds: updatedJob.fastMode ? 12.5 : 45.0,
|
| 134 |
+
dwiUrl: `${API_BASE}/files/${jobId}/${updatedJob.caseId}/dwi.nii.gz`,
|
| 135 |
+
predictionUrl: `${API_BASE}/files/${jobId}/${updatedJob.caseId}/prediction.nii.gz`,
|
| 136 |
+
}
|
| 137 |
+
}
|
| 138 |
+
|
| 139 |
+
return HttpResponse.json(response)
|
| 140 |
}),
|
| 141 |
]
|
| 142 |
|
|
|
|
| 153 |
return HttpResponse.error()
|
| 154 |
}),
|
| 155 |
|
| 156 |
+
segmentCreateError: http.post(`${API_BASE}/api/segment`, () => {
|
| 157 |
return HttpResponse.json(
|
| 158 |
+
{ detail: 'Failed to create job: case not found' },
|
| 159 |
+
{ status: 400 }
|
| 160 |
)
|
| 161 |
}),
|
| 162 |
|
| 163 |
+
jobNotFound: http.get(`${API_BASE}/api/jobs/:jobId`, () => {
|
| 164 |
+
return HttpResponse.json(
|
| 165 |
+
{ detail: 'Job not found or expired' },
|
| 166 |
+
{ status: 404 }
|
| 167 |
+
)
|
| 168 |
}),
|
| 169 |
+
|
| 170 |
+
// Simulate a job that fails during processing
|
| 171 |
+
jobFailed: [
|
| 172 |
+
http.post(`${API_BASE}/api/segment`, async ({ request }) => {
|
| 173 |
+
const body = (await request.json()) as { case_id: string }
|
| 174 |
+
const jobId = `fail-${++jobCounter}`
|
| 175 |
+
mockJobs.set(jobId, {
|
| 176 |
+
id: jobId,
|
| 177 |
+
caseId: body.case_id,
|
| 178 |
+
status: 'failed',
|
| 179 |
+
progress: 30,
|
| 180 |
+
progressMessage: 'Error occurred',
|
| 181 |
+
elapsedSeconds: 5.2,
|
| 182 |
+
fastMode: true,
|
| 183 |
+
createdAt: Date.now(),
|
| 184 |
+
})
|
| 185 |
+
return HttpResponse.json(
|
| 186 |
+
{ jobId, status: 'pending', message: 'Job queued' },
|
| 187 |
+
{ status: 202 }
|
| 188 |
+
)
|
| 189 |
+
}),
|
| 190 |
+
http.get(`${API_BASE}/api/jobs/:jobId`, ({ params }) => {
|
| 191 |
+
const jobId = params.jobId as string
|
| 192 |
+
const job = mockJobs.get(jobId)
|
| 193 |
+
if (!job) {
|
| 194 |
+
return HttpResponse.json({ detail: 'Not found' }, { status: 404 })
|
| 195 |
+
}
|
| 196 |
+
return HttpResponse.json({
|
| 197 |
+
jobId: job.id,
|
| 198 |
+
status: 'failed',
|
| 199 |
+
progress: 30,
|
| 200 |
+
progressMessage: 'Error occurred',
|
| 201 |
+
elapsedSeconds: 5.2,
|
| 202 |
+
error: 'Segmentation failed: out of memory',
|
| 203 |
+
})
|
| 204 |
+
}),
|
| 205 |
+
],
|
| 206 |
}
|
|
@@ -1,3 +1,4 @@
|
|
|
|
|
| 1 |
export interface Metrics {
|
| 2 |
caseId: string
|
| 3 |
diceScore: number | null
|
|
@@ -5,16 +6,19 @@ export interface Metrics {
|
|
| 5 |
elapsedSeconds: number
|
| 6 |
}
|
| 7 |
|
|
|
|
| 8 |
export interface SegmentationResult {
|
| 9 |
dwiUrl: string
|
| 10 |
predictionUrl: string
|
| 11 |
metrics: Metrics
|
| 12 |
}
|
| 13 |
|
|
|
|
| 14 |
export interface CasesResponse {
|
| 15 |
cases: string[]
|
| 16 |
}
|
| 17 |
|
|
|
|
| 18 |
export interface SegmentResponse {
|
| 19 |
caseId: string
|
| 20 |
diceScore: number | null
|
|
@@ -23,3 +27,24 @@ export interface SegmentResponse {
|
|
| 23 |
dwiUrl: string
|
| 24 |
predictionUrl: string
|
| 25 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Segmentation metrics
|
| 2 |
export interface Metrics {
|
| 3 |
caseId: string
|
| 4 |
diceScore: number | null
|
|
|
|
| 6 |
elapsedSeconds: number
|
| 7 |
}
|
| 8 |
|
| 9 |
+
// Final segmentation result with URLs and metrics
|
| 10 |
export interface SegmentationResult {
|
| 11 |
dwiUrl: string
|
| 12 |
predictionUrl: string
|
| 13 |
metrics: Metrics
|
| 14 |
}
|
| 15 |
|
| 16 |
+
// API Response Types
|
| 17 |
export interface CasesResponse {
|
| 18 |
cases: string[]
|
| 19 |
}
|
| 20 |
|
| 21 |
+
// Segmentation result data (embedded in job response)
|
| 22 |
export interface SegmentResponse {
|
| 23 |
caseId: string
|
| 24 |
diceScore: number | null
|
|
|
|
| 27 |
dwiUrl: string
|
| 28 |
predictionUrl: string
|
| 29 |
}
|
| 30 |
+
|
| 31 |
+
// Job Status Types
|
| 32 |
+
export type JobStatus = 'pending' | 'running' | 'completed' | 'failed'
|
| 33 |
+
|
| 34 |
+
// Response from POST /api/segment (job creation)
|
| 35 |
+
export interface CreateJobResponse {
|
| 36 |
+
jobId: string
|
| 37 |
+
status: JobStatus
|
| 38 |
+
message: string
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
// Response from GET /api/jobs/{jobId} (status polling)
|
| 42 |
+
export interface JobStatusResponse {
|
| 43 |
+
jobId: string
|
| 44 |
+
status: JobStatus
|
| 45 |
+
progress: number
|
| 46 |
+
progressMessage: string
|
| 47 |
+
elapsedSeconds?: number
|
| 48 |
+
result?: SegmentResponse
|
| 49 |
+
error?: string
|
| 50 |
+
}
|
|
@@ -0,0 +1,380 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""In-memory job store for async ML inference tasks.
|
| 2 |
+
|
| 3 |
+
This module provides a thread-safe job store for tracking long-running ML inference
|
| 4 |
+
jobs. Jobs are stored in-memory, which is appropriate for HuggingFace Spaces since:
|
| 5 |
+
1. HF Spaces runs a single uvicorn worker (no multi-worker sync needed)
|
| 6 |
+
2. Jobs are ephemeral (results cached, cleanup after TTL)
|
| 7 |
+
3. No external dependencies (Redis, DB) required
|
| 8 |
+
|
| 9 |
+
Note: Multi-worker deployments would require a shared store (Redis/DB).
|
| 10 |
+
|
| 11 |
+
Architecture:
|
| 12 |
+
- Jobs are created with PENDING status
|
| 13 |
+
- Background tasks update status to RUNNING, then COMPLETED/FAILED
|
| 14 |
+
- Frontend polls GET /api/jobs/{id} for status updates
|
| 15 |
+
- Cleanup thread removes old jobs to prevent memory leaks
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import re
|
| 21 |
+
import shutil
|
| 22 |
+
import threading
|
| 23 |
+
from dataclasses import dataclass
|
| 24 |
+
from datetime import datetime, timedelta
|
| 25 |
+
from enum import Enum
|
| 26 |
+
from pathlib import Path
|
| 27 |
+
from typing import Any
|
| 28 |
+
|
| 29 |
+
from stroke_deepisles_demo.core.logging import get_logger
|
| 30 |
+
|
| 31 |
+
logger = get_logger(__name__)
|
| 32 |
+
|
| 33 |
+
# Regex for safe job IDs (alphanumeric, hyphens, underscores only)
|
| 34 |
+
_SAFE_JOB_ID_PATTERN = re.compile(r"^[a-zA-Z0-9_-]+$")
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
class JobStatus(str, Enum):
|
| 38 |
+
"""Status of an async job."""
|
| 39 |
+
|
| 40 |
+
PENDING = "pending" # Job created, not yet started
|
| 41 |
+
RUNNING = "running" # Inference in progress
|
| 42 |
+
COMPLETED = "completed" # Success, results available
|
| 43 |
+
FAILED = "failed" # Error occurred
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@dataclass
|
| 47 |
+
class Job:
|
| 48 |
+
"""Represents an async segmentation job.
|
| 49 |
+
|
| 50 |
+
Attributes:
|
| 51 |
+
id: Unique job identifier (full UUID hex)
|
| 52 |
+
status: Current job status
|
| 53 |
+
case_id: The case being processed
|
| 54 |
+
fast_mode: Whether to use fast inference mode
|
| 55 |
+
created_at: When the job was created
|
| 56 |
+
started_at: When processing began (None if pending)
|
| 57 |
+
completed_at: When processing finished (None if not done)
|
| 58 |
+
progress: Progress percentage (0-100)
|
| 59 |
+
progress_message: Human-readable progress status
|
| 60 |
+
result: Segmentation results (None until completed)
|
| 61 |
+
error: Error message (None unless failed)
|
| 62 |
+
"""
|
| 63 |
+
|
| 64 |
+
id: str
|
| 65 |
+
status: JobStatus
|
| 66 |
+
case_id: str
|
| 67 |
+
fast_mode: bool
|
| 68 |
+
created_at: datetime
|
| 69 |
+
started_at: datetime | None = None
|
| 70 |
+
completed_at: datetime | None = None
|
| 71 |
+
progress: int = 0
|
| 72 |
+
progress_message: str = "Queued"
|
| 73 |
+
result: dict[str, Any] | None = None
|
| 74 |
+
error: str | None = None
|
| 75 |
+
|
| 76 |
+
@property
|
| 77 |
+
def elapsed_seconds(self) -> float:
|
| 78 |
+
"""Calculate elapsed time since job started."""
|
| 79 |
+
if self.started_at is None:
|
| 80 |
+
return 0.0
|
| 81 |
+
end_time = self.completed_at or datetime.now()
|
| 82 |
+
return (end_time - self.started_at).total_seconds()
|
| 83 |
+
|
| 84 |
+
def to_dict(self) -> dict[str, Any]:
|
| 85 |
+
"""Convert job to dictionary for API response."""
|
| 86 |
+
data: dict[str, Any] = {
|
| 87 |
+
"jobId": self.id,
|
| 88 |
+
"status": self.status.value,
|
| 89 |
+
"progress": self.progress,
|
| 90 |
+
"progressMessage": self.progress_message,
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
if self.started_at is not None:
|
| 94 |
+
data["elapsedSeconds"] = round(self.elapsed_seconds, 2)
|
| 95 |
+
|
| 96 |
+
if self.status == JobStatus.COMPLETED and self.result is not None:
|
| 97 |
+
data["result"] = self.result
|
| 98 |
+
|
| 99 |
+
if self.status == JobStatus.FAILED and self.error is not None:
|
| 100 |
+
data["error"] = self.error
|
| 101 |
+
|
| 102 |
+
return data
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
class JobStore:
|
| 106 |
+
"""Thread-safe in-memory job store.
|
| 107 |
+
|
| 108 |
+
Provides CRUD operations for jobs with automatic cleanup of old entries.
|
| 109 |
+
Uses a simple dict with a lock for thread safety.
|
| 110 |
+
|
| 111 |
+
Usage:
|
| 112 |
+
store = JobStore()
|
| 113 |
+
job = store.create_job("case123", fast_mode=True)
|
| 114 |
+
store.update_progress(job.id, 50, "Processing...")
|
| 115 |
+
store.complete_job(job.id, {"result": "data"})
|
| 116 |
+
"""
|
| 117 |
+
|
| 118 |
+
# Default time-to-live for completed jobs
|
| 119 |
+
DEFAULT_TTL = timedelta(hours=1)
|
| 120 |
+
|
| 121 |
+
# Cleanup interval (how often to check for expired jobs)
|
| 122 |
+
CLEANUP_INTERVAL_SECONDS = 600 # 10 minutes
|
| 123 |
+
|
| 124 |
+
def __init__(
|
| 125 |
+
self,
|
| 126 |
+
ttl: timedelta = DEFAULT_TTL,
|
| 127 |
+
results_dir: Path | None = None,
|
| 128 |
+
) -> None:
|
| 129 |
+
"""Initialize the job store.
|
| 130 |
+
|
| 131 |
+
Args:
|
| 132 |
+
ttl: How long to keep completed jobs before cleanup
|
| 133 |
+
results_dir: Directory where job results are stored (for cleanup)
|
| 134 |
+
"""
|
| 135 |
+
self._jobs: dict[str, Job] = {}
|
| 136 |
+
self._lock = threading.RLock()
|
| 137 |
+
self._ttl = ttl
|
| 138 |
+
self._results_dir = results_dir or Path("/tmp/stroke-results")
|
| 139 |
+
self._cleanup_thread: threading.Thread | None = None
|
| 140 |
+
self._shutdown = threading.Event()
|
| 141 |
+
|
| 142 |
+
@staticmethod
|
| 143 |
+
def _is_safe_job_id(job_id: str) -> bool:
|
| 144 |
+
"""Validate job ID to prevent path traversal attacks.
|
| 145 |
+
|
| 146 |
+
Only allows alphanumeric characters, hyphens, and underscores.
|
| 147 |
+
"""
|
| 148 |
+
return bool(job_id) and _SAFE_JOB_ID_PATTERN.match(job_id) is not None
|
| 149 |
+
|
| 150 |
+
def create_job(self, job_id: str, case_id: str, fast_mode: bool) -> Job:
|
| 151 |
+
"""Create a new job in PENDING status.
|
| 152 |
+
|
| 153 |
+
Args:
|
| 154 |
+
job_id: Unique identifier for the job
|
| 155 |
+
case_id: Case to process
|
| 156 |
+
fast_mode: Whether to use fast inference
|
| 157 |
+
|
| 158 |
+
Returns:
|
| 159 |
+
The created Job object
|
| 160 |
+
|
| 161 |
+
Raises:
|
| 162 |
+
ValueError: If job_id is invalid (contains unsafe characters)
|
| 163 |
+
KeyError: If job_id already exists
|
| 164 |
+
"""
|
| 165 |
+
if not self._is_safe_job_id(job_id):
|
| 166 |
+
raise ValueError(f"Invalid job_id: {job_id!r}")
|
| 167 |
+
|
| 168 |
+
job = Job(
|
| 169 |
+
id=job_id,
|
| 170 |
+
status=JobStatus.PENDING,
|
| 171 |
+
case_id=case_id,
|
| 172 |
+
fast_mode=fast_mode,
|
| 173 |
+
created_at=datetime.now(),
|
| 174 |
+
)
|
| 175 |
+
with self._lock:
|
| 176 |
+
if job_id in self._jobs:
|
| 177 |
+
raise KeyError(f"Job already exists: {job_id}")
|
| 178 |
+
self._jobs[job_id] = job
|
| 179 |
+
# Note: Don't log case_id as it may be sensitive (medical domain)
|
| 180 |
+
logger.info("Created job %s", job_id)
|
| 181 |
+
return job
|
| 182 |
+
|
| 183 |
+
def get_job(self, job_id: str) -> Job | None:
|
| 184 |
+
"""Get a job by ID.
|
| 185 |
+
|
| 186 |
+
Args:
|
| 187 |
+
job_id: The job identifier
|
| 188 |
+
|
| 189 |
+
Returns:
|
| 190 |
+
The Job object, or None if not found
|
| 191 |
+
"""
|
| 192 |
+
with self._lock:
|
| 193 |
+
return self._jobs.get(job_id)
|
| 194 |
+
|
| 195 |
+
def start_job(self, job_id: str) -> None:
|
| 196 |
+
"""Mark a job as started (RUNNING status).
|
| 197 |
+
|
| 198 |
+
Args:
|
| 199 |
+
job_id: The job identifier
|
| 200 |
+
"""
|
| 201 |
+
with self._lock:
|
| 202 |
+
job = self._jobs.get(job_id)
|
| 203 |
+
if job:
|
| 204 |
+
job.status = JobStatus.RUNNING
|
| 205 |
+
job.started_at = datetime.now()
|
| 206 |
+
job.progress = 5
|
| 207 |
+
job.progress_message = "Starting inference..."
|
| 208 |
+
logger.info("Started job %s", job_id)
|
| 209 |
+
|
| 210 |
+
def update_progress(
|
| 211 |
+
self,
|
| 212 |
+
job_id: str,
|
| 213 |
+
progress: int,
|
| 214 |
+
message: str,
|
| 215 |
+
) -> None:
|
| 216 |
+
"""Update job progress.
|
| 217 |
+
|
| 218 |
+
Args:
|
| 219 |
+
job_id: The job identifier
|
| 220 |
+
progress: Progress percentage (0-100)
|
| 221 |
+
message: Human-readable progress message
|
| 222 |
+
"""
|
| 223 |
+
with self._lock:
|
| 224 |
+
job = self._jobs.get(job_id)
|
| 225 |
+
if job and job.status == JobStatus.RUNNING:
|
| 226 |
+
job.progress = min(max(progress, 0), 100)
|
| 227 |
+
job.progress_message = message
|
| 228 |
+
|
| 229 |
+
def complete_job(self, job_id: str, result: dict[str, Any]) -> None:
|
| 230 |
+
"""Mark a job as successfully completed.
|
| 231 |
+
|
| 232 |
+
Args:
|
| 233 |
+
job_id: The job identifier
|
| 234 |
+
result: The segmentation results
|
| 235 |
+
"""
|
| 236 |
+
with self._lock:
|
| 237 |
+
job = self._jobs.get(job_id)
|
| 238 |
+
if job:
|
| 239 |
+
# Ensure started_at is set for elapsed time calculation
|
| 240 |
+
if job.started_at is None:
|
| 241 |
+
job.started_at = datetime.now()
|
| 242 |
+
job.status = JobStatus.COMPLETED
|
| 243 |
+
job.completed_at = datetime.now()
|
| 244 |
+
job.progress = 100
|
| 245 |
+
job.progress_message = "Segmentation complete"
|
| 246 |
+
job.result = result
|
| 247 |
+
logger.info(
|
| 248 |
+
"Completed job %s in %.2fs",
|
| 249 |
+
job_id,
|
| 250 |
+
job.elapsed_seconds,
|
| 251 |
+
)
|
| 252 |
+
|
| 253 |
+
def fail_job(self, job_id: str, error: str) -> None:
|
| 254 |
+
"""Mark a job as failed.
|
| 255 |
+
|
| 256 |
+
Args:
|
| 257 |
+
job_id: The job identifier
|
| 258 |
+
error: Error message describing the failure
|
| 259 |
+
"""
|
| 260 |
+
with self._lock:
|
| 261 |
+
job = self._jobs.get(job_id)
|
| 262 |
+
if job:
|
| 263 |
+
# Ensure started_at is set for elapsed time calculation
|
| 264 |
+
if job.started_at is None:
|
| 265 |
+
job.started_at = datetime.now()
|
| 266 |
+
job.status = JobStatus.FAILED
|
| 267 |
+
job.completed_at = datetime.now()
|
| 268 |
+
job.progress_message = "Error occurred"
|
| 269 |
+
job.error = error
|
| 270 |
+
logger.error("Failed job %s: %s", job_id, error)
|
| 271 |
+
|
| 272 |
+
def cleanup_old_jobs(self) -> int:
|
| 273 |
+
"""Remove jobs older than TTL to prevent memory leaks.
|
| 274 |
+
|
| 275 |
+
Also cleans up associated result files on disk.
|
| 276 |
+
|
| 277 |
+
Returns:
|
| 278 |
+
Number of jobs cleaned up
|
| 279 |
+
"""
|
| 280 |
+
now = datetime.now()
|
| 281 |
+
expired_ids: list[str] = []
|
| 282 |
+
|
| 283 |
+
with self._lock:
|
| 284 |
+
for job_id, job in self._jobs.items():
|
| 285 |
+
if job.completed_at and (now - job.completed_at) > self._ttl:
|
| 286 |
+
expired_ids.append(job_id)
|
| 287 |
+
|
| 288 |
+
for job_id in expired_ids:
|
| 289 |
+
del self._jobs[job_id]
|
| 290 |
+
|
| 291 |
+
# Clean up result files outside the lock
|
| 292 |
+
# Use path validation to prevent path traversal attacks
|
| 293 |
+
base_dir = self._results_dir.resolve()
|
| 294 |
+
for job_id in expired_ids:
|
| 295 |
+
# Skip cleanup for unsafe job IDs (shouldn't happen, but defense in depth)
|
| 296 |
+
if not self._is_safe_job_id(job_id):
|
| 297 |
+
logger.warning("Skipping cleanup for unsafe job id %r", job_id)
|
| 298 |
+
continue
|
| 299 |
+
|
| 300 |
+
result_dir = (self._results_dir / job_id).resolve()
|
| 301 |
+
# Verify path is within results directory (prevent traversal)
|
| 302 |
+
if not result_dir.is_relative_to(base_dir):
|
| 303 |
+
logger.warning("Path traversal attempt blocked for job %s", job_id)
|
| 304 |
+
continue
|
| 305 |
+
|
| 306 |
+
if result_dir.exists():
|
| 307 |
+
try:
|
| 308 |
+
shutil.rmtree(result_dir)
|
| 309 |
+
logger.debug("Cleaned up result files for job %s", job_id)
|
| 310 |
+
except OSError as e:
|
| 311 |
+
logger.warning("Failed to cleanup %s: %s", result_dir, e)
|
| 312 |
+
|
| 313 |
+
if expired_ids:
|
| 314 |
+
logger.info("Cleaned up %d expired jobs", len(expired_ids))
|
| 315 |
+
|
| 316 |
+
return len(expired_ids)
|
| 317 |
+
|
| 318 |
+
def start_cleanup_scheduler(self) -> None:
|
| 319 |
+
"""Start background thread for periodic job cleanup."""
|
| 320 |
+
if self._cleanup_thread is not None:
|
| 321 |
+
return # Already running
|
| 322 |
+
|
| 323 |
+
def cleanup_loop() -> None:
|
| 324 |
+
while not self._shutdown.wait(self.CLEANUP_INTERVAL_SECONDS):
|
| 325 |
+
try:
|
| 326 |
+
self.cleanup_old_jobs()
|
| 327 |
+
except Exception:
|
| 328 |
+
logger.exception("Error during job cleanup")
|
| 329 |
+
|
| 330 |
+
self._cleanup_thread = threading.Thread(
|
| 331 |
+
target=cleanup_loop,
|
| 332 |
+
daemon=True,
|
| 333 |
+
name="job-cleanup",
|
| 334 |
+
)
|
| 335 |
+
self._cleanup_thread.start()
|
| 336 |
+
logger.info("Started job cleanup scheduler (interval=%ds)", self.CLEANUP_INTERVAL_SECONDS)
|
| 337 |
+
|
| 338 |
+
def stop_cleanup_scheduler(self) -> None:
|
| 339 |
+
"""Stop the cleanup scheduler thread."""
|
| 340 |
+
self._shutdown.set()
|
| 341 |
+
if self._cleanup_thread:
|
| 342 |
+
self._cleanup_thread.join(timeout=5)
|
| 343 |
+
self._cleanup_thread = None
|
| 344 |
+
logger.info("Stopped job cleanup scheduler")
|
| 345 |
+
|
| 346 |
+
def __len__(self) -> int:
|
| 347 |
+
"""Return number of jobs in store."""
|
| 348 |
+
with self._lock:
|
| 349 |
+
return len(self._jobs)
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
# Global job store instance
|
| 353 |
+
# Initialized in main.py on app startup
|
| 354 |
+
job_store: JobStore | None = None
|
| 355 |
+
|
| 356 |
+
|
| 357 |
+
def get_job_store() -> JobStore:
|
| 358 |
+
"""Get the global job store instance.
|
| 359 |
+
|
| 360 |
+
Raises:
|
| 361 |
+
RuntimeError: If job store not initialized
|
| 362 |
+
"""
|
| 363 |
+
if job_store is None:
|
| 364 |
+
raise RuntimeError("Job store not initialized. Call init_job_store() first.")
|
| 365 |
+
return job_store
|
| 366 |
+
|
| 367 |
+
|
| 368 |
+
def init_job_store(results_dir: Path | None = None) -> JobStore:
|
| 369 |
+
"""Initialize the global job store.
|
| 370 |
+
|
| 371 |
+
Args:
|
| 372 |
+
results_dir: Directory for job results
|
| 373 |
+
|
| 374 |
+
Returns:
|
| 375 |
+
The initialized JobStore
|
| 376 |
+
"""
|
| 377 |
+
global job_store
|
| 378 |
+
job_store = JobStore(results_dir=results_dir)
|
| 379 |
+
job_store.start_cleanup_scheduler()
|
| 380 |
+
return job_store
|
|
@@ -1,6 +1,21 @@
|
|
| 1 |
-
"""FastAPI application for stroke segmentation API.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
import os
|
|
|
|
|
|
|
| 4 |
from pathlib import Path
|
| 5 |
from typing import Any
|
| 6 |
|
|
@@ -8,12 +23,49 @@ from fastapi import FastAPI
|
|
| 8 |
from fastapi.middleware.cors import CORSMiddleware
|
| 9 |
from fastapi.staticfiles import StaticFiles
|
| 10 |
|
|
|
|
| 11 |
from stroke_deepisles_demo.api.routes import router
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
app = FastAPI(
|
| 14 |
title="Stroke Segmentation API",
|
| 15 |
-
description="DeepISLES stroke lesion segmentation",
|
| 16 |
-
version="
|
|
|
|
| 17 |
)
|
| 18 |
|
| 19 |
# CORS configuration
|
|
@@ -41,8 +93,7 @@ app.add_middleware(
|
|
| 41 |
app.include_router(router, prefix="/api")
|
| 42 |
|
| 43 |
# Static files for NIfTI results
|
| 44 |
-
#
|
| 45 |
-
RESULTS_DIR = Path("/tmp/stroke-results")
|
| 46 |
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
| 47 |
app.mount("/files", StaticFiles(directory=str(RESULTS_DIR)), name="files")
|
| 48 |
|
|
@@ -50,4 +101,23 @@ app.mount("/files", StaticFiles(directory=str(RESULTS_DIR)), name="files")
|
|
| 50 |
@app.get("/")
|
| 51 |
async def root() -> dict[str, Any]:
|
| 52 |
"""Health check endpoint."""
|
| 53 |
-
return {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""FastAPI application for stroke segmentation API.
|
| 2 |
+
|
| 3 |
+
This API provides async ML inference for stroke lesion segmentation using DeepISLES.
|
| 4 |
+
It implements a job queue pattern to handle long-running inference without timeouts:
|
| 5 |
+
|
| 6 |
+
1. POST /api/segment - Creates job, returns immediately (202)
|
| 7 |
+
2. GET /api/jobs/{id} - Poll for status/progress/results
|
| 8 |
+
3. GET /files/{job_id}/... - Download result NIfTI files
|
| 9 |
+
|
| 10 |
+
Architecture designed to work within HuggingFace Spaces constraints:
|
| 11 |
+
- ~60s gateway timeout (avoided via async job pattern)
|
| 12 |
+
- Single worker (in-memory job store is sufficient)
|
| 13 |
+
- /tmp writable only (results stored there)
|
| 14 |
+
"""
|
| 15 |
|
| 16 |
import os
|
| 17 |
+
from collections.abc import AsyncIterator
|
| 18 |
+
from contextlib import asynccontextmanager
|
| 19 |
from pathlib import Path
|
| 20 |
from typing import Any
|
| 21 |
|
|
|
|
| 23 |
from fastapi.middleware.cors import CORSMiddleware
|
| 24 |
from fastapi.staticfiles import StaticFiles
|
| 25 |
|
| 26 |
+
from stroke_deepisles_demo.api.job_store import init_job_store
|
| 27 |
from stroke_deepisles_demo.api.routes import router
|
| 28 |
+
from stroke_deepisles_demo.core.logging import get_logger
|
| 29 |
+
|
| 30 |
+
logger = get_logger(__name__)
|
| 31 |
+
|
| 32 |
+
# Results directory (must be in /tmp for HF Spaces)
|
| 33 |
+
RESULTS_DIR = Path("/tmp/stroke-results")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
@asynccontextmanager
|
| 37 |
+
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
|
| 38 |
+
"""Application lifespan handler for startup/shutdown tasks.
|
| 39 |
+
|
| 40 |
+
Startup:
|
| 41 |
+
- Initialize job store with cleanup scheduler
|
| 42 |
+
- Create results directory
|
| 43 |
+
|
| 44 |
+
Shutdown:
|
| 45 |
+
- Stop cleanup scheduler
|
| 46 |
+
"""
|
| 47 |
+
# Startup
|
| 48 |
+
logger.info("Starting stroke segmentation API...")
|
| 49 |
+
|
| 50 |
+
# Create results directory
|
| 51 |
+
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
| 52 |
+
|
| 53 |
+
# Initialize job store with cleanup scheduler
|
| 54 |
+
job_store = init_job_store(results_dir=RESULTS_DIR)
|
| 55 |
+
logger.info("Job store initialized with %d jobs", len(job_store))
|
| 56 |
+
|
| 57 |
+
yield
|
| 58 |
+
|
| 59 |
+
# Shutdown
|
| 60 |
+
logger.info("Shutting down stroke segmentation API...")
|
| 61 |
+
job_store.stop_cleanup_scheduler()
|
| 62 |
+
|
| 63 |
|
| 64 |
app = FastAPI(
|
| 65 |
title="Stroke Segmentation API",
|
| 66 |
+
description="DeepISLES stroke lesion segmentation with async job queue",
|
| 67 |
+
version="2.0.0",
|
| 68 |
+
lifespan=lifespan,
|
| 69 |
)
|
| 70 |
|
| 71 |
# CORS configuration
|
|
|
|
| 93 |
app.include_router(router, prefix="/api")
|
| 94 |
|
| 95 |
# Static files for NIfTI results
|
| 96 |
+
# Note: Mount happens at import time; ensure directory exists here as well.
|
|
|
|
| 97 |
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
| 98 |
app.mount("/files", StaticFiles(directory=str(RESULTS_DIR)), name="files")
|
| 99 |
|
|
|
|
| 101 |
@app.get("/")
|
| 102 |
async def root() -> dict[str, Any]:
|
| 103 |
"""Health check endpoint."""
|
| 104 |
+
return {
|
| 105 |
+
"status": "healthy",
|
| 106 |
+
"service": "stroke-segmentation-api",
|
| 107 |
+
"version": "2.0.0",
|
| 108 |
+
"features": ["async-jobs", "progress-tracking"],
|
| 109 |
+
}
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
@app.get("/health")
|
| 113 |
+
async def health() -> dict[str, Any]:
|
| 114 |
+
"""Detailed health check endpoint."""
|
| 115 |
+
from stroke_deepisles_demo.api.job_store import get_job_store
|
| 116 |
+
|
| 117 |
+
store = get_job_store()
|
| 118 |
+
return {
|
| 119 |
+
"status": "healthy",
|
| 120 |
+
"jobs_in_memory": len(store),
|
| 121 |
+
"results_dir": str(RESULTS_DIR),
|
| 122 |
+
"results_dir_exists": RESULTS_DIR.exists(),
|
| 123 |
+
}
|
|
@@ -1,17 +1,37 @@
|
|
| 1 |
-
"""API route handlers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
import contextlib
|
| 4 |
import os
|
| 5 |
import uuid
|
| 6 |
from pathlib import Path
|
| 7 |
|
| 8 |
-
from fastapi import APIRouter, HTTPException, Request
|
| 9 |
|
| 10 |
-
from stroke_deepisles_demo.api.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
from stroke_deepisles_demo.data import list_case_ids
|
| 12 |
from stroke_deepisles_demo.metrics import compute_volume_ml
|
| 13 |
from stroke_deepisles_demo.pipeline import run_pipeline_on_case
|
| 14 |
|
|
|
|
|
|
|
| 15 |
router = APIRouter()
|
| 16 |
|
| 17 |
# Base directory for results
|
|
@@ -43,52 +63,201 @@ def get_cases() -> CasesResponse:
|
|
| 43 |
return CasesResponse(cases=cases)
|
| 44 |
except HTTPException:
|
| 45 |
raise
|
| 46 |
-
except Exception
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
-
|
| 51 |
-
def run_segmentation(request: Request, body: SegmentRequest) -> SegmentResponse:
|
| 52 |
-
"""Run DeepISLES segmentation on a case.
|
| 53 |
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
try:
|
| 59 |
-
#
|
| 60 |
-
|
| 61 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
result = run_pipeline_on_case(
|
| 64 |
-
|
| 65 |
output_dir=output_dir,
|
| 66 |
-
fast=
|
| 67 |
compute_dice=True,
|
| 68 |
cleanup_staging=True,
|
| 69 |
)
|
| 70 |
|
|
|
|
|
|
|
| 71 |
# Compute volume (may fail for edge cases)
|
| 72 |
volume_ml = None
|
| 73 |
with contextlib.suppress(Exception):
|
| 74 |
volume_ml = round(compute_volume_ml(result.prediction_mask, threshold=0.5), 2)
|
| 75 |
|
| 76 |
-
|
| 77 |
-
|
|
|
|
| 78 |
dwi_filename = result.input_files["dwi"].name
|
| 79 |
pred_filename = result.prediction_mask.name
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
-
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
predictionUrl=f"{backend_url}{file_path_prefix}/{pred_filename}",
|
| 90 |
)
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
| 1 |
+
"""API route handlers for stroke segmentation.
|
| 2 |
+
|
| 3 |
+
This module implements an async job queue pattern to handle long-running ML inference:
|
| 4 |
+
1. POST /api/segment creates a job and returns immediately (202 Accepted)
|
| 5 |
+
2. Background task runs the inference
|
| 6 |
+
3. Frontend polls GET /api/jobs/{job_id} for status/results
|
| 7 |
+
|
| 8 |
+
This pattern avoids HuggingFace Spaces' ~60s gateway timeout.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
|
| 13 |
import contextlib
|
| 14 |
import os
|
| 15 |
import uuid
|
| 16 |
from pathlib import Path
|
| 17 |
|
| 18 |
+
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
|
| 19 |
|
| 20 |
+
from stroke_deepisles_demo.api.job_store import JobStatus, get_job_store
|
| 21 |
+
from stroke_deepisles_demo.api.schemas import (
|
| 22 |
+
CasesResponse,
|
| 23 |
+
CreateJobResponse,
|
| 24 |
+
JobStatusResponse,
|
| 25 |
+
SegmentRequest,
|
| 26 |
+
SegmentResponse,
|
| 27 |
+
)
|
| 28 |
+
from stroke_deepisles_demo.core.logging import get_logger
|
| 29 |
from stroke_deepisles_demo.data import list_case_ids
|
| 30 |
from stroke_deepisles_demo.metrics import compute_volume_ml
|
| 31 |
from stroke_deepisles_demo.pipeline import run_pipeline_on_case
|
| 32 |
|
| 33 |
+
logger = get_logger(__name__)
|
| 34 |
+
|
| 35 |
router = APIRouter()
|
| 36 |
|
| 37 |
# Base directory for results
|
|
|
|
| 63 |
return CasesResponse(cases=cases)
|
| 64 |
except HTTPException:
|
| 65 |
raise
|
| 66 |
+
except Exception:
|
| 67 |
+
logger.exception("Failed to list cases")
|
| 68 |
+
raise HTTPException(status_code=500, detail="Failed to retrieve cases") from None
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
@router.post(
|
| 72 |
+
"/segment",
|
| 73 |
+
response_model=CreateJobResponse,
|
| 74 |
+
status_code=202,
|
| 75 |
+
responses={
|
| 76 |
+
202: {"description": "Job created successfully"},
|
| 77 |
+
400: {"description": "Invalid request"},
|
| 78 |
+
500: {"description": "Internal server error"},
|
| 79 |
+
},
|
| 80 |
+
)
|
| 81 |
+
def create_segment_job(
|
| 82 |
+
request: Request,
|
| 83 |
+
body: SegmentRequest,
|
| 84 |
+
background_tasks: BackgroundTasks,
|
| 85 |
+
) -> CreateJobResponse:
|
| 86 |
+
"""Create an async segmentation job.
|
| 87 |
+
|
| 88 |
+
Returns immediately with a job ID. The actual ML inference runs in the background.
|
| 89 |
+
Poll GET /api/jobs/{jobId} for status updates and results.
|
| 90 |
+
|
| 91 |
+
This async pattern is required because:
|
| 92 |
+
- DeepISLES inference takes 30-60 seconds
|
| 93 |
+
- HuggingFace Spaces has a ~60s gateway timeout
|
| 94 |
+
- Returning immediately avoids timeout errors
|
| 95 |
+
"""
|
| 96 |
+
try:
|
| 97 |
+
# Use full UUID hex for uniqueness (no truncation)
|
| 98 |
+
job_id = uuid.uuid4().hex
|
| 99 |
+
store = get_job_store()
|
| 100 |
+
backend_url = get_backend_base_url(request)
|
| 101 |
+
|
| 102 |
+
# Create job record
|
| 103 |
+
store.create_job(job_id, body.case_id, body.fast_mode)
|
| 104 |
+
|
| 105 |
+
# Queue background task
|
| 106 |
+
background_tasks.add_task(
|
| 107 |
+
run_segmentation_job,
|
| 108 |
+
job_id=job_id,
|
| 109 |
+
case_id=body.case_id,
|
| 110 |
+
fast_mode=body.fast_mode,
|
| 111 |
+
backend_url=backend_url,
|
| 112 |
+
)
|
| 113 |
+
|
| 114 |
+
# Note: Don't log case_id as it may be sensitive (medical domain)
|
| 115 |
+
logger.info("Created segmentation job %s", job_id)
|
| 116 |
+
|
| 117 |
+
return CreateJobResponse(
|
| 118 |
+
jobId=job_id,
|
| 119 |
+
status="pending",
|
| 120 |
+
message=f"Segmentation job queued for {body.case_id}",
|
| 121 |
+
)
|
| 122 |
+
|
| 123 |
+
except Exception:
|
| 124 |
+
logger.exception("Failed to create segmentation job")
|
| 125 |
+
raise HTTPException(status_code=500, detail="Failed to create segmentation job") from None
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
@router.get(
|
| 129 |
+
"/jobs/{job_id}",
|
| 130 |
+
response_model=JobStatusResponse,
|
| 131 |
+
responses={
|
| 132 |
+
200: {"description": "Job status retrieved"},
|
| 133 |
+
404: {"description": "Job not found"},
|
| 134 |
+
},
|
| 135 |
+
)
|
| 136 |
+
def get_job_status(job_id: str) -> JobStatusResponse:
|
| 137 |
+
"""Get the status of a segmentation job.
|
| 138 |
+
|
| 139 |
+
Poll this endpoint to track job progress and retrieve results.
|
| 140 |
+
|
| 141 |
+
Returns:
|
| 142 |
+
Job status including progress percentage and results when completed.
|
| 143 |
|
| 144 |
+
Raises:
|
| 145 |
+
404: Job not found (may have expired or never existed)
|
| 146 |
+
"""
|
| 147 |
+
store = get_job_store()
|
| 148 |
+
job = store.get_job(job_id)
|
| 149 |
+
|
| 150 |
+
if job is None:
|
| 151 |
+
raise HTTPException(
|
| 152 |
+
status_code=404,
|
| 153 |
+
detail=f"Job not found: {job_id}. Jobs expire after 1 hour.",
|
| 154 |
+
)
|
| 155 |
+
|
| 156 |
+
# Build response from job data
|
| 157 |
+
response = JobStatusResponse(
|
| 158 |
+
jobId=job.id,
|
| 159 |
+
status=job.status.value,
|
| 160 |
+
progress=job.progress,
|
| 161 |
+
progressMessage=job.progress_message,
|
| 162 |
+
elapsedSeconds=round(job.elapsed_seconds, 2) if job.started_at else None,
|
| 163 |
+
result=None,
|
| 164 |
+
error=None,
|
| 165 |
+
)
|
| 166 |
+
|
| 167 |
+
# Include result if completed
|
| 168 |
+
if job.status == JobStatus.COMPLETED and job.result:
|
| 169 |
+
response.result = SegmentResponse(**job.result)
|
| 170 |
+
|
| 171 |
+
# Include error if failed
|
| 172 |
+
if job.status == JobStatus.FAILED and job.error:
|
| 173 |
+
response.error = job.error
|
| 174 |
|
| 175 |
+
return response
|
|
|
|
|
|
|
| 176 |
|
| 177 |
+
|
| 178 |
+
def run_segmentation_job(
|
| 179 |
+
job_id: str,
|
| 180 |
+
case_id: str,
|
| 181 |
+
fast_mode: bool,
|
| 182 |
+
backend_url: str,
|
| 183 |
+
) -> None:
|
| 184 |
+
"""Execute segmentation in background thread.
|
| 185 |
+
|
| 186 |
+
This function runs in a threadpool (not the main event loop) because
|
| 187 |
+
the ML inference is CPU/GPU-bound and blocking.
|
| 188 |
+
|
| 189 |
+
Updates job status and progress throughout execution, allowing the
|
| 190 |
+
frontend to show meaningful progress updates.
|
| 191 |
+
|
| 192 |
+
Args:
|
| 193 |
+
job_id: Unique job identifier
|
| 194 |
+
case_id: Case to process
|
| 195 |
+
fast_mode: Whether to use fast inference mode
|
| 196 |
+
backend_url: Base URL for constructing result file URLs
|
| 197 |
"""
|
| 198 |
+
store = get_job_store()
|
| 199 |
+
job = store.get_job(job_id)
|
| 200 |
+
|
| 201 |
+
if job is None:
|
| 202 |
+
logger.error("Job %s not found when starting execution", job_id)
|
| 203 |
+
return
|
| 204 |
+
|
| 205 |
try:
|
| 206 |
+
# Mark as running
|
| 207 |
+
store.start_job(job_id)
|
| 208 |
+
store.update_progress(job_id, 10, "Loading case data...")
|
| 209 |
+
|
| 210 |
+
# Set up output directory
|
| 211 |
+
output_dir = RESULTS_BASE / job_id
|
| 212 |
+
|
| 213 |
+
store.update_progress(job_id, 20, "Staging files for DeepISLES...")
|
| 214 |
+
|
| 215 |
+
# Run the pipeline
|
| 216 |
+
store.update_progress(job_id, 30, "Running DeepISLES inference...")
|
| 217 |
|
| 218 |
result = run_pipeline_on_case(
|
| 219 |
+
case_id,
|
| 220 |
output_dir=output_dir,
|
| 221 |
+
fast=fast_mode,
|
| 222 |
compute_dice=True,
|
| 223 |
cleanup_staging=True,
|
| 224 |
)
|
| 225 |
|
| 226 |
+
store.update_progress(job_id, 85, "Computing metrics...")
|
| 227 |
+
|
| 228 |
# Compute volume (may fail for edge cases)
|
| 229 |
volume_ml = None
|
| 230 |
with contextlib.suppress(Exception):
|
| 231 |
volume_ml = round(compute_volume_ml(result.prediction_mask, threshold=0.5), 2)
|
| 232 |
|
| 233 |
+
store.update_progress(job_id, 95, "Preparing results...")
|
| 234 |
+
|
| 235 |
+
# Build result data
|
| 236 |
dwi_filename = result.input_files["dwi"].name
|
| 237 |
pred_filename = result.prediction_mask.name
|
| 238 |
+
file_path_prefix = f"/files/{job_id}/{result.case_id}"
|
| 239 |
+
|
| 240 |
+
result_data = {
|
| 241 |
+
"caseId": result.case_id,
|
| 242 |
+
"diceScore": result.dice_score,
|
| 243 |
+
"volumeMl": volume_ml,
|
| 244 |
+
"elapsedSeconds": round(result.elapsed_seconds, 2),
|
| 245 |
+
"dwiUrl": f"{backend_url}{file_path_prefix}/{dwi_filename}",
|
| 246 |
+
"predictionUrl": f"{backend_url}{file_path_prefix}/{pred_filename}",
|
| 247 |
+
}
|
| 248 |
|
| 249 |
+
# Mark as completed
|
| 250 |
+
store.complete_job(job_id, result_data)
|
| 251 |
|
| 252 |
+
logger.info(
|
| 253 |
+
"Job %s completed: case=%s, dice=%.3f, time=%.1fs",
|
| 254 |
+
job_id,
|
| 255 |
+
case_id,
|
| 256 |
+
result.dice_score or 0,
|
| 257 |
+
result.elapsed_seconds,
|
|
|
|
| 258 |
)
|
| 259 |
+
|
| 260 |
+
except Exception:
|
| 261 |
+
logger.exception("Job %s failed", job_id)
|
| 262 |
+
# Sanitize error message - don't expose internal details to clients
|
| 263 |
+
store.fail_job(job_id, "Segmentation failed")
|
|
@@ -1,6 +1,8 @@
|
|
| 1 |
"""Pydantic schemas for API requests and responses."""
|
| 2 |
|
| 3 |
-
from
|
|
|
|
|
|
|
| 4 |
|
| 5 |
|
| 6 |
class CasesResponse(BaseModel):
|
|
@@ -17,7 +19,7 @@ class SegmentRequest(BaseModel):
|
|
| 17 |
|
| 18 |
|
| 19 |
class SegmentResponse(BaseModel):
|
| 20 |
-
"""
|
| 21 |
|
| 22 |
caseId: str
|
| 23 |
diceScore: float | None
|
|
@@ -25,3 +27,44 @@ class SegmentResponse(BaseModel):
|
|
| 25 |
elapsedSeconds: float
|
| 26 |
dwiUrl: str
|
| 27 |
predictionUrl: str
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""Pydantic schemas for API requests and responses."""
|
| 2 |
|
| 3 |
+
from typing import Literal
|
| 4 |
+
|
| 5 |
+
from pydantic import BaseModel, Field
|
| 6 |
|
| 7 |
|
| 8 |
class CasesResponse(BaseModel):
|
|
|
|
| 19 |
|
| 20 |
|
| 21 |
class SegmentResponse(BaseModel):
|
| 22 |
+
"""Segmentation result data (embedded in job response when completed)."""
|
| 23 |
|
| 24 |
caseId: str
|
| 25 |
diceScore: float | None
|
|
|
|
| 27 |
elapsedSeconds: float
|
| 28 |
dwiUrl: str
|
| 29 |
predictionUrl: str
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
# Job status type for strong typing
|
| 33 |
+
JobStatusType = Literal["pending", "running", "completed", "failed"]
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
class CreateJobResponse(BaseModel):
|
| 37 |
+
"""Response for POST /api/segment (async job creation).
|
| 38 |
+
|
| 39 |
+
Returns immediately with job ID. Client should poll GET /api/jobs/{jobId}
|
| 40 |
+
for status updates and results.
|
| 41 |
+
"""
|
| 42 |
+
|
| 43 |
+
jobId: str = Field(..., description="Unique job identifier for polling")
|
| 44 |
+
status: JobStatusType = Field(..., description="Initial job status (always 'pending')")
|
| 45 |
+
message: str = Field(..., description="Human-readable status message")
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
class JobStatusResponse(BaseModel):
|
| 49 |
+
"""Response for GET /api/jobs/{job_id}.
|
| 50 |
+
|
| 51 |
+
Provides current job status, progress, and results when completed.
|
| 52 |
+
"""
|
| 53 |
+
|
| 54 |
+
jobId: str = Field(..., description="Unique job identifier")
|
| 55 |
+
status: JobStatusType = Field(..., description="Current job status")
|
| 56 |
+
progress: int = Field(..., ge=0, le=100, description="Progress percentage (0-100)")
|
| 57 |
+
progressMessage: str = Field(..., description="Human-readable progress status")
|
| 58 |
+
elapsedSeconds: float | None = Field(
|
| 59 |
+
None, description="Time elapsed since job started (seconds)"
|
| 60 |
+
)
|
| 61 |
+
result: SegmentResponse | None = Field(
|
| 62 |
+
None, description="Segmentation results (only present when status='completed')"
|
| 63 |
+
)
|
| 64 |
+
error: str | None = Field(None, description="Error message (only present when status='failed')")
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
class ErrorResponse(BaseModel):
|
| 68 |
+
"""Standard error response body."""
|
| 69 |
+
|
| 70 |
+
detail: str = Field(..., description="Error description")
|
|
@@ -1,20 +1,32 @@
|
|
| 1 |
"""TDD tests for API endpoints.
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
| 4 |
"""
|
| 5 |
|
| 6 |
-
from
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
import pytest
|
| 9 |
from fastapi.testclient import TestClient
|
| 10 |
|
| 11 |
from stroke_deepisles_demo.api import app
|
|
|
|
| 12 |
|
| 13 |
|
| 14 |
@pytest.fixture
|
| 15 |
-
def client() -> TestClient:
|
| 16 |
-
"""Create test client for FastAPI app."""
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
class TestHealthCheck:
|
|
@@ -63,67 +75,40 @@ class TestGetCases:
|
|
| 63 |
response = client.get("/api/cases")
|
| 64 |
|
| 65 |
assert response.status_code == 500
|
| 66 |
-
|
|
|
|
| 67 |
|
| 68 |
|
| 69 |
class TestPostSegment:
|
| 70 |
-
"""Tests for POST /api/segment endpoint."""
|
| 71 |
-
|
| 72 |
-
def test_runs_segmentation_and_returns_result(self, client: TestClient) -> None:
|
| 73 |
-
"""POST /api/segment runs pipeline and returns metrics + URLs."""
|
| 74 |
-
mock_result = MagicMock()
|
| 75 |
-
mock_result.case_id = "sub-stroke0001"
|
| 76 |
-
mock_result.dice_score = 0.847
|
| 77 |
-
mock_result.elapsed_seconds = 12.5
|
| 78 |
-
mock_result.prediction_mask.name = "prediction.nii.gz"
|
| 79 |
-
mock_result.input_files = {"dwi": MagicMock(name="dwi.nii.gz")}
|
| 80 |
-
mock_result.input_files["dwi"].name = "dwi.nii.gz"
|
| 81 |
-
|
| 82 |
-
with (
|
| 83 |
-
patch("stroke_deepisles_demo.api.routes.run_pipeline_on_case") as mock_pipeline,
|
| 84 |
-
patch("stroke_deepisles_demo.api.routes.compute_volume_ml") as mock_volume,
|
| 85 |
-
):
|
| 86 |
-
mock_pipeline.return_value = mock_result
|
| 87 |
-
mock_volume.return_value = 15.32
|
| 88 |
-
|
| 89 |
-
response = client.post(
|
| 90 |
-
"/api/segment",
|
| 91 |
-
json={"case_id": "sub-stroke0001", "fast_mode": True},
|
| 92 |
-
)
|
| 93 |
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
""
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
)
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
json={"case_id": "sub-stroke0001", "fast_mode": False},
|
| 122 |
-
)
|
| 123 |
-
|
| 124 |
-
mock_pipeline.assert_called_once()
|
| 125 |
-
call_kwargs = mock_pipeline.call_args[1]
|
| 126 |
-
assert call_kwargs["fast"] is False
|
| 127 |
|
| 128 |
def test_returns_422_on_missing_case_id(self, client: TestClient) -> None:
|
| 129 |
"""POST /api/segment returns 422 when case_id is missing."""
|
|
@@ -131,15 +116,78 @@ class TestPostSegment:
|
|
| 131 |
|
| 132 |
assert response.status_code == 422
|
| 133 |
|
| 134 |
-
def test_returns_500_on_pipeline_error(self, client: TestClient) -> None:
|
| 135 |
-
"""POST /api/segment returns 500 when pipeline raises exception."""
|
| 136 |
-
with patch("stroke_deepisles_demo.api.routes.run_pipeline_on_case") as mock_pipeline:
|
| 137 |
-
mock_pipeline.side_effect = RuntimeError("GPU out of memory")
|
| 138 |
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
json={"case_id": "sub-stroke0001"},
|
| 142 |
-
)
|
| 143 |
|
| 144 |
-
|
| 145 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""TDD tests for API endpoints.
|
| 2 |
|
| 3 |
+
Tests the FastAPI REST API with async job queue pattern.
|
| 4 |
+
POST /api/segment returns 202 Accepted with job ID.
|
| 5 |
+
GET /api/jobs/{id} returns job status/progress/results.
|
| 6 |
"""
|
| 7 |
|
| 8 |
+
from collections.abc import Generator
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
from tempfile import TemporaryDirectory
|
| 11 |
+
from unittest.mock import patch
|
| 12 |
|
| 13 |
import pytest
|
| 14 |
from fastapi.testclient import TestClient
|
| 15 |
|
| 16 |
from stroke_deepisles_demo.api import app
|
| 17 |
+
from stroke_deepisles_demo.api.job_store import init_job_store
|
| 18 |
|
| 19 |
|
| 20 |
@pytest.fixture
|
| 21 |
+
def client() -> Generator[TestClient, None, None]:
|
| 22 |
+
"""Create test client for FastAPI app with fresh job store."""
|
| 23 |
+
with TemporaryDirectory() as tmpdir:
|
| 24 |
+
# Initialize a fresh job store for each test
|
| 25 |
+
store = init_job_store(results_dir=Path(tmpdir))
|
| 26 |
+
try:
|
| 27 |
+
yield TestClient(app)
|
| 28 |
+
finally:
|
| 29 |
+
store.stop_cleanup_scheduler()
|
| 30 |
|
| 31 |
|
| 32 |
class TestHealthCheck:
|
|
|
|
| 75 |
response = client.get("/api/cases")
|
| 76 |
|
| 77 |
assert response.status_code == 500
|
| 78 |
+
# Note: Error message is sanitized (doesn't expose internal details)
|
| 79 |
+
assert "Failed to retrieve cases" in response.json()["detail"]
|
| 80 |
|
| 81 |
|
| 82 |
class TestPostSegment:
|
| 83 |
+
"""Tests for POST /api/segment endpoint (async job creation)."""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
|
| 85 |
+
def test_creates_job_and_returns_202(self, client: TestClient) -> None:
|
| 86 |
+
"""POST /api/segment creates a job and returns 202 Accepted."""
|
| 87 |
+
response = client.post(
|
| 88 |
+
"/api/segment",
|
| 89 |
+
json={"case_id": "sub-stroke0001", "fast_mode": True},
|
| 90 |
+
)
|
| 91 |
+
|
| 92 |
+
assert response.status_code == 202
|
| 93 |
+
data = response.json()
|
| 94 |
+
assert "jobId" in data
|
| 95 |
+
assert data["status"] == "pending"
|
| 96 |
+
assert "message" in data
|
| 97 |
+
|
| 98 |
+
def test_returns_job_id_for_polling(self, client: TestClient) -> None:
|
| 99 |
+
"""POST /api/segment returns a job ID that can be used for polling."""
|
| 100 |
+
response = client.post(
|
| 101 |
+
"/api/segment",
|
| 102 |
+
json={"case_id": "sub-stroke0001", "fast_mode": True},
|
| 103 |
+
)
|
| 104 |
+
|
| 105 |
+
job_id = response.json()["jobId"]
|
| 106 |
+
assert job_id is not None
|
| 107 |
+
assert len(job_id) > 0
|
| 108 |
+
|
| 109 |
+
# Job should be retrievable via GET /api/jobs/{id}
|
| 110 |
+
status_response = client.get(f"/api/jobs/{job_id}")
|
| 111 |
+
assert status_response.status_code == 200
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
def test_returns_422_on_missing_case_id(self, client: TestClient) -> None:
|
| 114 |
"""POST /api/segment returns 422 when case_id is missing."""
|
|
|
|
| 116 |
|
| 117 |
assert response.status_code == 422
|
| 118 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
+
class TestGetJobStatus:
|
| 121 |
+
"""Tests for GET /api/jobs/{job_id} endpoint."""
|
|
|
|
|
|
|
| 122 |
|
| 123 |
+
def test_returns_pending_job_status(self, client: TestClient) -> None:
|
| 124 |
+
"""GET /api/jobs/{id} returns status for a job in the store."""
|
| 125 |
+
from stroke_deepisles_demo.api.job_store import get_job_store
|
| 126 |
+
|
| 127 |
+
# Create a job directly in the store (without running inference)
|
| 128 |
+
store = get_job_store()
|
| 129 |
+
store.create_job("pending-job", "sub-stroke0001", fast_mode=True)
|
| 130 |
+
|
| 131 |
+
# Get status
|
| 132 |
+
response = client.get("/api/jobs/pending-job")
|
| 133 |
+
|
| 134 |
+
assert response.status_code == 200
|
| 135 |
+
data = response.json()
|
| 136 |
+
assert data["jobId"] == "pending-job"
|
| 137 |
+
assert data["status"] == "pending"
|
| 138 |
+
assert "progress" in data
|
| 139 |
+
assert "progressMessage" in data
|
| 140 |
+
|
| 141 |
+
def test_returns_404_for_unknown_job(self, client: TestClient) -> None:
|
| 142 |
+
"""GET /api/jobs/{id} returns 404 for unknown job ID."""
|
| 143 |
+
response = client.get("/api/jobs/nonexistent-job-id")
|
| 144 |
+
|
| 145 |
+
assert response.status_code == 404
|
| 146 |
+
assert "not found" in response.json()["detail"].lower()
|
| 147 |
+
|
| 148 |
+
def test_completed_job_includes_result(self, client: TestClient) -> None:
|
| 149 |
+
"""GET /api/jobs/{id} includes result data when job is completed."""
|
| 150 |
+
from stroke_deepisles_demo.api.job_store import get_job_store
|
| 151 |
+
|
| 152 |
+
# Create and manually complete a job (to avoid waiting for real inference)
|
| 153 |
+
store = get_job_store()
|
| 154 |
+
store.create_job("test-job", "sub-stroke0001", fast_mode=True)
|
| 155 |
+
store.start_job("test-job")
|
| 156 |
+
store.complete_job(
|
| 157 |
+
"test-job",
|
| 158 |
+
{
|
| 159 |
+
"caseId": "sub-stroke0001",
|
| 160 |
+
"diceScore": 0.847,
|
| 161 |
+
"volumeMl": 15.32,
|
| 162 |
+
"elapsedSeconds": 12.5,
|
| 163 |
+
"dwiUrl": "http://localhost/files/test-job/sub-stroke0001/dwi.nii.gz",
|
| 164 |
+
"predictionUrl": "http://localhost/files/test-job/sub-stroke0001/pred.nii.gz",
|
| 165 |
+
},
|
| 166 |
+
)
|
| 167 |
+
|
| 168 |
+
response = client.get("/api/jobs/test-job")
|
| 169 |
+
|
| 170 |
+
assert response.status_code == 200
|
| 171 |
+
data = response.json()
|
| 172 |
+
assert data["status"] == "completed"
|
| 173 |
+
assert data["progress"] == 100
|
| 174 |
+
assert data["result"] is not None
|
| 175 |
+
assert data["result"]["caseId"] == "sub-stroke0001"
|
| 176 |
+
assert data["result"]["diceScore"] == 0.847
|
| 177 |
+
|
| 178 |
+
def test_failed_job_includes_error(self, client: TestClient) -> None:
|
| 179 |
+
"""GET /api/jobs/{id} includes error message when job failed."""
|
| 180 |
+
from stroke_deepisles_demo.api.job_store import get_job_store
|
| 181 |
+
|
| 182 |
+
# Create and manually fail a job
|
| 183 |
+
store = get_job_store()
|
| 184 |
+
store.create_job("test-job", "sub-stroke0001", fast_mode=True)
|
| 185 |
+
store.start_job("test-job")
|
| 186 |
+
store.fail_job("test-job", "GPU out of memory")
|
| 187 |
+
|
| 188 |
+
response = client.get("/api/jobs/test-job")
|
| 189 |
+
|
| 190 |
+
assert response.status_code == 200
|
| 191 |
+
data = response.json()
|
| 192 |
+
assert data["status"] == "failed"
|
| 193 |
+
assert data["error"] == "GPU out of memory"
|
|
@@ -0,0 +1,314 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Unit tests for the async job store.
|
| 2 |
+
|
| 3 |
+
Tests the JobStore class that manages background ML inference jobs.
|
| 4 |
+
Follows Uncle Bob's testing principles:
|
| 5 |
+
- Test behavior, not implementation
|
| 6 |
+
- Each test verifies one thing
|
| 7 |
+
- Tests are independent and repeatable
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
from collections.abc import Generator
|
| 11 |
+
from datetime import datetime, timedelta
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
from tempfile import TemporaryDirectory
|
| 14 |
+
from unittest.mock import patch
|
| 15 |
+
|
| 16 |
+
import pytest
|
| 17 |
+
|
| 18 |
+
from stroke_deepisles_demo.api.job_store import (
|
| 19 |
+
Job,
|
| 20 |
+
JobStatus,
|
| 21 |
+
JobStore,
|
| 22 |
+
get_job_store,
|
| 23 |
+
init_job_store,
|
| 24 |
+
)
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
class TestJob:
|
| 28 |
+
"""Tests for the Job dataclass."""
|
| 29 |
+
|
| 30 |
+
def test_new_job_has_zero_elapsed_seconds(self) -> None:
|
| 31 |
+
"""A job that hasn't started should report 0 elapsed seconds."""
|
| 32 |
+
job = Job(
|
| 33 |
+
id="abc123",
|
| 34 |
+
status=JobStatus.PENDING,
|
| 35 |
+
case_id="sub-stroke0001",
|
| 36 |
+
fast_mode=True,
|
| 37 |
+
created_at=datetime.now(),
|
| 38 |
+
)
|
| 39 |
+
|
| 40 |
+
assert job.elapsed_seconds == 0.0
|
| 41 |
+
|
| 42 |
+
def test_running_job_tracks_elapsed_time(self) -> None:
|
| 43 |
+
"""A running job should report elapsed time since start."""
|
| 44 |
+
start = datetime.now() - timedelta(seconds=10)
|
| 45 |
+
job = Job(
|
| 46 |
+
id="abc123",
|
| 47 |
+
status=JobStatus.RUNNING,
|
| 48 |
+
case_id="sub-stroke0001",
|
| 49 |
+
fast_mode=True,
|
| 50 |
+
created_at=start - timedelta(seconds=1),
|
| 51 |
+
started_at=start,
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
# Should be approximately 10 seconds (with some tolerance)
|
| 55 |
+
assert 9.5 <= job.elapsed_seconds <= 11.0
|
| 56 |
+
|
| 57 |
+
def test_completed_job_has_fixed_elapsed_time(self) -> None:
|
| 58 |
+
"""A completed job should report time from start to completion."""
|
| 59 |
+
start = datetime.now() - timedelta(seconds=30)
|
| 60 |
+
end = start + timedelta(seconds=15)
|
| 61 |
+
job = Job(
|
| 62 |
+
id="abc123",
|
| 63 |
+
status=JobStatus.COMPLETED,
|
| 64 |
+
case_id="sub-stroke0001",
|
| 65 |
+
fast_mode=True,
|
| 66 |
+
created_at=start - timedelta(seconds=1),
|
| 67 |
+
started_at=start,
|
| 68 |
+
completed_at=end,
|
| 69 |
+
)
|
| 70 |
+
|
| 71 |
+
# Should be exactly 15 seconds (completed job doesn't change)
|
| 72 |
+
assert job.elapsed_seconds == 15.0
|
| 73 |
+
|
| 74 |
+
def test_to_dict_includes_required_fields(self) -> None:
|
| 75 |
+
"""Job.to_dict() should include all fields needed by the API."""
|
| 76 |
+
job = Job(
|
| 77 |
+
id="abc123",
|
| 78 |
+
status=JobStatus.RUNNING,
|
| 79 |
+
case_id="sub-stroke0001",
|
| 80 |
+
fast_mode=True,
|
| 81 |
+
created_at=datetime.now(),
|
| 82 |
+
started_at=datetime.now(),
|
| 83 |
+
progress=50,
|
| 84 |
+
progress_message="Processing...",
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
data = job.to_dict()
|
| 88 |
+
|
| 89 |
+
assert data["jobId"] == "abc123"
|
| 90 |
+
assert data["status"] == "running"
|
| 91 |
+
assert data["progress"] == 50
|
| 92 |
+
assert data["progressMessage"] == "Processing..."
|
| 93 |
+
assert "elapsedSeconds" in data
|
| 94 |
+
|
| 95 |
+
def test_to_dict_includes_result_when_completed(self) -> None:
|
| 96 |
+
"""Completed jobs should include result data in to_dict()."""
|
| 97 |
+
job = Job(
|
| 98 |
+
id="abc123",
|
| 99 |
+
status=JobStatus.COMPLETED,
|
| 100 |
+
case_id="sub-stroke0001",
|
| 101 |
+
fast_mode=True,
|
| 102 |
+
created_at=datetime.now(),
|
| 103 |
+
started_at=datetime.now(),
|
| 104 |
+
completed_at=datetime.now(),
|
| 105 |
+
result={"caseId": "sub-stroke0001", "diceScore": 0.847},
|
| 106 |
+
)
|
| 107 |
+
|
| 108 |
+
data = job.to_dict()
|
| 109 |
+
|
| 110 |
+
assert "result" in data
|
| 111 |
+
assert data["result"]["diceScore"] == 0.847
|
| 112 |
+
|
| 113 |
+
def test_to_dict_includes_error_when_failed(self) -> None:
|
| 114 |
+
"""Failed jobs should include error message in to_dict()."""
|
| 115 |
+
job = Job(
|
| 116 |
+
id="abc123",
|
| 117 |
+
status=JobStatus.FAILED,
|
| 118 |
+
case_id="sub-stroke0001",
|
| 119 |
+
fast_mode=True,
|
| 120 |
+
created_at=datetime.now(),
|
| 121 |
+
error="GPU out of memory",
|
| 122 |
+
)
|
| 123 |
+
|
| 124 |
+
data = job.to_dict()
|
| 125 |
+
|
| 126 |
+
assert "error" in data
|
| 127 |
+
assert data["error"] == "GPU out of memory"
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
class TestJobStore:
|
| 131 |
+
"""Tests for the JobStore class."""
|
| 132 |
+
|
| 133 |
+
@pytest.fixture
|
| 134 |
+
def store(self) -> Generator[JobStore, None, None]:
|
| 135 |
+
"""Create a fresh JobStore for each test."""
|
| 136 |
+
with TemporaryDirectory() as tmpdir:
|
| 137 |
+
yield JobStore(results_dir=Path(tmpdir))
|
| 138 |
+
|
| 139 |
+
def test_create_job_returns_pending_job(self, store: JobStore) -> None:
|
| 140 |
+
"""Creating a job should return a job in PENDING status."""
|
| 141 |
+
job = store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 142 |
+
|
| 143 |
+
assert job.id == "job-1"
|
| 144 |
+
assert job.status == JobStatus.PENDING
|
| 145 |
+
assert job.case_id == "sub-stroke0001"
|
| 146 |
+
assert job.fast_mode is True
|
| 147 |
+
|
| 148 |
+
def test_get_job_returns_created_job(self, store: JobStore) -> None:
|
| 149 |
+
"""get_job() should return a previously created job."""
|
| 150 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 151 |
+
|
| 152 |
+
job = store.get_job("job-1")
|
| 153 |
+
|
| 154 |
+
assert job is not None
|
| 155 |
+
assert job.id == "job-1"
|
| 156 |
+
|
| 157 |
+
def test_get_job_returns_none_for_unknown_id(self, store: JobStore) -> None:
|
| 158 |
+
"""get_job() should return None for unknown job IDs."""
|
| 159 |
+
job = store.get_job("nonexistent")
|
| 160 |
+
|
| 161 |
+
assert job is None
|
| 162 |
+
|
| 163 |
+
def test_start_job_changes_status_to_running(self, store: JobStore) -> None:
|
| 164 |
+
"""start_job() should update job status to RUNNING."""
|
| 165 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 166 |
+
|
| 167 |
+
store.start_job("job-1")
|
| 168 |
+
|
| 169 |
+
job = store.get_job("job-1")
|
| 170 |
+
assert job is not None
|
| 171 |
+
assert job.status == JobStatus.RUNNING
|
| 172 |
+
assert job.started_at is not None
|
| 173 |
+
|
| 174 |
+
def test_update_progress_changes_progress_fields(self, store: JobStore) -> None:
|
| 175 |
+
"""update_progress() should update progress and message."""
|
| 176 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 177 |
+
store.start_job("job-1")
|
| 178 |
+
|
| 179 |
+
store.update_progress("job-1", 75, "Computing metrics...")
|
| 180 |
+
|
| 181 |
+
job = store.get_job("job-1")
|
| 182 |
+
assert job is not None
|
| 183 |
+
assert job.progress == 75
|
| 184 |
+
assert job.progress_message == "Computing metrics..."
|
| 185 |
+
|
| 186 |
+
def test_update_progress_clamps_to_valid_range(self, store: JobStore) -> None:
|
| 187 |
+
"""update_progress() should clamp progress to 0-100."""
|
| 188 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 189 |
+
store.start_job("job-1")
|
| 190 |
+
|
| 191 |
+
store.update_progress("job-1", 150, "Over 100")
|
| 192 |
+
job = store.get_job("job-1")
|
| 193 |
+
assert job is not None
|
| 194 |
+
assert job.progress == 100
|
| 195 |
+
|
| 196 |
+
store.update_progress("job-1", -10, "Negative")
|
| 197 |
+
job = store.get_job("job-1")
|
| 198 |
+
assert job is not None
|
| 199 |
+
assert job.progress == 0
|
| 200 |
+
|
| 201 |
+
def test_complete_job_sets_status_and_result(self, store: JobStore) -> None:
|
| 202 |
+
"""complete_job() should mark job as completed with result."""
|
| 203 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 204 |
+
store.start_job("job-1")
|
| 205 |
+
|
| 206 |
+
result = {"caseId": "sub-stroke0001", "diceScore": 0.847}
|
| 207 |
+
store.complete_job("job-1", result)
|
| 208 |
+
|
| 209 |
+
job = store.get_job("job-1")
|
| 210 |
+
assert job is not None
|
| 211 |
+
assert job.status == JobStatus.COMPLETED
|
| 212 |
+
assert job.progress == 100
|
| 213 |
+
assert job.result == result
|
| 214 |
+
assert job.completed_at is not None
|
| 215 |
+
|
| 216 |
+
def test_fail_job_sets_status_and_error(self, store: JobStore) -> None:
|
| 217 |
+
"""fail_job() should mark job as failed with error message."""
|
| 218 |
+
store.create_job("job-1", "sub-stroke0001", fast_mode=True)
|
| 219 |
+
store.start_job("job-1")
|
| 220 |
+
|
| 221 |
+
store.fail_job("job-1", "GPU out of memory")
|
| 222 |
+
|
| 223 |
+
job = store.get_job("job-1")
|
| 224 |
+
assert job is not None
|
| 225 |
+
assert job.status == JobStatus.FAILED
|
| 226 |
+
assert job.error == "GPU out of memory"
|
| 227 |
+
assert job.completed_at is not None
|
| 228 |
+
|
| 229 |
+
def test_len_returns_number_of_jobs(self, store: JobStore) -> None:
|
| 230 |
+
"""len(store) should return the number of jobs."""
|
| 231 |
+
assert len(store) == 0
|
| 232 |
+
|
| 233 |
+
store.create_job("job-1", "case1", fast_mode=True)
|
| 234 |
+
assert len(store) == 1
|
| 235 |
+
|
| 236 |
+
store.create_job("job-2", "case2", fast_mode=True)
|
| 237 |
+
assert len(store) == 2
|
| 238 |
+
|
| 239 |
+
|
| 240 |
+
class TestJobStoreCleanup:
|
| 241 |
+
"""Tests for job cleanup functionality."""
|
| 242 |
+
|
| 243 |
+
def test_cleanup_removes_old_completed_jobs(self) -> None:
|
| 244 |
+
"""cleanup_old_jobs() should remove jobs older than TTL."""
|
| 245 |
+
with TemporaryDirectory() as tmpdir:
|
| 246 |
+
# Use a very short TTL for testing
|
| 247 |
+
store = JobStore(ttl=timedelta(seconds=0), results_dir=Path(tmpdir))
|
| 248 |
+
|
| 249 |
+
store.create_job("job-1", "case1", fast_mode=True)
|
| 250 |
+
store.start_job("job-1")
|
| 251 |
+
store.complete_job("job-1", {"result": "data"})
|
| 252 |
+
|
| 253 |
+
# Job is "old" immediately (TTL=0)
|
| 254 |
+
cleaned = store.cleanup_old_jobs()
|
| 255 |
+
|
| 256 |
+
assert cleaned == 1
|
| 257 |
+
assert store.get_job("job-1") is None
|
| 258 |
+
|
| 259 |
+
def test_cleanup_keeps_running_jobs(self) -> None:
|
| 260 |
+
"""cleanup_old_jobs() should not remove running jobs."""
|
| 261 |
+
with TemporaryDirectory() as tmpdir:
|
| 262 |
+
store = JobStore(ttl=timedelta(seconds=0), results_dir=Path(tmpdir))
|
| 263 |
+
|
| 264 |
+
store.create_job("job-1", "case1", fast_mode=True)
|
| 265 |
+
store.start_job("job-1")
|
| 266 |
+
# Job is running, not completed
|
| 267 |
+
|
| 268 |
+
cleaned = store.cleanup_old_jobs()
|
| 269 |
+
|
| 270 |
+
assert cleaned == 0
|
| 271 |
+
assert store.get_job("job-1") is not None
|
| 272 |
+
|
| 273 |
+
def test_cleanup_removes_result_files(self) -> None:
|
| 274 |
+
"""cleanup_old_jobs() should also remove result files on disk."""
|
| 275 |
+
with TemporaryDirectory() as tmpdir:
|
| 276 |
+
results_dir = Path(tmpdir)
|
| 277 |
+
store = JobStore(ttl=timedelta(seconds=0), results_dir=results_dir)
|
| 278 |
+
|
| 279 |
+
# Create job and its result directory
|
| 280 |
+
store.create_job("job-1", "case1", fast_mode=True)
|
| 281 |
+
store.start_job("job-1")
|
| 282 |
+
job_results = results_dir / "job-1"
|
| 283 |
+
job_results.mkdir()
|
| 284 |
+
(job_results / "prediction.nii.gz").touch()
|
| 285 |
+
store.complete_job("job-1", {"result": "data"})
|
| 286 |
+
|
| 287 |
+
# Cleanup should remove both job record and files
|
| 288 |
+
store.cleanup_old_jobs()
|
| 289 |
+
|
| 290 |
+
assert not job_results.exists()
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
class TestGlobalJobStore:
|
| 294 |
+
"""Tests for the global job store singleton."""
|
| 295 |
+
|
| 296 |
+
def test_get_job_store_raises_before_init(self) -> None:
|
| 297 |
+
"""get_job_store() should raise if not initialized."""
|
| 298 |
+
# Patch the global to simulate uninitialized state
|
| 299 |
+
with (
|
| 300 |
+
patch("stroke_deepisles_demo.api.job_store.job_store", None),
|
| 301 |
+
pytest.raises(RuntimeError, match="not initialized"),
|
| 302 |
+
):
|
| 303 |
+
get_job_store()
|
| 304 |
+
|
| 305 |
+
def test_init_job_store_creates_global_instance(self) -> None:
|
| 306 |
+
"""init_job_store() should create and return a JobStore."""
|
| 307 |
+
with TemporaryDirectory() as tmpdir:
|
| 308 |
+
store = init_job_store(results_dir=Path(tmpdir))
|
| 309 |
+
|
| 310 |
+
assert store is not None
|
| 311 |
+
assert isinstance(store, JobStore)
|
| 312 |
+
|
| 313 |
+
# Clean up the scheduler
|
| 314 |
+
store.stop_cleanup_scheduler()
|