Claude commited on
Commit
54d6b50
·
unverified ·
1 Parent(s): b661ed7

docs: add comprehensive CLAUDE.md AI assistant guide

Browse files

Created a detailed guide for AI assistants working on the Mosaic codebase,
including:
- Complete repository structure and organization
- Development workflows and Makefile targets
- Code architecture and key design patterns
- Testing conventions and CI/CD pipelines
- Common tasks with step-by-step instructions
- Environment setup and configuration
- Git workflow and contribution guidelines
- Quick reference cards and troubleshooting

This document provides AI assistants with all necessary context to
effectively understand and work with the codebase.

Files changed (1) hide show
  1. CLAUDE.md +1152 -0
CLAUDE.md ADDED
@@ -0,0 +1,1152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CLAUDE.md - AI Assistant Guide for Mosaic
2
+
3
+ **Last Updated**: 2026-01-21
4
+ **Repository**: Mosaic - H&E Whole Slide Image Cancer Subtype and Biomarker Inference
5
+
6
+ This document provides AI assistants with comprehensive information about the Mosaic codebase structure, development workflows, and key conventions.
7
+
8
+ ---
9
+
10
+ ## Table of Contents
11
+
12
+ - [Project Overview](#project-overview)
13
+ - [Repository Structure](#repository-structure)
14
+ - [Key Technologies](#key-technologies)
15
+ - [Development Workflows](#development-workflows)
16
+ - [Code Organization](#code-organization)
17
+ - [Testing Conventions](#testing-conventions)
18
+ - [CI/CD Pipelines](#cicd-pipelines)
19
+ - [Common Tasks](#common-tasks)
20
+ - [Important Patterns](#important-patterns)
21
+ - [Environment Setup](#environment-setup)
22
+ - [Git Workflow](#git-workflow)
23
+ - [Critical Files Reference](#critical-files-reference)
24
+
25
+ ---
26
+
27
+ ## Project Overview
28
+
29
+ **Mosaic** is a production-grade deep learning pipeline for analyzing H&E whole slide images (WSIs) to predict:
30
+
31
+ 1. **Cancer Subtypes** using the Aeon model (100+ cancer types)
32
+ 2. **Biomarkers** using the Paladin model suite (cancer subtype-specific)
33
+
34
+ ### Key Features
35
+
36
+ - **Multi-interface**: Web UI (Gradio), CLI (single/batch), programmatic API
37
+ - **Hardware-aware**: Auto-detects ZeroGPU (H100), T4, or standard GPUs
38
+ - **Batch optimization**: Models loaded once and reused across slides (1.25x-1.45x speedup)
39
+ - **Production-ready**: Docker deployment, HuggingFace Spaces support, comprehensive testing
40
+
41
+ ### Technology Stack
42
+
43
+ - **Language**: Python 3.10-3.11
44
+ - **Package Manager**: `uv` (Astral)
45
+ - **Web Framework**: Gradio 6.0+
46
+ - **Deep Learning**: PyTorch 2.0+, Lightning 2.6+
47
+ - **WSI Processing**: OpenSlide, Mussel (tissue segmentation & feature extraction)
48
+ - **Models**: CTransPath, Optimus (feature extraction), Aeon (subtyping), Paladin (biomarkers)
49
+
50
+ ---
51
+
52
+ ## Repository Structure
53
+
54
+ ```
55
+ mosaic/
56
+ ├── src/mosaic/ # Main application code
57
+ │ ├── gradio_app.py # CLI entry point (230 lines)
58
+ │ ├── analysis.py # Core slide analysis pipeline (500+ lines)
59
+ │ ├── model_manager.py # Model caching & memory management (286 lines)
60
+ │ ├── hardware.py # GPU detection & optimization (85 lines)
61
+ │ ├── data_directory.py # Data path management
62
+ │ ├── inference/ # Model inference modules
63
+ │ │ ├── aeon.py # Cancer subtype prediction
64
+ │ │ ├── paladin.py # Biomarker prediction
65
+ │ │ └── data.py # Inference data structures
66
+ │ └── ui/ # Web interface modules
67
+ │ ├── app.py # Gradio interface (474 lines)
68
+ │ └── utils.py # UI utilities (OncoTree, CSV validation)
69
+
70
+ ├── tests/ # Test suite (pytest)
71
+ │ ├── conftest.py # Pytest configuration & mocks
72
+ │ ├── test_cli.py # CLI argument parsing tests
73
+ │ ├── test_gradio_app.py # Gradio interface tests
74
+ │ ├── test_model_manager.py # Model caching tests
75
+ │ ├── test_ui_*.py # UI component tests
76
+ │ ├── test_regression_*.py # End-to-end regression tests
77
+ │ ├── benchmark_*.py # Performance benchmarks
78
+ │ └── inference/ # Inference module tests
79
+
80
+ ├── scripts/ # Utility scripts
81
+ │ ├── export_aeon_checkpoint.py
82
+ │ └── verify_aeon_results.py
83
+
84
+ ├── data/ # Model metadata & mappings
85
+ │ ├── paladin_model_map.csv # Cancer subtype → model mapping
86
+ │ ├── sex_original_to_idx.csv
87
+ │ ├── tissue_site_original_to_idx.csv
88
+ │ └── metadata/ # Model-specific metadata
89
+
90
+ ├── .github/workflows/ # CI/CD automation
91
+ │ ├── tests.yml # Test pipeline (Python 3.10, 3.11)
92
+ │ ├── code-quality.yml # Black & Pylint checks
93
+ │ └── docker.yml # Docker build & push to GHCR/Docker Hub
94
+
95
+ ├── pyproject.toml # Project configuration (uv/pip)
96
+ ├── Makefile # Development commands (264 lines, 40+ targets)
97
+ ├── Dockerfile # Container definition (multi-stage build)
98
+ ├── app.py # HuggingFace Spaces entry point
99
+ ├── README.md # User documentation (430 lines)
100
+ ├── ARCHITECTURE.md # Code organization guide
101
+ ├── CONTRIBUTING.md # Developer guide
102
+ └── BATCH_PROCESSING_IMPLEMENTATION.md # Batch optimization details
103
+ ```
104
+
105
+ ### Source Code Organization
106
+
107
+ **Module Hierarchy:**
108
+ ```
109
+ mosaic.gradio_app:main()
110
+ ├── download_and_process_models()
111
+ ├── analyze_slide() [from analysis]
112
+ │ ├── segment_tissue() [from mussel]
113
+ │ ├── get_features() [CTransPath, Optimus]
114
+ │ ├── filter_features() [Marker Classifier]
115
+ │ ├── run_aeon() [from inference]
116
+ │ └── run_paladin() [from inference]
117
+ └── launch_gradio() [from ui]
118
+ ├── analyze_slides() [batch coordinator]
119
+ └── validate_settings() [from ui.utils]
120
+ ```
121
+
122
+ ---
123
+
124
+ ## Key Technologies
125
+
126
+ ### Core Dependencies
127
+
128
+ | Package | Version | Purpose |
129
+ |---------|---------|---------|
130
+ | `gradio` | >=6.0.0 | Web UI framework |
131
+ | `torch` | >=2.0.0 | Deep learning backend |
132
+ | `lightning` | >=2.6.0 | PyTorch trainer |
133
+ | `mussel[torch-gpu]` | git | Tissue segmentation & feature extraction |
134
+ | `paladin` | git (private) | Biomarker prediction models |
135
+ | `openslide-python` | - | Whole slide image reading |
136
+ | `huggingface-hub` | - | Model downloading |
137
+ | `loguru` | >=0.7.3 | Enhanced logging |
138
+
139
+ ### Development Dependencies
140
+
141
+ | Package | Purpose |
142
+ |---------|---------|
143
+ | `black` | Code formatting (PEP 8) |
144
+ | `pylint` | Linting |
145
+ | `pytest` | Test framework |
146
+ | `pytest-cov` | Coverage reporting |
147
+ | `pytest-mock` | Test mocking |
148
+
149
+ ### Private Dependencies
150
+
151
+ **Important**: This project depends on two private repositories:
152
+
153
+ 1. **Paladin** (`ssh://git@github.com/pathology-data-mining/paladin.git@dev`)
154
+ - Biomarker prediction models
155
+ - Requires SSH key or GH_TOKEN
156
+
157
+ 2. **Mussel** (`https://github.com/pathology-data-mining/Mussel.git@mosaic-dev`)
158
+ - Tissue segmentation and feature extraction
159
+ - Public repository
160
+
161
+ **Installation Methods:**
162
+ ```bash
163
+ # Local dev (SSH)
164
+ uv sync
165
+
166
+ # Docker build (token-based)
167
+ GH_TOKEN=<token> docker build --secret id=GH_TOKEN .
168
+ ```
169
+
170
+ ---
171
+
172
+ ## Development Workflows
173
+
174
+ ### Makefile Targets (40+ commands)
175
+
176
+ The Makefile is the primary interface for development tasks. Key targets:
177
+
178
+ #### Installation & Setup
179
+ ```bash
180
+ make install # Production dependencies only
181
+ make install-dev # Production + development dependencies
182
+ ```
183
+
184
+ #### Testing (Most Common)
185
+ ```bash
186
+ make test # Run all tests with coverage
187
+ make test-fast # Without coverage (2-3x faster)
188
+ make test-coverage # Detailed coverage + HTML report
189
+ make test-ui # UI component tests only
190
+ make test-cli # CLI routing tests only
191
+ make test-verbose # With print statements (-s flag)
192
+ make test-specific TEST=tests/test_file.py::test_name # Single test
193
+ ```
194
+
195
+ #### Code Quality
196
+ ```bash
197
+ make format # Auto-format with black
198
+ make format-check # Check formatting (CI-safe)
199
+ make lint # Pylint on src/mosaic/
200
+ make lint-strict # Pylint on src/ + tests/
201
+ make quality # format-check + lint
202
+ ```
203
+
204
+ #### Running the Application
205
+ ```bash
206
+ make run-ui # Launch web interface
207
+ make run-ui-public # Web UI with public sharing
208
+ make run-single SLIDE=path.svs OUTPUT=dir/ # Single slide CLI
209
+ make run-batch CSV=settings.csv OUTPUT=dir/ # Batch processing CLI
210
+ ```
211
+
212
+ #### Docker Workflows
213
+ ```bash
214
+ make docker-build # Build with SSH forwarding
215
+ make docker-run # Run web UI container
216
+ make docker-shell # Interactive shell in container
217
+ make docker-push # Push to registries
218
+ make docker-prune # Clean build cache
219
+ ```
220
+
221
+ #### Development Utilities
222
+ ```bash
223
+ make shell # Python REPL
224
+ make ipython # IPython REPL
225
+ make notebook # Jupyter notebook
226
+ make check-deps # List outdated dependencies
227
+ make profile SLIDE=path.svs # Profile single slide analysis
228
+ make benchmark # Performance benchmarks
229
+ ```
230
+
231
+ #### Git Hooks
232
+ ```bash
233
+ make pre-commit-install # Install hooks (format-check + test-fast)
234
+ make pre-commit-uninstall # Remove hooks
235
+ ```
236
+
237
+ ### Quick Reference
238
+
239
+ **Most Common Development Commands:**
240
+ 1. `make install-dev` - Set up environment
241
+ 2. `make format` - Format code before committing
242
+ 3. `make test-fast` - Quick validation
243
+ 4. `make test-coverage` - Before PR submission
244
+ 5. `make run-ui` - Test web interface locally
245
+
246
+ ---
247
+
248
+ ## Code Organization
249
+
250
+ ### Architecture Principles
251
+
252
+ 1. **Separation of Concerns**: UI, analysis, and CLI logic are in separate modules
253
+ 2. **Hardware Abstraction**: `hardware.py` provides centralized GPU detection
254
+ 3. **Model Caching**: `model_manager.py` implements load-once-reuse-many pattern
255
+ 4. **Batch Optimization**: Automatic batch mode for multi-slide processing
256
+ 5. **Backward Compatibility**: Original functions unchanged, batch functions parallel
257
+
258
+ ### Key Modules
259
+
260
+ #### `src/mosaic/hardware.py` - Hardware Detection
261
+ **Purpose**: Centralized GPU detection and configuration
262
+
263
+ **Exports:**
264
+ - `IS_ZEROGPU` - True if running on HuggingFace ZeroGPU (H100)
265
+ - `IS_T4_GPU` - True if running on NVIDIA T4
266
+ - `GPU_TYPE` - Human-readable GPU name
267
+ - `DEFAULT_BATCH_SIZE` - Hardware-appropriate batch size
268
+ - `DEFAULT_NUM_WORKERS` - Hardware-appropriate worker count
269
+ - `spaces` - Decorator for ZeroGPU allocation (no-op if not available)
270
+
271
+ **Pattern:**
272
+ ```python
273
+ from mosaic.hardware import IS_T4_GPU, DEFAULT_BATCH_SIZE, spaces
274
+
275
+ if IS_T4_GPU:
276
+ # Use aggressive memory management
277
+ batch_size = DEFAULT_BATCH_SIZE
278
+ ```
279
+
280
+ #### `src/mosaic/model_manager.py` - Model Lifecycle
281
+ **Purpose**: Pre-load and cache models for batch processing
282
+
283
+ **Key Class: `ModelCache`**
284
+ ```python
285
+ class ModelCache:
286
+ def load_all_models(self) -> None:
287
+ """Load core models once (CTransPath, Optimus, Aeon, Marker Classifier)"""
288
+
289
+ def load_paladin_model_for_inference(self, cancer_subtype: str):
290
+ """Lazy-load Paladin models (with T4-aware cleanup)"""
291
+
292
+ def cleanup(self) -> None:
293
+ """Release all GPU memory"""
294
+ ```
295
+
296
+ **Memory Management:**
297
+ - **T4 GPU**: Aggressive cleanup (Paladin models loaded/deleted per inference)
298
+ - **A100 GPU**: Caching strategy (Paladin models cached for reuse)
299
+
300
+ #### `src/mosaic/analysis.py` - Core Pipeline
301
+ **Purpose**: Orchestrate slide analysis workflow
302
+
303
+ **Main Functions:**
304
+ - `analyze_slide()` - Single slide (original, backward compatible)
305
+ - `analyze_slide_with_models()` - Batch-aware (uses pre-loaded models)
306
+
307
+ **Data Flow:**
308
+ ```
309
+ WSI → Tissue Segmentation (Mussel) → CTransPath Features →
310
+ Marker Classification → Optimus Features → Aeon (Subtypes) → Paladin (Biomarkers)
311
+ ```
312
+
313
+ #### `src/mosaic/ui/app.py` - Web Interface
314
+ **Purpose**: Gradio web UI
315
+
316
+ **Key Functions:**
317
+ - `launch_gradio()` - Main entry point
318
+ - `analyze_slides()` - Event handler (auto-detects batch vs single)
319
+ - `set_cancer_subtype_maps()` - Global cancer subtype mapping management
320
+
321
+ **Automatic Batch Detection:**
322
+ ```python
323
+ if len(slides) > 1:
324
+ # Use batch mode (model caching)
325
+ else:
326
+ # Use single slide mode (original pipeline)
327
+ ```
328
+
329
+ #### `src/mosaic/ui/utils.py` - UI Utilities
330
+ **Key Functions:**
331
+ - `validate_settings()` - CSV settings validation
332
+ - `load_settings()` - Parse CSV configuration
333
+ - `get_oncotree_code_name()` - OncoTree API integration (with caching)
334
+ - `create_user_directory()` - Session directory management
335
+
336
+ #### `src/mosaic/inference/` - Model Inference
337
+
338
+ **`aeon.py` - Cancer Subtype Prediction:**
339
+ - `run()` - Original function (loads model each time)
340
+ - `run_with_model()` - Batch-aware (uses pre-loaded model)
341
+
342
+ **`paladin.py` - Biomarker Prediction:**
343
+ - `run()` - Original function
344
+ - `run_with_models()` - Batch-aware function
345
+
346
+ **`data.py` - Data Structures:**
347
+ - `SiteType` enum (Primary/Metastatic)
348
+ - `TileFeatureTensorDataset` (PyTorch DataLoader)
349
+ - `CANCER_TYPE_TO_INT_MAP` (100+ cancer types)
350
+
351
+ ---
352
+
353
+ ## Testing Conventions
354
+
355
+ ### Test Organization
356
+
357
+ **Framework**: pytest with coverage reporting
358
+
359
+ **Configuration** (`pyproject.toml`):
360
+ ```toml
361
+ [tool.pytest.ini_options]
362
+ testpaths = ["tests"]
363
+ addopts = "-v --cov=src/mosaic --cov-report=term-missing"
364
+ ```
365
+
366
+ ### Test Categories
367
+
368
+ | Category | Files | Purpose |
369
+ |----------|-------|---------|
370
+ | **Unit Tests** | `test_model_manager.py`, `test_data.py` | Component isolation |
371
+ | **Integration Tests** | `test_cli.py`, `test_ui_*.py` | Multi-component workflows |
372
+ | **Regression Tests** | `test_regression_single_slide.py` | Backward compatibility |
373
+ | **Benchmarks** | `benchmark_batch_performance.py` | Performance validation |
374
+
375
+ ### Key Testing Patterns
376
+
377
+ #### Mocking Heavy Dependencies
378
+
379
+ **`tests/conftest.py`** provides mocks for:
380
+ - `mussel` - Tissue segmentation & feature extraction
381
+ - `huggingface_hub` - Model downloading
382
+ - `gradio` - UI components (with visibility tracking)
383
+
384
+ **Example:**
385
+ ```python
386
+ # Tests run WITHOUT downloading models or processing real slides
387
+ @pytest.fixture
388
+ def mock_mussel_segment(monkeypatch):
389
+ """Mock tissue segmentation to return fake coordinates"""
390
+ def mock_segment(*args, **kwargs):
391
+ return (fake_coords, fake_attrs)
392
+ monkeypatch.setattr("mosaic.analysis.segment_tissue", mock_segment)
393
+ ```
394
+
395
+ #### Testing Without GPU
396
+
397
+ All tests are designed to run **without GPU access**:
398
+ ```python
399
+ # Good: Mock model loading
400
+ def test_analysis(mock_aeon_model, mock_paladin_model):
401
+ result = analyze_slide(slide_path, settings)
402
+ assert result is not None
403
+
404
+ # Bad: Requires actual GPU and models
405
+ def test_analysis():
406
+ result = analyze_slide(slide_path, settings) # Will fail in CI
407
+ ```
408
+
409
+ ### Running Tests
410
+
411
+ ```bash
412
+ # All tests with coverage
413
+ make test
414
+
415
+ # Fast (no coverage)
416
+ make test-fast
417
+
418
+ # Specific test
419
+ make test-specific TEST=tests/test_cli.py::test_single_slide_mode
420
+
421
+ # With print statements
422
+ make test-verbose
423
+
424
+ # Coverage report with HTML
425
+ make test-coverage
426
+ # Open htmlcov/index.html to view
427
+ ```
428
+
429
+ ### Writing New Tests
430
+
431
+ **Template:**
432
+ ```python
433
+ """Test module for [component name]."""
434
+ import pytest
435
+ from mosaic.module import function_to_test
436
+
437
+
438
+ def test_basic_functionality(mock_dependency):
439
+ """Test [describe what is being tested]."""
440
+ # Arrange
441
+ input_data = ...
442
+
443
+ # Act
444
+ result = function_to_test(input_data)
445
+
446
+ # Assert
447
+ assert result == expected_output
448
+
449
+
450
+ def test_error_handling():
451
+ """Test error handling for invalid input."""
452
+ with pytest.raises(ValueError, match="expected error message"):
453
+ function_to_test(invalid_input)
454
+ ```
455
+
456
+ ---
457
+
458
+ ## CI/CD Pipelines
459
+
460
+ ### GitHub Actions Workflows
461
+
462
+ #### 1. **Tests** (`.github/workflows/tests.yml`)
463
+ **Triggers**: Push to main/dev, PRs, manual dispatch
464
+
465
+ **Matrix**: Python 3.10, 3.11
466
+
467
+ **Steps:**
468
+ 1. Checkout with full history
469
+ 2. Set up SSH key for private repos
470
+ 3. Install Python & uv
471
+ 4. Install dependencies (`uv sync`)
472
+ 5. Run `make test-coverage`
473
+ 6. Upload coverage to Codecov
474
+ 7. Upload HTML coverage reports (artifacts)
475
+
476
+ **Required Secrets:**
477
+ - `SSH_PRIVATE_KEY` - Access to private paladin repo
478
+ - `HF_TOKEN` - HuggingFace API access
479
+ - `CODECOV_TOKEN` - Coverage reporting (optional)
480
+
481
+ #### 2. **Code Quality** (`.github/workflows/code-quality.yml`)
482
+ **Checks:**
483
+ - `format-check` - Black formatting validation (blocking)
484
+ - `lint` - Pylint checks (informational, non-blocking)
485
+
486
+ #### 3. **Docker Build & Push** (`.github/workflows/docker.yml`)
487
+ **Triggers**: Push to main/dev, tags, PRs
488
+
489
+ **Registries:**
490
+ - **GitHub Container Registry** (ghcr.io) - All branches
491
+ - **Docker Hub** (docker.io/mskmind/mosaic) - Main branch only
492
+
493
+ **Tags:**
494
+ - `main` - Latest stable
495
+ - `dev` - Development branch
496
+ - `pr-123` - Pull request builds
497
+ - `sha-abc1234` - Git commit SHA
498
+ - `v1.2.3` - Semantic version tags
499
+
500
+ **Required Secrets:**
501
+ - `SSH_PRIVATE_KEY` or `GH_TOKEN` - Private repo access
502
+ - `DOCKER_HUB_USERNAME` - Docker Hub login (optional)
503
+ - `DOCKER_HUB_TOKEN` - Docker Hub token (optional)
504
+
505
+ **Build Process:**
506
+ ```yaml
507
+ # Uses BuildKit with SSH secret mounting
508
+ docker build --ssh default --secret id=GH_TOKEN -t mosaic .
509
+ ```
510
+
511
+ ### Pre-commit Hooks
512
+
513
+ Install local pre-commit hooks:
514
+ ```bash
515
+ make pre-commit-install
516
+ ```
517
+
518
+ **Runs on commit:**
519
+ 1. `make format-check` - Verify formatting
520
+ 2. `make test-fast` - Quick validation
521
+
522
+ ---
523
+
524
+ ## Common Tasks
525
+
526
+ ### Task 1: Add a New Feature
527
+
528
+ **Steps:**
529
+ 1. **Create feature branch**
530
+ ```bash
531
+ git checkout -b feature/your-feature-name
532
+ ```
533
+
534
+ 2. **Implement changes**
535
+ - Follow existing code organization (UI in `ui/`, analysis in `analysis.py`, etc.)
536
+ - Add type hints where appropriate
537
+ - Use `loguru` logger for logging
538
+
539
+ 3. **Write tests**
540
+ ```bash
541
+ # Create test file
542
+ touch tests/test_your_feature.py
543
+
544
+ # Follow existing test patterns
545
+ # Mock heavy dependencies (models, mussel, etc.)
546
+ ```
547
+
548
+ 4. **Format and validate**
549
+ ```bash
550
+ make format # Auto-format
551
+ make test-fast # Quick validation
552
+ make lint # Check code quality
553
+ ```
554
+
555
+ 5. **Run full test suite**
556
+ ```bash
557
+ make test-coverage
558
+ ```
559
+
560
+ 6. **Commit and push**
561
+ ```bash
562
+ git add .
563
+ git commit -m "Add [feature description]"
564
+ git push origin feature/your-feature-name
565
+ ```
566
+
567
+ ### Task 2: Fix a Bug
568
+
569
+ **Steps:**
570
+ 1. **Reproduce the bug**
571
+ ```bash
572
+ make run-ui # Or make run-single SLIDE=... OUTPUT=...
573
+ ```
574
+
575
+ 2. **Write regression test first** (TDD approach)
576
+ ```python
577
+ def test_bug_fix():
578
+ """Test that [bug] is fixed."""
579
+ result = buggy_function(problematic_input)
580
+ assert result == expected_correct_output
581
+ ```
582
+
583
+ 3. **Fix the bug**
584
+ - Modify relevant module
585
+ - Ensure test passes
586
+
587
+ 4. **Validate**
588
+ ```bash
589
+ make test-specific TEST=tests/test_bug_fix.py::test_bug_fix
590
+ make test-fast # Ensure no regressions
591
+ ```
592
+
593
+ ### Task 3: Update Dependencies
594
+
595
+ **Check for outdated packages:**
596
+ ```bash
597
+ make check-deps
598
+ ```
599
+
600
+ **Update specific package:**
601
+ ```bash
602
+ # Edit pyproject.toml to change version constraint
603
+ # Then regenerate lock file
604
+ make lock
605
+ ```
606
+
607
+ **Update all dependencies:**
608
+ ```bash
609
+ make update-deps # CAUTION: May break compatibility
610
+ make test # Validate everything still works
611
+ ```
612
+
613
+ ### Task 4: Profile Performance
614
+
615
+ **Profile single slide:**
616
+ ```bash
617
+ make profile SLIDE=test_slides/example.svs
618
+ ```
619
+
620
+ **Run benchmarks:**
621
+ ```bash
622
+ make benchmark
623
+ ```
624
+
625
+ **Custom profiling:**
626
+ ```bash
627
+ python -m cProfile -o profile.stats -m mosaic.gradio_app --slide-path slide.svs --output-dir output/
628
+ python -c "import pstats; p = pstats.Stats('profile.stats'); p.sort_stats('cumulative'); p.print_stats(30)"
629
+ ```
630
+
631
+ ### Task 5: Build and Test Docker Image
632
+
633
+ **Build locally:**
634
+ ```bash
635
+ make docker-build
636
+ ```
637
+
638
+ **Run container (web UI):**
639
+ ```bash
640
+ make docker-run
641
+ ```
642
+
643
+ **Run container (single slide):**
644
+ ```bash
645
+ make docker-run-single SLIDE=example.svs
646
+ ```
647
+
648
+ **Interactive debugging:**
649
+ ```bash
650
+ make docker-shell
651
+ # Inside container:
652
+ python -m mosaic.gradio_app --slide-path /app/data/slide.svs --output-dir /app/output
653
+ ```
654
+
655
+ ### Task 6: Download Models
656
+
657
+ **Download all models:**
658
+ ```bash
659
+ make download-models
660
+ ```
661
+
662
+ **Or via CLI:**
663
+ ```bash
664
+ mosaic --download-models-only
665
+ ```
666
+
667
+ **Models are cached to:**
668
+ - `$HF_HOME/hub/` (if HF_HOME set)
669
+ - `~/.cache/huggingface/hub/` (default)
670
+
671
+ ---
672
+
673
+ ## Important Patterns
674
+
675
+ ### Pattern 1: Hardware-Aware Batch Sizing
676
+
677
+ **Always use hardware detection for GPU operations:**
678
+
679
+ ```python
680
+ from mosaic.hardware import IS_T4_GPU, IS_ZEROGPU, DEFAULT_BATCH_SIZE
681
+
682
+ if IS_ZEROGPU:
683
+ batch_size = 128
684
+ num_workers = 0
685
+ elif IS_T4_GPU:
686
+ batch_size = 64
687
+ num_workers = 4
688
+ else:
689
+ batch_size = 64
690
+ num_workers = 8
691
+ ```
692
+
693
+ **Never hardcode batch sizes or worker counts.**
694
+
695
+ ### Pattern 2: Model Caching for Batch Processing
696
+
697
+ **Original (single slide):**
698
+ ```python
699
+ def analyze_slide(slide_path, settings):
700
+ # Load models each time
701
+ aeon_model = load_aeon_model()
702
+ # ... process ...
703
+ return results
704
+ ```
705
+
706
+ **Batch-optimized:**
707
+ ```python
708
+ def analyze_slides_batch(slides, settings):
709
+ # Load models ONCE
710
+ cache = ModelCache()
711
+ cache.load_all_models()
712
+
713
+ try:
714
+ for slide in slides:
715
+ # Reuse pre-loaded models
716
+ result = analyze_slide_with_models(slide, cache, settings)
717
+ finally:
718
+ cache.cleanup() # Always cleanup
719
+ ```
720
+
721
+ ### Pattern 3: Gradio Component Visibility
722
+
723
+ **Use update() for dynamic UI:**
724
+ ```python
725
+ import gradio as gr
726
+
727
+ def on_upload(files):
728
+ if len(files) > 1:
729
+ return gr.update(visible=True) # Show batch options
730
+ else:
731
+ return gr.update(visible=False) # Hide batch options
732
+
733
+ # In interface definition
734
+ with gr.Column(visible=False) as batch_column:
735
+ batch_options = gr.Checkbox(label="Batch Options")
736
+
737
+ upload_button.upload(on_upload, inputs=files, outputs=batch_column)
738
+ ```
739
+
740
+ ### Pattern 4: Error Handling with Logging
741
+
742
+ **Use loguru for structured logging:**
743
+ ```python
744
+ from loguru import logger
745
+
746
+ def risky_operation(input_data):
747
+ try:
748
+ logger.info(f"Starting operation with {len(input_data)} items")
749
+ result = process(input_data)
750
+ logger.success(f"Operation completed successfully")
751
+ return result
752
+ except ValueError as e:
753
+ logger.error(f"Validation failed: {e}")
754
+ raise
755
+ except Exception as e:
756
+ logger.exception(f"Unexpected error during operation: {e}")
757
+ raise
758
+ ```
759
+
760
+ **Never use print() for logging in production code.**
761
+
762
+ ### Pattern 5: Backward Compatibility
763
+
764
+ **When adding batch optimization:**
765
+
766
+ ```python
767
+ # Original function - UNCHANGED
768
+ def analyze_slide(slide_path, settings):
769
+ """Single slide analysis (original, backward compatible)."""
770
+ # ... original implementation ...
771
+
772
+ # New function - PARALLEL
773
+ def analyze_slide_with_models(slide_path, model_cache, settings):
774
+ """Single slide analysis using pre-loaded models (batch-aware)."""
775
+ # ... new implementation using model_cache ...
776
+
777
+ # Batch coordinator
778
+ def analyze_slides_batch(slides, settings):
779
+ """Batch analysis with model caching."""
780
+ cache = ModelCache()
781
+ cache.load_all_models()
782
+
783
+ try:
784
+ results = []
785
+ for slide in slides:
786
+ result = analyze_slide_with_models(slide, cache, settings)
787
+ results.append(result)
788
+ return results
789
+ finally:
790
+ cache.cleanup()
791
+ ```
792
+
793
+ **Key: Original function unchanged, new function added in parallel.**
794
+
795
+ ### Pattern 6: Configuration via Environment Variables
796
+
797
+ **Respect environment variables:**
798
+ ```python
799
+ import os
800
+
801
+ # HuggingFace token
802
+ HF_TOKEN = os.environ.get("HF_TOKEN")
803
+
804
+ # HuggingFace cache directory
805
+ HF_HOME = os.environ.get("HF_HOME", os.path.expanduser("~/.cache/huggingface"))
806
+
807
+ # Data directory
808
+ DATA_DIR = os.environ.get("MOSAIC_DATA_DIR", "./data")
809
+
810
+ # Gradio server configuration
811
+ SERVER_PORT = int(os.environ.get("GRADIO_SERVER_PORT", "7860"))
812
+ ```
813
+
814
+ ### Pattern 7: ZeroGPU Decorator
815
+
816
+ **For functions that need GPU on HuggingFace Spaces:**
817
+ ```python
818
+ from mosaic.hardware import spaces
819
+
820
+ @spaces.GPU(duration=120) # Allocate GPU for 120 seconds
821
+ def gpu_intensive_operation(data):
822
+ # This runs on GPU when available
823
+ return process_on_gpu(data)
824
+ ```
825
+
826
+ **The decorator is a no-op when not on HuggingFace Spaces.**
827
+
828
+ ---
829
+
830
+ ## Environment Setup
831
+
832
+ ### Prerequisites
833
+
834
+ - **Python**: 3.10 or 3.11 (3.12 not yet supported)
835
+ - **uv**: Package manager (`curl -LsSf https://astral.sh/uv/install.sh | sh`)
836
+ - **GPU**: NVIDIA CUDA-capable GPU (optional but recommended)
837
+ - **SSH Key**: Access to private paladin repository
838
+
839
+ ### Setup Steps
840
+
841
+ #### 1. Clone Repository
842
+ ```bash
843
+ git clone https://github.com/pathology-data-mining/mosaic.git
844
+ cd mosaic
845
+ ```
846
+
847
+ #### 2. Configure SSH Key
848
+
849
+ **For private repository access:**
850
+ ```bash
851
+ # Generate SSH key (if needed)
852
+ ssh-keygen -t ed25519 -C "your_email@example.com"
853
+
854
+ # Add to GitHub account
855
+ cat ~/.ssh/id_ed25519.pub
856
+ # Copy and add to GitHub: Settings → SSH and GPG keys
857
+ ```
858
+
859
+ #### 3. Install Dependencies
860
+ ```bash
861
+ # Development installation (recommended)
862
+ make install-dev
863
+
864
+ # Or production only
865
+ make install
866
+
867
+ # Activate virtual environment
868
+ source .venv/bin/activate
869
+ ```
870
+
871
+ #### 4. Set Environment Variables
872
+ ```bash
873
+ # Required for model access
874
+ export HF_TOKEN="your_huggingface_token_here"
875
+
876
+ # Optional: Set cache directory
877
+ export HF_HOME="/path/to/huggingface/cache"
878
+
879
+ # Optional: Set data directory
880
+ export MOSAIC_DATA_DIR="/path/to/mosaic/data"
881
+
882
+ # Optional: Set Gradio server port
883
+ export GRADIO_SERVER_PORT="7860"
884
+ ```
885
+
886
+ **Add to `~/.bashrc` or `~/.zshrc` for persistence.**
887
+
888
+ #### 5. Verify Installation
889
+ ```bash
890
+ # Check version
891
+ python -c "import mosaic; print('Mosaic installed successfully')"
892
+
893
+ # Run tests
894
+ make test-fast
895
+
896
+ # Download models (requires HF_TOKEN)
897
+ make download-models
898
+ ```
899
+
900
+ ### Docker Setup
901
+
902
+ **Build Docker image:**
903
+ ```bash
904
+ # With SSH key
905
+ make docker-build
906
+
907
+ # Or manually
908
+ ./build.sh
909
+ ```
910
+
911
+ **Run Docker container:**
912
+ ```bash
913
+ # Web UI
914
+ make docker-run
915
+
916
+ # Single slide
917
+ docker run -v $(PWD)/data:/app/data mskmind/mosaic \
918
+ --slide-path /app/data/slide.svs --output-dir /app/output
919
+ ```
920
+
921
+ ### HuggingFace Spaces Deployment
922
+
923
+ **Requirements:**
924
+ 1. Added to PDM Group on HuggingFace
925
+ 2. HuggingFace access token with read permissions
926
+
927
+ **Deployment:**
928
+ 1. Create new Space (Gradio SDK, ZeroGPU hardware)
929
+ 2. Push code to Space repository
930
+ 3. Add `HF_TOKEN` secret in Space settings
931
+ 4. App will auto-start and download models
932
+
933
+ ---
934
+
935
+ ## Git Workflow
936
+
937
+ ### Branch Strategy
938
+
939
+ - **`main`** - Production-ready code
940
+ - **`dev`** - Development branch
941
+ - **`feature/name`** - Feature branches
942
+ - **`fix/name`** - Bug fix branches
943
+
944
+ ### Commit Message Convention
945
+
946
+ **Format:**
947
+ ```
948
+ <type>: <short description>
949
+
950
+ <optional detailed explanation>
951
+
952
+ <optional footer>
953
+ ```
954
+
955
+ **Types:**
956
+ - `feat:` - New feature
957
+ - `fix:` - Bug fix
958
+ - `refactor:` - Code restructuring (no functional changes)
959
+ - `test:` - Add or update tests
960
+ - `docs:` - Documentation changes
961
+ - `style:` - Code formatting (black, etc.)
962
+ - `perf:` - Performance improvements
963
+ - `chore:` - Maintenance tasks
964
+
965
+ **Examples:**
966
+ ```
967
+ feat: add batch processing optimization for multi-slide analysis
968
+
969
+ Implemented ModelCache class to load models once and reuse across
970
+ slides in a batch. This provides 1.25x-1.45x speedup for batches.
971
+
972
+ Closes #42
973
+
974
+ ---
975
+
976
+ fix: resolve T4 GPU out-of-memory errors in Paladin inference
977
+
978
+ Added aggressive memory management for T4 GPUs that deletes Paladin
979
+ models immediately after inference instead of caching.
980
+
981
+ ---
982
+
983
+ test: add regression tests for single slide analysis
984
+
985
+ Ensures backward compatibility with original analyze_slide() function.
986
+ ```
987
+
988
+ ### Pull Request Process
989
+
990
+ 1. **Create feature branch**
991
+ ```bash
992
+ git checkout -b feature/your-feature
993
+ ```
994
+
995
+ 2. **Make changes and commit**
996
+ ```bash
997
+ git add .
998
+ git commit -m "feat: add your feature"
999
+ ```
1000
+
1001
+ 3. **Run quality checks**
1002
+ ```bash
1003
+ make format
1004
+ make test-coverage
1005
+ make lint
1006
+ ```
1007
+
1008
+ 4. **Push to remote**
1009
+ ```bash
1010
+ git push origin feature/your-feature
1011
+ ```
1012
+
1013
+ 5. **Create Pull Request**
1014
+ - Go to GitHub repository
1015
+ - Click "New Pull Request"
1016
+ - Select your branch
1017
+ - Fill in PR template:
1018
+ - **Description**: What does this PR do?
1019
+ - **Testing**: How was it tested?
1020
+ - **Related Issues**: Closes #123
1021
+
1022
+ 6. **Wait for CI checks**
1023
+ - Tests must pass (Python 3.10, 3.11)
1024
+ - Code quality checks must pass
1025
+ - Docker build must succeed
1026
+
1027
+ 7. **Address review feedback**
1028
+ ```bash
1029
+ # Make changes
1030
+ git add .
1031
+ git commit -m "fix: address review feedback"
1032
+ git push origin feature/your-feature
1033
+ ```
1034
+
1035
+ 8. **Merge** (after approval)
1036
+ - Squash and merge (recommended for feature branches)
1037
+ - Regular merge (for larger changes)
1038
+
1039
+ ---
1040
+
1041
+ ## Critical Files Reference
1042
+
1043
+ ### Configuration Files
1044
+
1045
+ | File | Purpose | Key Contents |
1046
+ |------|---------|--------------|
1047
+ | `pyproject.toml` | Project configuration | Dependencies, scripts, pytest config, pylint settings |
1048
+ | `uv.lock` | Dependency lock file | Exact versions of all packages |
1049
+ | `Makefile` | Development commands | 40+ targets for testing, building, running |
1050
+ | `Dockerfile` | Container definition | Multi-stage build, SSH secret mounting |
1051
+ | `.github/workflows/*.yml` | CI/CD pipelines | Tests, code quality, Docker builds |
1052
+
1053
+ ### Documentation Files
1054
+
1055
+ | File | Purpose | Audience |
1056
+ |------|---------|----------|
1057
+ | `README.md` | User documentation | End users |
1058
+ | `ARCHITECTURE.md` | Code organization | Developers |
1059
+ | `CONTRIBUTING.md` | Contribution guide | Contributors |
1060
+ | `BATCH_PROCESSING_IMPLEMENTATION.md` | Batch optimization details | Developers |
1061
+ | `CLAUDE.md` (this file) | AI assistant guide | AI assistants |
1062
+
1063
+ ### Data Files
1064
+
1065
+ | File | Purpose |
1066
+ |------|---------|
1067
+ | `data/paladin_model_map.csv` | Cancer subtype → Paladin model mapping |
1068
+ | `data/sex_original_to_idx.csv` | Sex encoding (Male=0, Female=1) |
1069
+ | `data/tissue_site_original_to_idx.csv` | Tissue site location encoding |
1070
+ | `data/metadata/target_dict.tsv` | Model output class mappings |
1071
+ | `data/metadata/int_to_name_class_mapping.tsv` | Cancer type name mappings |
1072
+
1073
+ ### Entry Points
1074
+
1075
+ | File | CLI Command | Purpose |
1076
+ |------|-------------|---------|
1077
+ | `mosaic` script | `mosaic` | Main CLI entry point |
1078
+ | `src/mosaic/gradio_app.py:main` | `mosaic` | CLI implementation |
1079
+ | `src/mosaic/inference/aeon.py:main` | `aeon_inference` | Standalone Aeon inference |
1080
+ | `src/mosaic/inference/paladin.py:main` | `paladin_inference` | Standalone Paladin inference |
1081
+ | `app.py` | (HF Spaces) | HuggingFace Spaces entry point |
1082
+
1083
+ ---
1084
+
1085
+ ## Quick Reference Card
1086
+
1087
+ ### Most Common Commands
1088
+
1089
+ ```bash
1090
+ # Setup
1091
+ make install-dev
1092
+ export HF_TOKEN="your_token"
1093
+
1094
+ # Development
1095
+ make format # Before committing
1096
+ make test-fast # Quick validation
1097
+ make test-coverage # Before PR
1098
+
1099
+ # Running
1100
+ make run-ui # Web interface
1101
+ make run-single SLIDE=x.svs OUTPUT=out/ # CLI single
1102
+ make run-batch CSV=settings.csv OUTPUT=out/ # CLI batch
1103
+
1104
+ # Docker
1105
+ make docker-build # Build image
1106
+ make docker-run # Run container
1107
+ make docker-shell # Debug container
1108
+
1109
+ # Quality
1110
+ make format-check # CI formatting check
1111
+ make lint # Code quality
1112
+ make quality # All checks
1113
+ ```
1114
+
1115
+ ### File Location Quick Lookup
1116
+
1117
+ | What | Where |
1118
+ |------|-------|
1119
+ | CLI entry point | `src/mosaic/gradio_app.py` |
1120
+ | Core analysis | `src/mosaic/analysis.py` |
1121
+ | Web UI | `src/mosaic/ui/app.py` |
1122
+ | GPU detection | `src/mosaic/hardware.py` |
1123
+ | Model caching | `src/mosaic/model_manager.py` |
1124
+ | Cancer subtype inference | `src/mosaic/inference/aeon.py` |
1125
+ | Biomarker inference | `src/mosaic/inference/paladin.py` |
1126
+ | Test mocks | `tests/conftest.py` |
1127
+ | Main tests | `tests/test_*.py` |
1128
+
1129
+ ### Common Error Solutions
1130
+
1131
+ | Error | Solution |
1132
+ |-------|----------|
1133
+ | `ModuleNotFoundError: No module named 'paladin'` | Ensure SSH key configured, run `uv sync` |
1134
+ | `HuggingFace authentication failed` | Set `HF_TOKEN` environment variable |
1135
+ | `CUDA out of memory` | Reduce `--num-workers`, use T4-aware settings |
1136
+ | `Port 7860 already in use` | Set `GRADIO_SERVER_PORT=7861` |
1137
+ | `Tests fail with import errors` | Run `make install-dev` to install test dependencies |
1138
+ | `Docker build fails on paladin` | Ensure `GH_TOKEN` secret or SSH key is configured |
1139
+
1140
+ ---
1141
+
1142
+ ## Change Log
1143
+
1144
+ | Date | Changes |
1145
+ |------|---------|
1146
+ | 2026-01-21 | Initial CLAUDE.md creation |
1147
+
1148
+ ---
1149
+
1150
+ **End of AI Assistant Guide**
1151
+
1152
+ For questions or updates to this guide, please submit a pull request or create an issue.