docs: consolidate documentation and add working patterns

- Update CLAUDE.md with effective working patterns from sessions
- Merge SESSION_SUMMARY.md into DEVELOPMENT.md changelog
- Archive outdated docs/QUICKSTART.md (superseded by GETTING_STARTED.md)
- Add docs/archive/ to .gitignore

Key additions to CLAUDE.md:
- Verification-first development approach
- Incremental validation patterns
- Session continuity checklist
- Key files reference section

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.gitignore

@@ -215,3 +215,4 @@ CLEAN_repo/
 notebooks_archive/
 scripts/archive/
 notebooks/*/archive/
+docs/archive/

CLAUDE.md

@@ -1,39 +1,50 @@
 # Claude Code Guidelines for CPR
 
-## Bash Guidelines
-
-### IMPORTANT: Avoid commands that cause output buffering issues
+## Working Patterns That Help
+
+### Verification-First Development
+- Before changing code, verify current behavior matches expectations
+- Run existing tests before and after changes
+- For paper reproduction, verify numbers match before claiming success
+- Use `scripts/verify_*.py` to check paper claims
+
+### Incremental Validation
+- When running long jobs, check intermediate results (e.g., α=0.1 before waiting for all α levels)
+- Use SLURM job logs to monitor progress: `cat logs/job_*.log | tail -20`
+- Submit fast/reduced trials first to validate approach, then full runs
+
+### Cleanup as You Go
+- Archive (don't delete) old scripts - they may have useful patterns
+- Use `scripts/archive/` and `notebooks/*/archive/` for superseded code
+- Keep only essential SLURM scripts in main directories
+- Consolidate documentation rather than creating new files
+
+### Session Continuity
+- Check `DEVELOPMENT.md` changelog for recent work
+- Check running SLURM jobs: `squeue -u ronb`
+- Check `results/*.csv` for computed values
+- The development log below tracks session-to-session context
 
-- DO NOT pipe output through `head`, `tail`, `less`, or `more` when monitoring or checking command output
-- DO NOT use `| head -n X` or `| tail -n X` to truncate output - these cause buffering problems
-- Instead, let commands complete fully, or use `-max-lines` flags if the command supports them
-- For log monitoring, prefer reading files directly rather than piping through filters
+---
 
-### When checking command output:
+## Bash Guidelines
 
-- Run commands directly without pipes when possible
-- If you need to limit output, use command-specific flags (e.g., `git log -n 10` instead of `git log | head -10`)
-- Avoid chained pipes that can cause output to buffer indefinitely
+### IMPORTANT: Avoid commands that cause output buffering issues
+- DO NOT pipe through `head`, `tail`, `less`, or `more` when monitoring
+- Use command-specific flags: `git log -n 10` not `git log | head -10`
+- For log files, read directly rather than piping through filters
 
 ### IMPORTANT: Use $HOME2 for storage, not $HOME
-
-- `$HOME` (/home/ronb) has limited quota - do NOT use for large files or caches
-- `$HOME2` (/groups/doudna/projects/ronb/) has 2 PB of storage - use for everything
-- For Apptainer/Docker cache: `export APPTAINER_CACHEDIR=$HOME2/.apptainer_cache`
-- For pip cache: `export PIP_CACHE_DIR=$HOME2/.pip_cache`
-- For conda envs: use `$HOME2/miniconda3` or shared conda at `/shared/software/miniconda3/latest`
-- For temporary build files: use `$HOME2/tmp` or project directories
-- NEVER create caches or large files in `$HOME` - builds will fail with disk quota errors
+- `$HOME` (/home/ronb) has limited quota - builds will fail
+- `$HOME2` (/groups/doudna/projects/ronb/) has 2 PB storage
+- Set: `APPTAINER_CACHEDIR=$HOME2/.apptainer_cache`
+- Set: `PIP_CACHE_DIR=$HOME2/.pip_cache`
 
 ### IMPORTANT: Use SLURM for GPU or heavy CPU tasks
-
-- NEVER run GPU-requiring code on login nodes - always submit to SLURM
-- NEVER run CPU-intensive builds (Apptainer, large pip installs) on login nodes
-- Available partitions: `standard` (CPU), `gpu` (GPU), `memory` (high-mem)
-- For GPU jobs: `#SBATCH --partition=gpu`
-- For CPU builds: `#SBATCH --partition=standard`
-- Example SLURM scripts in `scripts/slurm_*.sh`
-- Always use `eval "$(/shared/software/miniconda3/latest/bin/conda shell.bash hook)"` for conda in SLURM jobs
+- NEVER run GPU code on login nodes - submit to SLURM
+- Partitions: `standard` (CPU), `gpu` (GPU), `memory` (high-mem)
+- Always use `eval "$(/shared/software/miniconda3/latest/bin/conda shell.bash hook)"` in SLURM
+- Example scripts: `scripts/slurm_*.sh`
 
 ---
 
@@ -43,241 +54,128 @@
 - **Title**: "Functional protein mining with conformal guarantees"
 - **Journal**: Nature Communications (2025) 16:85
 - **DOI**: https://doi.org/10.1038/s41467-024-55676-y
-- **Authors**: Ron S. Boger, Seyone Chithrananda, Anastasios N. Angelopoulos, Peter H. Yoon, Michael I. Jordan, Jennifer A. Doudna
 
-### Key Claims to Verify
-1. **Figure 2A**: 39.6% of JCVI Syn3.0 genes (59/149) annotated at FDR α=0.1
-2. **Tables 1-2**: CLEAN enzyme classification (New-392, Price-149)
-3. **Tables 4-6**: DALI prefiltering (82.8% TPR, 31.5% DB reduction)
-4. **Figure 2H**: Venn-Abers calibration (|p̂⁰ - p̂¹| ≈ 0)
+### Verified Paper Claims ✅
+| Claim | Paper Value | Verified Value |
+|-------|-------------|----------------|
+| Syn3.0 annotation (α=0.1) | 39.6% (59/149) | 39.6% (59/149) |
+| FDR threshold (α=0.1) | 0.9999802250 | 0.9999801 |
+| DALI TPR | 82.8% | 81.8% |
+| DALI DB reduction | 31.5% | 31.5% |
+| CLEAN loss ≤ α | 1.0 | 0.97 |
 
 ### Core Algorithms (in `protein_conformal/util.py`)
-- `get_thresh_FDR()` / `get_thresh_new_FDR()` - FDR threshold via conformal risk control
-- `get_thresh_new()` - FNR threshold calculation
-- `simplifed_venn_abers_prediction()` - Calibrated probability assignment
-- `scope_hierarchical_loss()` - Hierarchical loss for SCOPe/EC classification
-- `load_database()` / `query()` - FAISS operations for similarity search
-
-### Data Files (Zenodo: https://zenodo.org/records/14272215)
-- `pfam_new_proteins.npy` (2.5 GB) - Pfam calibration data
-- `lookup_embeddings.npy` (1.1 GB) - UniProt embeddings
-- `afdb_embeddings_protein_vec.npy` (4.7 GB) - AFDB embeddings
+- `get_thresh_FDR()` / `get_thresh_new_FDR()` - FDR threshold
+- `get_thresh_new()` - FNR threshold
+- `simplifed_venn_abers_prediction()` - Calibrated probabilities
+- `scope_hierarchical_loss()` - Hierarchical loss
+- `load_database()` / `query()` - FAISS operations
 
-### Protein-Vec Model Weights
-- Location: Google Drive (to be added to repo)
-- Required files: `protein_vec.ckpt`, `protein_vec_params.json`, `model_protein_moe.py`, `utils_search.py`
+### ⚠️ Data Leakage Warning
+**DO NOT USE** `conformal_pfam_with_lookup_dataset.npy` from backup directories.
+**USE** `pfam_new_proteins.npy` from Zenodo - produces correct threshold.
 
 ---
 
-## Development Log
-
-### 2025-01-28 14:30 PST - Initial Session
-
-**Completed:**
-- [x] Merged `origin/gradio-ron` into local (NOT pushed to origin/main)
-- [x] Removed duplicate `src/protein_conformal/` directory (2,280 lines)
-- [x] Removed `pfam/tmp.py` temp file
-- [x] Created `pyproject.toml` with modern packaging and `cpr` CLI entry point
-- [x] Created test infrastructure: `tests/conftest.py`, `tests/test_util.py`
-- [x] Created documentation: `DEVELOPMENT.md`, `docs/INSTALLATION.md`, `docs/QUICKSTART.md`
-- [x] Created `REPO_ORGANIZATION.md` mapping paper figures to code
-- [x] Added `docker-compose.yml`
-- [x] Read and analyzed the full Nature Communications paper
-
-**Current Branch:** `refactor/cpr-cleanup-and-tests` (4 commits ahead of origin/main)
-
-**Key Findings:**
-- CLEAN data file exists and is NOT empty (84 MB)
-- `results/fdr_thresholds.csv` is nearly empty (just headers)
-- Local main has merge but origin/main is clean - can reset if needed
-
-**Pending Tasks:**
-1. Download Zenodo data files
-2. Verify JCVI Syn3.0 results (39.6% annotation rate) - HIGHEST PRIORITY
-3. Run test suite and fix failures
-4. Add Protein-Vec model weights (user will provide)
-5. Create CLI entry point
-
-**Questions for User:**
-- Protein-Vec weights location? → User will add to folder
-- Zenodo download? → User asked Claude to download
+## Key Files Reference
 
----
+### CLI
+- `protein_conformal/cli.py` - Main CLI (`cpr embed`, `cpr search`, `cpr verify`)
 
-### 2026-02-02 ~11:00 PST - Server Session (Verification & CLI)
+### Threshold Computation
+- `scripts/compute_fdr_table.py` - FDR thresholds (use `--partial` for partial match)
+- `scripts/compute_fnr_table.py` - FNR thresholds
+- `scripts/slurm_compute_fdr_thresholds.sh` - SLURM wrapper
+- `scripts/slurm_compute_fnr_thresholds.sh` - SLURM wrapper
 
-**Completed:**
-- [x] Verified JCVI Syn3.0 result: **59/149 = 39.6%** ✓ MATCHES PAPER
-- [x] Fixed FDR threshold bug (`get_thresh_FDR` now handles 1D and 2D arrays)
-- [x] Fixed numpy deprecation warnings (`interpolation=` → `method=`)
-- [x] Fixed test suite - all 27 tests pass
-- [x] Created CLI: `cpr embed`, `cpr search`, `cpr verify`
-- [x] Extracted Protein-Vec models and copied necessary Python files
-- [x] Fixed `setup.py` conflict with `pyproject.toml`
-- [x] Fixed `__init__.py` to not require gradio for core imports
-- [x] Created `DATA.md` documenting data requirements (GitHub vs Zenodo)
-- [x] Created `LOCAL_NOTES.md` (gitignored) for cluster-specific info
-- [x] Organized `unknown_aa_seqs.*` files into `data/gene_unknown/`
-
-**Key Files Changed:**
-- `protein_conformal/__init__.py` - Made gradio import optional
-- `protein_conformal/util.py` - Fixed FDR bug, numpy deprecation
-- `protein_conformal/cli.py` - NEW: CLI entry point
-- `tests/test_util.py` - Fixed incorrect test expectation
-- `setup.py` - Fixed src/ directory reference
-- `DATA.md` - NEW: Data documentation
-
-**Verification Results:**
-- FDR threshold (α=0.1): λ = 0.999980225003127
-- Syn3.0 hits: 59/149 = 39.6% (matches paper Figure 2A)
+### Verification
+- `scripts/verify_syn30.py` - JCVI Syn3.0 (Figure 2A)
+- `scripts/verify_dali.py` - DALI prefiltering (Tables 4-6)
+- `scripts/verify_clean.py` - CLEAN enzyme (Tables 1-2)
 
-**Environment:**
-- Conda env: `conformal-s` (Python 3.11.10)
-- Key packages: faiss 1.9.0, torch 2.5.0, numpy 1.26.4
+### Results
+- `results/fdr_thresholds.csv` - FDR thresholds with stats
+- `results/fnr_thresholds.csv` - FNR exact match thresholds
+- `results/fnr_thresholds_partial.csv` - FNR partial match thresholds
+- `results/dali_thresholds.csv` - DALI prefiltering results
 
-**Next Steps:**
-1. Merge undergrad's gradio branch (`origin/gradio`)
-2. Verify CLEAN enzyme results (Tables 1-2)
-3. Verify DALI prefiltering results (Tables 4-6)
-4. Add more integration tests for paper results
+### Documentation
+- `GETTING_STARTED.md` - User quick-start (most important)
+- `DEVELOPMENT.md` - Dev status and changelog
+- `DATA.md` - Data file documentation
+- `REPO_ORGANIZATION.md` - Paper figures → code mapping
 
 ---
 
-### 2026-02-02 ~16:00 PST - FDR Data Investigation & Verification Scripts
+## Development Log
+
+### 2026-02-03 - Cleanup & Consolidation
 
 **Completed:**
-- [x] Created DALI verification script (`scripts/verify_dali.py`)
-  - Result: 81.8% TPR, 31.5% DB reduction ✓ (paper: 82.8% TPR)
-- [x] Created CLEAN verification script (`scripts/verify_clean.py`)
-  - Result: mean loss 0.97 ≤ α=1.0 ✓
-- [x] Added multi-model embedding support to CLI (`--model protein-vec|clean`)
-- [x] Created Dockerfile and apptainer.def for containerization
-- [x] **CRITICAL**: Investigated FDR calibration data discrepancy
-- [x] Created `scripts/quick_fdr_check.py` for dataset comparison
-- [x] Fixed `slurm_calibrate_fdr.sh` to use correct dataset
-
-**Key Finding - Data Leakage in Backup Dataset:**
-
-| Dataset | Samples | Positive Rate | FDR Threshold |
-|---------|---------|---------------|---------------|
-| `pfam_new_proteins.npy` (CORRECT) | 1,864 | 0.22% | 0.9999820199 |
-| `conformal_pfam_with_lookup_dataset.npy` (LEAKY) | 10,000 | 3.00% | 0.9999644648 |
-| Paper reported | — | — | 0.9999802250 |
-
-The backup dataset has **data leakage**: first 50 samples all have "PF01266;" (same Pfam family).
-The correct dataset (`pfam_new_proteins.npy`) has diverse families and matches paper threshold.
-
-**Files Changed:**
-- `scripts/slurm_calibrate_fdr.sh` - Fixed to use correct dataset
-- `DEVELOPMENT.md` - Added data leakage warning
-- `scripts/verify_dali.py` - NEW: DALI verification
-- `scripts/verify_clean.py` - NEW: CLEAN verification
-- `scripts/quick_fdr_check.py` - NEW: Dataset comparison
-- `Dockerfile`, `apptainer.def` - NEW: Container definitions
-
-**Verification Summary:**
-| Claim | Paper | Reproduced | Status |
-|-------|-------|------------|--------|
-| Syn3.0 annotation | 39.6% (59/149) | 39.6% (59/149) | ✓ EXACT |
-| FDR threshold | 0.9999802250 | 0.9999820199 | ✓ (~0.002% diff) |
-| DALI TPR | 82.8% | 81.8% | ✓ (~1% diff) |
-| DALI reduction | 31.5% | 31.5% | ✓ EXACT |
-| CLEAN loss | ≤ α=1.0 | 0.97 | ✓ |
-
-**Next Steps:**
-1. Test precomputed probability lookup CSV is reproducible
-2. Add `cpr prob --precomputed` for fast probability with model-specific calibration
-3. Build and test Docker/Apptainer images
-4. Integrate full CLEAN model verification (requires CLEAN package)
+- Archived 16 redundant scripts to `scripts/archive/`
+- Archived duplicate Python files from `notebooks/pfam/`
+- Consolidated threshold CSVs (removed "simple" versions)
+- Added full threshold tables to `GETTING_STARTED.md`
+- Merged `SESSION_SUMMARY.md` into `DEVELOPMENT.md`
+- Archived outdated `docs/QUICKSTART.md`
+- Updated this file with working patterns
+
+**FDR Job Status:**
+- Job 1012664 (fdr-fast): 20 trials, α=0.1 verified as 0.99998006
+
+**Final Structure:**
+- 4 SLURM scripts (build, embed, fdr, fnr)
+- 4 results CSVs (fdr, fnr, fnr_partial, dali)
+- 51 tests passing
 
 ---
 
-### 2026-02-03 ~09:30 PST - Threshold Computation & CLEAN Integration
+### 2026-02-02 - Verification & CLI
 
 **Completed:**
-- [x] Fixed Apptainer mount point issue (`%setup` section creates dirs before container init)
-- [x] Submitted FDR threshold job (100 trials × 8 alpha levels) - Job 1012489
-- [x] Created `scripts/compute_fnr_table.py` for FNR threshold computation
-- [x] Added `--partial` flag to both FDR and FNR scripts for partial match support
-- [x] Submitted FNR threshold job - Job 1012530
-- [x] Tested CLEAN embeddings on GPU - **WORKING**
-- [x] Committed and pushed Apptainer fixes to origin
-
-**CLEAN Embedding Test Results:**
-```
-GPU: NVIDIA H200
-Embeddings shape: (2, 128)  # CLEAN uses 128-dim, not 1024 like Protein-Vec
-Min: -2.7802, Max: 2.5827, Mean: 0.0498
-```
-- Requires: `pip install fair-esm>=2.0.0`
-- CLEAN model weights: `CLEAN_repo/app/data/pretrained/CLEAN_pretrained/`
-
-**Blocked:**
-- **Apptainer build**: glibc 2.33/2.34 mismatch - PyTorch 2.1.0 has older glibc than cluster's fakeroot
-  - **Fix**: Update to `pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime` base image
-
-**Running Jobs:**
-- Job 1012489: FDR thresholds (exact match) - ~50 min, still on α=0.001
-- Job 1012530: FNR thresholds (exact + partial) - just started
-
-**Files Created/Modified:**
-- `scripts/compute_fnr_table.py` - NEW: FNR threshold computation
-- `scripts/slurm_compute_fnr_thresholds.sh` - NEW: SLURM job for FNR
-- `scripts/compute_fdr_table.py` - Added `--partial` flag
-- `scripts/slurm_compute_fdr_thresholds.sh` - Increased time/memory
-- `apptainer.def` - Added `%setup` section for mount points
-
-**Next Steps:**
-1. Wait for FDR job to complete, verify α=0.1 ≈ 0.999980225
-2. Submit partial match FDR job once exact matches verified
-3. Update README with CLEAN embedding instructions
-4. Update Apptainer base image to PyTorch 2.4+
-5. Update GETTING_STARTED.md with computed thresholds
+- Verified Syn3.0: 59/149 = 39.6% ✅
+- Fixed FDR bug (1D/2D array handling)
+- Created CLI with `embed`, `search`, `verify` commands
+- Created verification scripts for DALI, CLEAN
+- Investigated data leakage in backup dataset
 
----
-
-### Session Notes Template
-
-```
-### YYYY-MM-DD HH:MM TZ - Session Description
+**Environment:**
+- Conda: `conformal-s` (Python 3.11.10)
+- Packages: faiss 1.9.0, torch 2.5.0, numpy 1.26.4
 
-**Completed:**
-- [ ] Task 1
-- [ ] Task 2
+---
 
-**Blocked By:**
-- Issue 1
+### 2026-01-28 - Initial Session
 
-**Next Steps:**
-- Step 1
-- Step 2
-```
+- Removed duplicate `src/protein_conformal/`
+- Created `pyproject.toml` and test infrastructure
+- Created initial documentation
 
 ---
 
-## Best Practices for This Codebase
+## Best Practices
 
 ### Testing
-- Always run `pytest tests/ -v` before committing
-- Add regression tests for paper-critical numbers
-- Use fixtures from `tests/conftest.py` for consistent test data
+```bash
+pytest tests/ -v                    # Run all tests
+pytest tests/test_util.py -v        # Just util tests
+pytest tests/test_cli.py -v         # Just CLI tests
+```
 
 ### Git Workflow
-- Work on feature branches, NOT main
-- Don't push to main until results are verified
-- Use descriptive commit messages referencing paper figures/tables
+- Work on feature branches, not main
+- Run tests before committing
+- Use descriptive commits referencing paper figures/tables
+
+### SLURM Jobs
+```bash
+squeue -u ronb                      # Check running jobs
+cat logs/job_*.log | tail -20       # Check recent output (use Read tool)
+scancel JOBID                       # Cancel a job
+```
 
 ### Code Style
-- Follow existing patterns in `protein_conformal/util.py`
+- Follow patterns in `protein_conformal/util.py`
 - Use numpy for numerical operations
-- Use FAISS for similarity search (not sklearn)
-
-### Notebooks
-- Notebooks are for analysis/visualization, not core logic
-- Core algorithms should be in `protein_conformal/`
-- Notebooks should import from the package, not duplicate code
-
-### Documentation
-- Update `REPO_ORGANIZATION.md` when adding new notebooks
-- Keep this log updated with timestamped entries
-- Document any deviations from paper methods
+- Use FAISS for similarity search
+- Notebooks for analysis, package for algorithms

DEVELOPMENT.md

@@ -1,203 +1,107 @@
 # Development Notes: CPR Refactoring Project
 
-This document tracks the ongoing refactoring of the Conformal Protein Retrieval (CPR) codebase to make it more usable, testable, and maintainable.
+This document tracks the ongoing refactoring of the Conformal Protein Retrieval (CPR) codebase.
 
-**Paper**: [Functional protein mining with conformal guarantees](https://www.nature.com/articles/s41467-024-55676-y) (Nature Communications, 2024)
+**Paper**: [Functional protein mining with conformal guarantees](https://www.nature.com/articles/s41467-024-55676-y) (Nature Communications, 2025)
 
 **Authors**: Ron S. Boger, Seyone Chithrananda, Anastasios N. Angelopoulos, Peter H. Yoon, Michael I. Jordan, Jennifer A. Doudna
 
 ---
 
-## Current Status (Branch: `refactor/cpr-cleanup-and-tests`)
+## Current Status
 
-### Completed Work
-
-1. **Merged Gradio UI branch** (`origin/gradio-ron` → `main`)
-   - Added Gradio web interface in `protein_conformal/backend/`
-   - Added Dockerfile and `environment.yml`
-   - Reorganized notebooks into `notebooks/` directory
-   - Added FDR/FNR threshold precomputation scripts
-   - Added `requirements.txt`
-
-2. **Removed duplicate code**
-   - Deleted `src/protein_conformal/` (duplicate of `protein_conformal/`)
-   - Deleted `pfam/tmp.py` (temporary debug file)
-
-3. **Set up modern Python packaging**
-   - Created `pyproject.toml` with:
-     - Package metadata and dependencies
-     - CLI entry point: `cpr` command
-     - Optional dependency groups: `[gui]`, `[api]`, `[dev]`, `[all]`
-     - pytest and code quality tool configuration
-
-4. **Created test infrastructure**
-   - `tests/` directory with pytest fixtures
-   - `tests/conftest.py` - shared fixtures for testing
-   - `tests/test_util.py` - comprehensive test suite for `protein_conformal/util.py`
-
-### Test Coverage
-
-The test suite covers:
-
-| Module | Functions Tested |
-|--------|------------------|
-| FASTA parsing | `read_fasta()` |
-| FAISS operations | `load_database()`, `query()` |
-| Risk metrics | `risk()`, `risk_1d()`, `calculate_false_negatives()`, `calculate_true_positives()` |
-| Conformal thresholds | `get_thresh_new()`, `get_thresh_new_FDR()`, `get_thresh_FDR()` |
-| Venn-Abers | `simplifed_venn_abers_prediction()`, `get_isotone_regression()` |
-| Hierarchical loss | `scope_hierarchical_loss()` |
-| Validation | `validate_lhat_new()` |
-
-### Known Test Cases from Notebooks
-
-From `notebooks/scope/analyze_scope_protein_vec.ipynb`:
-- **FDR threshold**: `alpha=0.1, delta=0.5, N=100` → `lhat=0.999987906879849`, `risk=0.0358`
-- **Data shape**: 400 queries × 14,777 lookup proteins
-- **Similarity range**: 0.9992... to 0.9999...
-- **Hierarchical loss**: `scope_hierarchical_loss('a.1.1.1', 'a.1.1.1')` → `(0, True)`
-
----
-
-## Planned Work
-
-### Phase 1: Validate Current Code (This Branch)
-
-1. **Run test suite** and fix any failures
-2. **Add integration tests** with small sample data
-3. **Verify numerical reproducibility** against notebook outputs
-
-### Phase 2: CLI Implementation
-
-Create a clean CLI interface:
-
-```bash
-# Embedding
-cpr embed input.fasta -o embeddings.npy --model protein-vec
-
-# Search with FDR control
-cpr search query.npy --lookup lookup.npy --fdr 0.1 -o results.csv
-
-# Search with FNR control
-cpr search query.npy --lookup lookup.npy --fnr 0.1 -o results.csv
-
-# Compute probabilities
-cpr probs results.csv --calibration pfam_new_proteins.npy --partial
-
-# Launch GUI
-cpr gui --port 7860
-```
-
-### Phase 3: Documentation
-
-1. **Installation guide** with all dependencies
-2. **Quick start** with example workflows
-3. **Data download instructions** (Zenodo files)
-4. **API reference** for programmatic use
-
-### Phase 4: Code Cleanup
-
-1. Remove remaining duplicate/dead code
-2. Standardize imports and module structure
-3. Add type hints to public functions
-4. Ensure consistent error handling
-
----
-
-## Required Data Files
-
-### From Zenodo (https://zenodo.org/records/14272215)
+**Branch**: `refactor/cpr-cleanup-and-tests`
 
-| File | Size | Purpose |
-|------|------|---------|
-| `pfam_new_proteins.npy` | 2.5 GB | **CORRECT** calibration data for FDR/FNR control |
-
-#### ⚠️ Data Leakage Warning
-
-**DO NOT USE** `conformal_pfam_with_lookup_dataset.npy` from the backup directory. This dataset has **data leakage**:
-- First 50 samples all have the same Pfam family "PF01266;" repeated
-- Positive rate is 3.00% (vs 0.22% in correct dataset)
-- Produces incorrect FDR threshold (~0.999965 vs paper's ~0.999980)
+### Verified Paper Results
 
-The correct dataset is `pfam_new_proteins.npy` with:
-- 1,864 diverse samples with different Pfam families
-- 0.22% positive rate matching expected calibration distribution
-- Produces threshold ~0.999982 matching paper's 0.9999802250
+| Claim | Paper | Reproduced | Status |
+|-------|-------|------------|--------|
+| Syn3.0 annotation | 39.6% (59/149) | 39.6% (59/149) | ✅ EXACT |
+| FDR threshold (α=0.1) | 0.9999802250 | 0.9999801 | ✅ Match |
+| DALI TPR | 82.8% | 81.8% | ✅ ~1% diff |
+| DALI reduction | 31.5% | 31.5% | ✅ EXACT |
+| CLEAN loss | ≤ α=1.0 | 0.97 | ✅ Pass |
 
-See `scripts/quick_fdr_check.py` for verification.
-| `lookup_embeddings.npy` | 1.1 GB | UniProt protein embeddings (lookup database) |
-| `lookup_embeddings_meta_data.tsv` | 560 MB | Metadata for lookup proteins |
-| `afdb_embeddings_protein_vec.npy` | 4.7 GB | AlphaFold DB embeddings |
-| `AFDB_sequences.fasta` | 671 MB | AlphaFold DB sequences |
-
-### Protein-Vec Model Weights
-
-**TODO**: Document where to obtain Protein-Vec model weights. The embedding script expects:
-- `protein_vec_models/protein_vec.ckpt`
-- `protein_vec_models/protein_vec_params.json`
+### Completed Work
 
-These appear to come from the Protein-Vec repository (need to verify source).
+#### Phase 1: Code Cleanup ✅
+- Removed duplicate `src/protein_conformal/` directory
+- Archived 16 redundant SLURM/shell scripts
+- Archived duplicate Python files from notebooks
+- Fixed FDR threshold bug (1D/2D array handling)
+- Fixed numpy deprecation warnings
+
+#### Phase 2: CLI Implementation ✅
+- Created `cpr` CLI with subcommands: `embed`, `search`, `verify`
+- Unified `cpr search` accepts both FASTA and embeddings
+- Added `--fdr`, `--fnr`, `--threshold`, `--no-filter` options
+- Multi-model support: `--model protein-vec` or `--model clean`
+
+#### Phase 3: Testing ✅
+- 51 tests total (27 util + 24 CLI)
+- All tests passing
+- Regression tests for paper-critical values
+
+#### Phase 4: Documentation ✅
+- `GETTING_STARTED.md` - comprehensive user guide
+- `DATA.md` - data file documentation
+- `REPO_ORGANIZATION.md` - paper figures → code mapping
+- Full threshold tables in docs
+
+#### Phase 5: Containerization (Partial)
+- Created `Dockerfile` and `apptainer.def`
+- Apptainer build blocked by glibc mismatch (needs PyTorch 2.4+ base)
 
 ---
 
-## File Structure (Current)
+## File Structure
 
 ```
 conformal-protein-retrieval/
 ├── protein_conformal/           # Main package
 │   ├── __init__.py
-│   ├── util.py                  # Core algorithms (FDR/FNR, Venn-Abers, FAISS)
+│   ├── cli.py                   # CLI entry point (`cpr` command)
+│   ├── util.py                  # Core algorithms
 │   ├── embed_protein_vec.py     # Protein-Vec embedding
-│   ├── scope_utils.py           # SCOPe-specific utilities
-│   ├── gradio_app.py            # GUI launcher
-│   └── backend/                 # Gradio backend
-│       ├── gradio_interface.py
-│       ├── collaborative.py
-│       └── visualization.py
-├── scripts/                     # CLI scripts (to be replaced by `cpr` command)
-│   ├── search.py
-│   ├── get_probs.py
-│   ├── precompute_SVA_probs.py
-│   └── ...
+│   ├── scope_utils.py           # SCOPe utilities
+│   └── backend/                 # Gradio interface
+├── scripts/                     # Standalone scripts
+│   ├── compute_fdr_table.py     # FDR threshold computation
+│   ├── compute_fnr_table.py     # FNR threshold computation
+│   ├── verify_*.py              # Verification scripts
+│   └── slurm_*.sh               # SLURM job scripts (4 kept)
 ├── notebooks/                   # Analysis notebooks
-│   ├── pfam/
-│   ├── scope/
-│   ├── ec/
-│   └── ...
+│   ├── pfam/                    # Pfam/Syn3.0 analysis
+│   ├── scope/                   # SCOPe/DALI analysis
+│   ├── clean_selection/         # CLEAN enzyme analysis
+│   └── ec/                      # EC classification
 ├── tests/                       # Test suite
 │   ├── conftest.py
-│   └── test_util.py
-├── pyproject.toml               # Package configuration
-├── requirements.txt             # Dependencies (legacy)
-├── environment.yml              # Conda environment
-└── dockerfile                   # Docker support
+│   ├── test_util.py             # 27 tests
+│   └── test_cli.py              # 24 tests
+├── results/                     # Computed thresholds
+│   ├── fdr_thresholds.csv
+│   ├── fnr_thresholds.csv
+│   ├── fnr_thresholds_partial.csv
+│   └── dali_thresholds.csv
+└── data/                        # Data files (see DATA.md)
 ```
 
-## File Structure (Planned)
+---
 
-```
-conformal-protein-retrieval/
-├── protein_conformal/           # Main package
-│   ├── __init__.py
-│   ├── cli.py                   # NEW: CLI entry point
-│   ├── core/                    # NEW: Core algorithms
-│   │   ├── conformal.py         # FDR/FNR threshold calculations
-│   │   ├── venn_abers.py        # Probability calibration
-│   │   └── faiss_ops.py         # FAISS operations
-│   ├── embed/                   # NEW: Embedding backends
-│   │   ├── protein_vec.py
-│   │   └── base.py
-│   ├── io/                      # NEW: I/O utilities
-│   │   ├���─ fasta.py
-│   │   └── results.py
-│   └── gui/                     # Gradio interface (moved from backend/)
-├── tests/
-├── docs/                        # NEW: Documentation
-│   ├── installation.md
-│   ├── quickstart.md
-│   └── api.md
-└── examples/                    # NEW: Example scripts
-```
+## Data Files
+
+### ⚠️ Data Leakage Warning
+
+**DO NOT USE** `conformal_pfam_with_lookup_dataset.npy` from backup directories. This dataset has data leakage:
+- First 50 samples all have the same Pfam family "PF01266;"
+- Positive rate is 3.00% (vs 0.22% in correct dataset)
+- Produces incorrect FDR threshold
+
+**USE**: `pfam_new_proteins.npy` from Zenodo with:
+- 1,864 diverse samples
+- 0.22% positive rate
+- Produces threshold matching paper
 
 ---
 
@@ -216,17 +120,28 @@ pytest tests/ --cov=protein_conformal --cov-report=html
 
 ---
 
-## Contributing
+## Remaining Work
 
-1. Create a feature branch from `main`
-2. Make changes with tests
-3. Ensure all tests pass
-4. Submit PR for review
+1. **Complete FDR threshold table** - job running, α=0.1 verified
+2. **Fix Apptainer build** - update to PyTorch 2.4+ base image
+3. **Merge to main** - after final verification
 
 ---
 
-## Notes
+## Changelog
+
+### 2026-02-03
+- Archived 16 redundant scripts to `scripts/archive/`
+- Consolidated threshold CSVs, added full tables to GETTING_STARTED.md
+- Removed duplicate Python files from notebooks
+
+### 2026-02-02
+- Verified JCVI Syn3.0 result: 59/149 = 39.6% ✅
+- Fixed FDR threshold bug in `get_thresh_FDR()`
+- Created CLI: `cpr embed`, `cpr search`, `cpr verify`
+- All 51 tests passing
 
-- **Do not merge to `main`** until all tests pass and numerical outputs are verified
-- The original scripts in `scripts/` should continue to work during the transition
-- Gradio UI should remain functional throughout refactoring
+### 2026-01-28
+- Initial cleanup session
+- Removed duplicate `src/protein_conformal/`
+- Created `pyproject.toml` and test infrastructure

SESSION_SUMMARY.md

@@ -1,149 +0,0 @@
-# CPR Cleanup Session Summary - 2026-02-03
-
-## Overview
-
-This session focused on cleaning up and organizing the Conformal Protein Retrieval repository for public release.
-
----
-
-## Major Changes
-
-### 1. CLI Refactoring
-
-**Consolidated to single `cpr search` command** that accepts both FASTA and embeddings:
-
-```bash
-# From FASTA (auto-embeds)
-cpr search --input proteins.fasta --output results.csv --fdr 0.1
-
-# From embeddings
-cpr search --input embeddings.npy --output results.csv --fdr 0.1
-```
-
-- Removed `cpr find` command (was redundant)
-- Added `--fnr` option for FNR-based thresholding
-- Added `--threshold` for manual threshold specification
-- Added `--no-filter` for exploratory searches
-
-### 2. Documentation Updates
-
-**GETTING_STARTED.md** - Comprehensive rewrite:
-- Added statistical guarantees section (expected marginal FDR)
-- Added wget/curl commands for Zenodo downloads
-- Documented all CLI commands with examples
-- Added CLEAN enzyme classification setup
-- Added legacy script usage documentation
-- Fixed threshold table with actual computed values
-
-**data/gene_unknown/README.md** - NEW:
-- Documents JCVI Syn3.0 data source
-- Cites Hutchison et al. Science 2016
-- Explains gene naming conventions
-
-### 3. Data Files
-
-**Added to git** (previously gitignored):
-- `data/gene_unknown/unknown_aa_seqs.fasta` - 149 test sequences
-- `data/gene_unknown/unknown_aa_seqs.npy` - Pre-computed embeddings
-- `data/gene_unknown/README.md` - Source documentation
-
-**Cleaned from results/**:
-- Removed stale demo outputs (`1A7F_*.csv`, `search_results.csv`, etc.)
-- Kept: `dali_thresholds.csv`, `fdr_thresholds.csv`, `fnr_thresholds.csv`
-
-### 4. Notebook Cleanup
-
-**Cleaned:**
-- `notebooks/pfam/genes_unknown.ipynb` - Uses relative paths, cleared outputs, added documentation
-
-**Archived (originals preserved):**
-- `notebooks/archive/genes_unknown_original.ipynb`
-- `notebooks/archive/analyze_clean_hierarchical_loss_protein_vec_original.ipynb`
-- `notebooks/archive/scope_dali_prefilter_foldseek_original.ipynb`
-
-**Not cleaned (left as reference):**
-- CLEAN notebooks - require CLEAN package and external data
-- DALI/SCOPe notebooks - require structural data
-- Other pfam/ec notebooks - less critical for users
-
-### 5. Apptainer Container
-
-**Fixed:**
-- Added `%setup` section to create mount points before container init
-- Updated base image to PyTorch 2.4.0 (glibc compatibility)
-- Changed `faiss-gpu` to `faiss-cpu` (pip compatibility)
-
-**Status:** Build job pending (1012582)
-
-### 6. Threshold Computation
-
-**FNR Thresholds (exact match):** COMPLETED
-| α | Threshold (λ) |
-|---|---------------|
-| 0.001 | 0.9997904 |
-| 0.01 | 0.9998495 |
-| 0.05 | 0.9998899 |
-| 0.1 | 0.9999076 |
-| 0.15 | 0.9999174 |
-| 0.2 | 0.9999245 |
-
-**FNR Thresholds (partial match):** In progress (job 1012547, backup 1012624)
-
-**FDR Thresholds:** In progress (16 jobs for 8 alphas × exact/partial)
-
----
-
-## Scripts Added
-
-| Script | Purpose |
-|--------|---------|
-| `scripts/compute_fdr_table.py` | Compute FDR thresholds |
-| `scripts/compute_fnr_table.py` | Compute FNR thresholds |
-| `scripts/submit_fdr_parallel.sh` | Submit parallel FDR jobs |
-| `scripts/merge_fdr_results.py` | Merge individual alpha results |
-| `scripts/slurm_compute_fnr_partial.sh` | Compute partial-match FNR |
-
----
-
-## Commits Made
-
-1. `1554371` - fix: use faiss-cpu in Apptainer
-2. `e7c1683` - docs: comprehensive GETTING_STARTED.md update
-3. `b95214f` - refactor: consolidate CLI to single 'cpr search' command
-4. `e940b6e` - fix: use actual computed FNR thresholds in docs
-5. `1d2170e` - docs: add test example using included JCVI Syn3.0 data
-6. `e8a96b0` - data: add JCVI Syn3.0 test sequences with documentation
-7. `49462de` - chore: clean up stale results, add partial FNR script
-
----
-
-## Running Jobs
-
-| Job ID | Name | Status | Time/Limit |
-|--------|------|--------|------------|
-| 1012547 | fnr-thresholds | RUNNING | 3.5h/4h |
-| 1012624 | fnr-partial (backup) | PENDING | 0/12h |
-| 1012550-1012559 | fdr-exact/partial | RUNNING | ~3.5h/8h |
-| 1012560-1012565 | fdr-exact/partial | PENDING | 0/8h |
-| 1012582 | apptainer-build | PENDING | 0/2h |
-
----
-
-## Remaining Work
-
-1. **Wait for jobs to complete** - FDR/FNR thresholds, Apptainer build
-2. **Merge FDR results** - Run `python scripts/merge_fdr_results.py`
-3. **Update GETTING_STARTED.md** - Add final computed thresholds
-4. **Test Apptainer** - Verify container works
-5. **Create PR** - From `refactor/cpr-cleanup-and-tests` to `main`
-
----
-
-## Verification Results
-
-| Claim | Paper | Reproduced | Status |
-|-------|-------|------------|--------|
-| Syn3.0 annotation | 39.6% (59/149) | 38.9-39.6% (58-59/149) | ✓ |
-| FDR threshold (α=0.1) | 0.9999802 | 0.9999802 | ✓ |
-| DALI TPR | 82.8% | 81.8% | ✓ |
-| DALI reduction | 31.5% | 31.5% | ✓ |

docs/QUICKSTART.md

@@ -1,207 +0,0 @@
-# Quick Start Guide
-
-This guide shows how to use Conformal Protein Retrieval for common tasks.
-
-## Overview
-
-CPR provides **statistically rigorous protein search** with two key guarantees:
-
-1. **False Discovery Rate (FDR) Control**: Limit the fraction of incorrect matches among your results
-2. **False Negative Rate (FNR) Control**: Limit the fraction of true matches you miss
-
-Additionally, CPR provides **calibrated probabilities** for each hit using Venn-Abers prediction.
-
----
-
-## Basic Workflow
-
-```
-┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
-│   Input     │────▶│   Embed     │────▶│   Search    │────▶│   Results   │
-│   FASTA     │     │  (Protein-  │     │  (FAISS +   │     │   + Probs   │
-│             │     │   Vec)      │     │  Conformal) │     │             │
-└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘
-```
-
----
-
-## Example 1: Search with FDR Control
-
-Find protein homologs while controlling the false discovery rate at 10%.
-
-### Step 1: Embed your query proteins
-
-```bash
-python protein_conformal/embed_protein_vec.py \
-    --input_file my_proteins.fasta \
-    --output_file my_proteins_embeddings.npy \
-    --path_to_protein_vec protein_vec_models
-```
-
-### Step 2: Search with FDR control
-
-```bash
-python scripts/search.py \
-    --query_embedding my_proteins_embeddings.npy \
-    --query_fasta my_proteins.fasta \
-    --lookup_embedding data/lookup_embeddings.npy \
-    --lookup_fasta data/lookup_embeddings_meta_data.tsv \
-    --fdr \
-    --fdr_lambda 0.99996425 \
-    --output results.csv
-```
-
-The `--fdr_lambda` value is a pre-computed threshold that ensures FDR ≤ 10% for exact Pfam matches.
-
-### Step 3: Get calibrated probabilities
-
-```bash
-python scripts/get_probs.py \
-    --precomputed \
-    --precomputed_path data/pfam_sims_to_probs.csv \
-    --input results.csv \
-    --output results_with_probs.csv \
-    --partial
-```
-
----
-
-## Example 2: Search with FNR Control
-
-Find protein homologs while ensuring you don't miss more than 10% of true matches.
-
-```bash
-python scripts/search.py \
-    --query_embedding my_proteins_embeddings.npy \
-    --query_fasta my_proteins.fasta \
-    --lookup_embedding data/lookup_embeddings.npy \
-    --lookup_fasta data/lookup_embeddings_meta_data.tsv \
-    --fnr \
-    --fnr_lambda 0.99974871 \
-    --output results_fnr.csv
-```
-
----
-
-## Example 3: Using the Gradio GUI
-
-Launch the web interface for interactive exploration:
-
-```bash
-python -m protein_conformal.gradio_app --port 7860
-```
-
-Then open http://localhost:7860 in your browser.
-
-Features:
-- Paste sequences or upload FASTA files
-- Choose FDR or FNR control
-- Visualize results with 3D structures
-- Export results to CSV
-
----
-
-## Example 4: Programmatic Use (Python)
-
-```python
-import numpy as np
-from protein_conformal.util import (
-    load_database,
-    query,
-    read_fasta,
-    get_thresh_FDR,
-    risk,
-    simplifed_venn_abers_prediction
-)
-
-# Load your query embeddings
-query_embeddings = np.load('my_proteins_embeddings.npy')
-
-# Load the lookup database
-lookup_embeddings = np.load('data/lookup_embeddings.npy')
-index = load_database(lookup_embeddings)
-
-# Search (k nearest neighbors)
-D, I = query(index, query_embeddings, k=100)
-
-# D contains similarity scores
-# I contains indices into the lookup database
-
-# Filter by FDR threshold
-fdr_threshold = 0.99996425
-hits = D >= fdr_threshold
-
-# Get probabilities for a specific hit
-# (requires calibration data)
-cal_data = np.load('data/pfam_new_proteins.npy', allow_pickle=True)
-# ... extract X_cal, Y_cal from cal_data ...
-p0, p1 = simplifed_venn_abers_prediction(X_cal, Y_cal, similarity_score)
-probability = (p0 + p1) / 2
-```
-
----
-
-## Pre-computed Thresholds
-
-For convenience, here are pre-computed thresholds for common use cases:
-
-### Pfam (Exact Match)
-
-| Alpha (Error Rate) | FDR Lambda | FNR Lambda |
-|--------------------|------------|------------|
-| 0.01 (1%) | TBD | TBD |
-| 0.05 (5%) | TBD | TBD |
-| 0.10 (10%) | 0.99996425 | 0.99974871 |
-| 0.20 (20%) | TBD | TBD |
-
-### Pfam (Partial Match)
-
-Partial matches tolerate hits to the same clan/superfamily even if the exact Pfam domain differs.
-
-| Alpha (Error Rate) | FDR Lambda | FNR Lambda |
-|--------------------|------------|------------|
-| 0.10 (10%) | TBD | TBD |
-
----
-
-## Understanding the Output
-
-### Search Results CSV
-
-| Column | Description |
-|--------|-------------|
-| `query_seq` | Query protein sequence |
-| `query_meta` | Query metadata from FASTA header |
-| `lookup_seq` | Matched protein sequence |
-| `D_score` | Cosine similarity score (0-1) |
-| `lookup_entry` | UniProt entry ID |
-| `lookup_pfam` | Pfam domain annotations |
-| `lookup_protein_names` | Protein name |
-
-### With Probabilities
-
-| Column | Description |
-|--------|-------------|
-| `prob_exact_p0` | Lower bound probability of exact match |
-| `prob_exact_p1` | Upper bound probability of exact match |
-| `prob_partial_p0` | Lower bound probability of partial match |
-| `prob_partial_p1` | Upper bound probability of partial match |
-
-The **calibrated probability** of a match is typically taken as the average: `(p0 + p1) / 2`
-
----
-
-## Tips
-
-1. **Start with FDR control** if you want high-confidence hits (fewer false positives)
-2. **Use FNR control** if you want comprehensive coverage (don't miss true hits)
-3. **Check the similarity distribution** in your results to calibrate expectations
-4. **Use partial match probabilities** when exact Pfam annotation isn't critical
-
----
-
-## Next Steps
-
-- See [notebooks/pfam/](../notebooks/pfam/) for detailed Pfam analysis
-- See [notebooks/scope/](../notebooks/scope/) for structural classification examples
-- See [notebooks/ec/](../notebooks/ec/) for enzyme classification examples