feat(eda): normalize dataset paths and deduplicate results in summary

feat(eda): retrieve chunk size for each dataset in batch processing

feat(eda): update large file processing to support parallel workers and enhance metadata caching

feat(config): fix some hard code config and docs

feat(eda): categorize datasets into small, dask-ready, and xlarge for improved processing

feat(eda): adjust worker settings and add emergency mode for handling failed slices for extremly large

feat(eda): enhance dataset processing for extra large

feat(eda): add adaptive scaling parameters and initial worker configuration for improved resource management

feat(eda): enhance resource utilization by optimizing worker allocation and processing parameters

feat(metadata): add handling for missing datasets in CELLxGENE metadata and update status reporting

feat(eda): add cache validation and retry mechanism for metadata build

fix(eda): optimize gene statistics calculation in distributed EDA

fix(eda): remove undefined 'info' variable reference causing crash

refactor(slurm): update resource allocation and remove deprecated script

feat(eda): add resume capability and graceful error handling

feat(eda): implement hybrid processing strategy for small and large datasets

feat(eda): refactor distributed EDA script for improved performance and memory management

fix(eda): optimize memory usage and ensure complete data computation

feat(eda): migrate to Dask distributed with adaptive scaling and memory limits

fix(eda): use recent throughput instead of cumulative average for adaptive scaling

feat(eda): add adaptive worker reduction based on throughput monitoring

fix(config): remove mem_per_worker_gib from config files and calculate dynamically in resource_probe script

fix(config): clarify max_memory_gib allocation for staged processing

fix(retry): add size categorization after merge to prevent null categories

fix(retry): include corrupted status in retry logic

fix(cache): categorize ok_retry and ok_h5py datasets by size

fix(eda): include all successfully scanned datasets (ok_retry, ok_h5py)

feat(recovery): add corrupted file redownload script and documentation

feat(retry_failed_cache): implement dataset retry mechanism and merging of results

fix(cache): implement two-phase scanning to handle large files serially and prevent OOM

fix(cache): use ProcessPoolExecutor for HDF5 thread-safety

feat(pipeline): add YAML config, metadata-aware scheduling, and dataset slicing

fix(eda): prevent BrokenProcessPool cascade failures

Initial commit: distributed EDA pipeline, max non-zero reporting, and notebook