social-data-attribution-w2/docs/PACE_GPU_PHOENIX.md

PACE - GT Login - Phoenix Cluster Resources

Excerpt

Knowledge Article

---

Skip to page content

Knowledge Article

Detailed Node Specs¶

• Most nodes includes the following common features:
  • Dual Intel Xeon Gold 6226 CPUs @ 2.7 GHz (24 cores/node)
  • DDR4-2933 MHz DRAM
  • Infiniband 100HDR interconnect
• 4 gpu-h100 nodes include the following common features (1 DGX added November 2023; 3 HGX added January 2024)
  • 1 DGX Node:
    • Intel Xeon Platinum 8480CL CPUs @ 2.0 GHz (112 cores/node)
    • 8x Nvidia Tensor Core H100 80GB
    • Note: Only available using the embers QoS for most Phoenix users
  • 3 HGX Nodes:
    • Dual Intel Xeon Platinum 8462Y+ CPUs @ 2.8 GHz (64 cores/node)
    • 2TB DDR5 DRAM
    • 4TB NVMe
    • 100 Gbps InfiniBand NICs
    • 8x Nvidia Tensor Core H100 80GB
• 12 gpu-h200 nodes, including the following common features:
  • 2x INTEL(R) XEON(R) PLATINUM 8562Y (32c, 2.8GHz) - 64 cores/node
  • 2TB DDR5-5600MHz DRAM
  • 4x200 Gbps InfiniBand NICs
  • 28TB Local NVMe
  • 8x NVIDIA Tensor Core H200-HGX 142GB
• 10 gpu-l40s nodes include the following common features (8 added August 30 2024)
  • Dual Intel Xeon 6426Y CPUs @ 2.5 GHz (32 cores/node)
  • 512GB DDR5 RAM
  • 61.44TB NVMe
  • 100 Gbps InfiniBand NICs
  • 8x Nvidia L40S 48GB GPUs
  • **Note: 8 of these are only available using the embers QoS for most Phoenix users, 2 are available through Inferno
    **
• 54 cpu-gnr nodes include the following common features (54 added October 2025)
  • Dual Intel(R) Xeon(R) 6972P @ 3.8 GHz (192 cores/node)
  • 1.5 TiB DDR5 RAM
  • 4 TB NVMe scratch
  • Dual 400 Gbps InfiniBand NICs
  • 8GB/CPU core
• 2 gpu-rtxpro-blackwell nodes include the following common features (2 added October 2025)
  • Dual 32 core Intel(R) Xeon(R) 6737P  @ 2.9GHz (64 cores/node)
  • 2 TB DDR5 RAM
  • 7 TB NVMe scratch 
  • Dual 400 Gbps InfiniBand NICs
  • 8x Nvidia RTX6000Pro Blackwell 96GB GPUs
• 4 cpu-amd nodes include the following common features (4 added November 2023)
  • Dual AMD Epyc 9534 CPUs @ 2.45 GHz (128 cores/node)
  • 3TB DDR5 DRAM
  • 4TB NVMe
  • 100 Gbps InfiniBand NICs
• 40 cpu-large nodes have been added on February 21, 2023, with Dual Intel Xeon Gold 6226R CPUs @ 2.9 GHz (32 cores/node) and 768 GB of RAM
• 8 cpu-amd nodes include the following common features (4 added November 7, 2022; 4 added February 23, 2023):
  • Dual AMD Epyc 7713 CPUs @ 2.0 GHz (128 cores/node)
  • 512GB DDR4 DRAM
  • 1.6TB NVMe
• 12 gpu-a100 nodes include the following common features (5 added November 7, 2022; 6 added Feburary 23, 2023, 1 added February 28, 2023):
  • Dual AMD Epyc 7513 CPUs @ 2.6 GHz (64 cores/node)
  • 512GB DDR4 DRAM
  • 2x Nvidia Tensor Core A100 40GB (6 nodes) or 80GB (6 nodes) GPUs
  • 1.6TB NVMe
• 1 additional gpu-a100 node includes the following common features (added April 1, 2024):
  • Dual AMD Epyc 7543 CPUs @ 2.8 GHz (64 cores/node)
  • 2TB DDR4 DRAM
  • 8x Nvidia Tensor Core A100 80GB GPUs
  • 180TB NVMe
  • Note: Only available using the embers QoS for most Phoenix users
• 1 cpu-pmem node (added on April 11, 2023) has a large amount of memory (1.5 TB). This memory is composed of 192 GB of DDR4-2933 ECC-DRAM and 1.3125 TB of 2666MHz DCPMM (Intel Optane persistent memory). It has 24 cores (Dual Intel Xeon Gold 6226 CPUs @ 2.7 GHz) and Infiniband 100HDR interconnect.
• The following chart provides detailed specifications for the 1393 nodes that were part of the Phoenix-Slurm cluster migration for Phases 1 (October 10-12, 2022) through 6 (January 30-February 3, 2023). Details are also included for additional nodes: 1 cpu-pmem node, 40 cpu-large nodes, 12 cpu-amd nodes, 12 gpu-a100, and 4 gpu-h100 nodes:

Node Class	Quantity	RAM	Storage	CPU Specs	GPU Specs
CPU-192GB	851	192 GB	1.6 TB NVMe storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-384GB	239	384 GB	1.6 TB NVMe storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-768GB	104	768 GB	1.6 TB NVMe storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-384GB-SAS	75	192 GB	8.0 TB SAS storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-768GB-SAS	4	384 GB	8.0 TB SAS storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-PMEM	1	1.5 TB	1.6 TB NVMe storage	2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)
CPU-GNR	54	1.5 TB	4 TB NVMe storage	Intel(R) Xeon(R) 6972P (192 cores/node)
GPU-192GB-V100	21	192 GB		2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)	2x Nvidia Tesla V100 (16GB or 32GB)
GPU-384GB-V100	27	384 GB		2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)	2x Nvidia Tesla V100 (16GB or 32GB)
GPU-768GB-V100	5	768 GB		2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)	2x Nvidia Tesla V100 (16GB)
GPU-384GB-RTX6000	32	384 GB		2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)	4x Nvidia Quadro RTX6000 (24GB)
GPU-768GB-RTX6000	5	768 GB		2x Intel Xeon Gold 6226 @ 2.7GHz (24 cores/node)	4x Nvidia Quadro RTX6000 (24GB)
GPU-2TB-RTX-PRO-6000-BLACKWELL	2	2TB	7 TB NVMe Storage	Intel(R) Xeon(R) 6737P (64 cores/node)	8x Nvidia RTX 6000 Pro Blackwell  (96GB)
CPU-512GB-AMD	8	512 GB	1.6 TB NVMe storage	2x AMD Epyc 7713 @ 2.0GHz (128 cores/node)
CPU-3TB-AMD*	4	3 TB	4.0 TB NVMe storage	2x AMD Epyc 9534 @ 2.45GHz (128 cores/node)
GPU-512GB-A100	12	512 GB	1.6 TB NVMe storage	2x AMD Epyc 7513 @ 2.6GHz (64 cores/node)	2x Nvidia Tensor Core A100 (40GB or 80GB)
GPU-A100-HGX*	1	2 TB	180 TB NVMe storage	2x AMD Epyc 7543 @ 2.8GHz (64 cores/node)	8x Nvidia Tensor Core A100 (80GB)
GPU-H100-HGX	3	2 TB	4.0 TB NVMe storage	2x Intel Xeon Platinum 8462Y+ @ 2.8GHz (64 cores/node)	8x Nvidia Tensor Core H100 (80GB)
GPU-H100-DGX*	1	2 TB
2x Intel Xeon Platinum 8480CL @ 2.0GHz (112 cores/node)

| 8x Nvidia Tensor Core H100 (80GB) | | GPU-H200-HGX | 12 | 2TB | 28.0 TB NVMe storage | 2x Intel Xeon Platinum 8562Y+ @ 2.8GHz (64 cores/node) | 8x Nvidia Tensor Core H200 (142GB) | | GPU-L40S | 10 | 512 GB | 61.44 TB NVMe storage | 2x Intel Xeon 6426Y @ 2.5GHz (32 cores/node) | 8x Nvidia L40S (48GB) | |   | 1393** |   |   |   |   |

*Note: Only available using the embers QoS for most Phoenix users

**Note: Total node count may vary due to maintenance

Partitions¶

• Jobs are assigned to Slurm partitions automatically based on your charge account (internal or external) and the most significant resources requested (gpu, memory requirements, etc).
• Jobs will only be charged if the inferno QOS is selected.
• Slurm partitions assigned determine how much users are charged based on current rates.
• Slurm partitions include the following node classes and are assigned by the scheduler based on availability:

Partition	Node Class
cpu-small	CPU-192GB, CPU-384GB, CPU-384GB-SAS, CPU-768GB, CPU-768GB-SAS
cpu-medium	CPU-384GB, CPU-384GB-SAS, CPU-768GB, CPU-768GB-SAS
cpu-large	CPU-768GB, CPU-768GB-SAS
cpu-gnr	CPU-GNR
cpu-sas	CPU-384GB-SAS, CPU-768GB-SAS
cpu-pmem	CPU-PMEM
gpu-v100	GPU-192GB-V100, GPU-384GB-V100, GPU-768GB-V100
gpu-rtx6000	GPU-384GB-RTX6000, GPU-768GB-RTX6000
cpu-amd	CPU-512GB-AMD, CPU-3TB-AMD
gpu-a100	GPU-512GB-A100
gpu-h100	GPU-H100-DGX, GPU-H100-HGX
gpu-l40s	GPU-L40S
gpu-rtxpro-blackwell	GPU-2TB-RTX-PRO-6000-BLACKWELL

• The partitions for external users have the same names with "-X" added (i.e. cpu-small-X, cpu-medium-X).

Job Submit Flowchart¶

• When submitting a job to the Slurm scheduler using interactive mode (using salloc) or with a script (using sbatch), the resources requested will determine the partition assigned, as illustrated in the following flowchart:

Important The scheduler reserves 8GB of memory for system processes, so the total available memory for jobs on a given node is reduced accordingly. For memory requests on the border between partitions (i.e. --mem-per-cpu=8GB for cpu-small/cpu-medium, --mem-per-cpu=16GB for cpu-medium/cpu-large), jobs will allocate nodes of the larger partition node type but charge for the smaller partition type.