Compiling and Linking on ThetaGPU

Help Desk

Theta GPU Nodes

Compiling and Linking on ThetaGPU

ThetaGPU has AMD processors on the service nodes (thetagpusn1,2) and AMD processors and NVIDIA A100 GPUs on the compute nodes [see overview page]. The service nodes can be used to create containers and launch jobs, and eventually to use as a cross-compiling environment for compute nodes. Until the cross-compiling environment is set up, the compute nodes will have to be used for compiling. This can be done by using an interactive Cobalt job (via qsub -I), or until we have reserved or added a dedicated build node.

The default programming environment on the ThetaGPU compute nodes is the GNU compiler tools coupled with NVIDIA’s CUDA toolkit. 

Note: Symlinks to the project directories are not available on the compute nodes. Use the full path (eg: /lus/theta-fs0/projects/<projectname>) to access the project directory.

For non-GPU codes:

  • gcc – for C compiler
  • g++ – for C++
  • gfortran – for Fortran

For CUDA codes:

  • nvcc

For MPI, the latest MPI is in /lus/theta-fs0/software/thetagpu/openmpi-4.0.5.  

  • mpicc
  • mpicxx
  • mpif77/mpif90 not configured yet

mpirun is a wrapper in /usr/local/bin that sets the appropriate options and uses the mpirun in the MPI directory above.

On the service nodes, GNU compilers are available.

Modules on ThetaGPU

Available modules can be listed (on thetagpusn1,2) via the command:

user@thetagpusn1:~$ module avail


----------------------- /usr/local/lmod/lmod/modulefiles -----------------------

   Core/lmod    Core/settarg


-------- /lus/theta-fs0/software/environment/thetagpu/lmod/modulefiles ---------

   Core/StdEnv                 (L,D)    conda/tensorflow/2021-01-08

   aocl/blis-3.0                        conda/tensorflow/2021-03-02      (D)

   conda/pytorch/2020-11-25             nccl/nccl-v2.8.4-1_CUDA11

   conda/pytorch/2021-03-02    (D)      openmpi/openmpi-4.0.5            (L)

   conda/tensorflow/2020-11-11          openmpi/openmpi-4.1.0_ucx-1.10.0

   conda/tensorflow/2020-12-17          openmpi/openmpi-4.1.0            (D)

   conda/tensorflow/2020-12-23


-- /lus/theta-fs0/software/spack/share/spack/modules/linux-ubuntu18.04-x86_64 --

   autoconf-2.69-gcc-7.5.0-wmttzuv

   autoconf-archive-2019.01.06-gcc-7.5.0-bdyarrk

....

Loaded modules in your environment can be listed (on thetagpusn1,2) via the command:

user@thetagpusn1:~$ module list


Currently Loaded Modules:

  1) openmpi/openmpi-4.0.5   2) Core/StdEnv

To load new modules use:

user@thetagpusn1:~$ module load <module_name>

There are few modules available at this time, but the number will grow as more packages become available.

Usage: csh and zsh users do not have to do anything special to their environments. bash users, however, will need to add the following to any job scripts:

#!/bin/bash
. /etc/profile

bash users are also encouraged to modify their ~/.bashrc to ensure the ubuntu system /etc/bash.bashrc file is sourced properly:

# Source global definitions
if [ -f /etc/bashrc ]
then
    . /etc/bashrc
elif [ -f /etc/bash.bashrc ]
then
    . /etc/bash.bashrc
fi