Quantum Monte Carlo (QMC) methods are ideal candidates for the next generation of material-design tools, which target not only simple bulk properties but collective effects in strongly correlated materials such as magnetic ordering, phase transitions, and quantum coherence.

The open-source QMCPACK simulation code performs electronic structure calculations via accurate approximation of the Schrodinger equation. Run on all U.S. Department of Energy (DOE) supercomputers, it has been used for research at the Argonne Leadership Computing Facility (ALCF) since 2011.

A multi-institutional effort that includes researchers from Argonne, Lawrence Berkeley, and Oak Ridge National Laboratories is now underway to prepare QMCPACK for deployment on forthcoming, GPU-powered exascale machines, including the ALCF’s Aurora supercomputer. The greatly expanded computational power and parallelism of exascale will enable predictive capabilities far beyond the capacity of QMCPACK’s current implementation.

Porting to exascale

At Argonne, computational scientists Ye Luo, Mark Dewing, Hyeondeok Shin, Thomas Applencourt and Anouar Benali, who are also part of the QMCPACK development team, are leading efforts to prepare the code for Aurora.

The developers began the process of building an exascale version of QMCPACK under the auspices of DOE’s Exascale Computing Project and ALCF's Aurora Early Science Program, with the goal of achieving portability across a diverse array of architectures. This desire for performance portability provides the core of their development strategy.

Writing application code with premature software for hardware that does not yet fully exist poses a complex problem, but it is nonetheless a problem that needs to be solved.

The developers have adopted an approach whereby they excise portions of the QMCPACK code, which comprises numerous applications and kernels, so as to anatomize the components and functionality of each portion on different levels via miniapps. Of course, just because a miniapp works in isolation doesn’t mean it will work in the greater context of the code; to avoid aiming too broadly and/or over-engineering, the developers narrow their problems as much as possible and effect changes incrementally, practicing continuous integration (CI).

The developers initially investigated a variety of programming models for GPU-accelerated, vendor-agnostic source code. They explored both OpenMP offload and Kokkos before deciding the former would be the primary programming model for exascale implementations of QMCPACK. The OpenMP port of QMCPACK has proved resilient enough that it today runs on NVIDIA, AMD, and Intel GPUs

However, as one would expect, each new implementation comes with its own bugs. The team often codesigns with vendors the miniapps and isolated kernels it employs in the development process; such codesign helps ensure the delivery of functional compilers, performance libraries, and runtime libraries for OpenMP.

Vendor-specific libraries

The other component of the developers’ portability strategy concerns vendor-optimized libraries. QMCPACK requires optimized linear algebra libraries, which, given that they are provided by vendors, typically are coupled with the vendors’ preferred programming model.

To facilitate a variety of vendor-specific libraries, the developers are redesigning the entire infrastructure of QMCPACK. The intent is to ensure there exists a general framework that enables the flexible implementation of higher-level algorithms while allowing specialized implementations to run at a lower level as needed.

This lower level can be leveraged to connect the vendor-specific portions with only minimal divergence from the source code. C++ provides the means by which to mediate between high-level algorithms and low-level specialized vendor-specific code.

Legacy code

An earlier GPU code the developers wrote was integrated into QMCPACK so as to prototype different approaches for harnessing GPU accelerators. While its capabilities and integration are unsuitable for release or production work, this “legacy” CUDA code, whose behaviors are well understood, serves as a useful benchmark and reference for the NVIDIA platforms on which the developers are building the new, performance-portable QMCPACK implementation for contemporary GPU architectures.

To minimize functionality gaps, this CUDA-based, somewhat ad hoc approach needs to be supplanted in the long run with a more elegant strategy for consistent code design that agrees with both CPUs and GPUs.

For now, however, the compartmented structure of the code permits a sort of divide-and-conquer approach whereby the developers can independently validate different portions (for example, the OpenMP section of the code can be validated separate from the vendor-specific library portion), eliminating a major source of interference.

The exascale port of QMCPACK has been deployed for production science on leadership machines including the ALCF’s Theta and the Oak Ridge Leadership Computing Facility’s Summit; the developers continue to consolidate QMCPACK for optimal performance on Intel and AMD hardware.

Achieving optimal performance

In attempting to attain optimal performance, the developers take a holistic approach that accounts for the entirety of their software stack and hardware. In addition to creating fast compute kernels, the developers focus on minimizing resource idle on host processors.

This stems from the developers’ own experience: overhead from dispatching workloads to accelerators and data transfers can dominate the whole execution if not managed properly. Reducing GPU runtime costs is perhaps the developers’ most significant challenge. While on a CPU a function runs immediately after its dispatch, kernel execution on a GPU may be delayed until the GPU has completed prior tasks—the host must wait. This also poses the difficulty of how to hide overhead. Satisfactory solution of these problems necessitates asynchronous computation.

To avoid inefficient executions bogged down by lengthy synchronizations or data transfers, the team utilized OpenMP runtime implementations to incorporate more asynchronous features that run in tandem with the offload compute component. Overlapping computation and communication is an optimization technique frequently used with accelerators to hide the cost of data transfers over slow PCI-E buses. It can be achieved through OpenMP runtime without any user intervention when offload multiple host threads generate overhead concurrently. Its behavior can be easily visualized by tracing GPU activity with standard vendor tools. Exposing as many offload computations as possible, OpenMP threading and tasking are extremely powerful features which are now more than a decade mature. In brief, OpenMP is used to its maximal extent to mobilize resource within a compute node.

Collaborating with Intel

QMCPACK operations in large part depend on an LLVM Clang compiler. Propitiously aligning with this dependency, Intel transitioned their legacy compiler to an LLVM-based model.

The developers work with Intel to optimize the OpenMP portion of the Intel LLVM Clang compiler, providing Intel with QMCPACK- and OpenMP-related publications, tests, and benchmarks to guide and refine compiler development. Compiler validation, on the other hand, is aided by daily use in QMCPACK operations.

Close collaboration with Intel also occurs in the modification of certain application programming interfaces (APIs) in the Math Kernel Library (MKL). The APIs, as implemented in QMCPACK, are not designed for GPU performance. In addition to generating benchmarks and performing validation for MKL, the developers interact with Intel to design new MKL APIs.

DPC++ functions as a glue in places where OpenMP is insufficient or where there is no OpenMP alternative. In this sense the presence of DPC++ helps the developers identify areas where there are potential performance gaps and take advantage of features exclusive to DPC++.

The ALCF is a DOE Office of Science user facility located at Argonne National Laboratory.