Bringing computational chemistry into the exascale era

The NWChemEx project, when realized, has the potential to accelerate the development of next-generation batteries, drive the design of new functional materials, and advance the simulation of combustive chemical processes, in addition to addressing a wealth of other pressing challenges at the forefront of molecular modeling.

As the original NWChem code is some quarter-century old, the NWChemEx developers decided to rewrite the application from the ground up, with the ultimate goal of providing the framework for a next-generation molecular modeling package. The new package is capable of enabling chemistry research on a variety of leading-edge high-performance computing (HPC) systems. Prominent among these systems will be the forthcoming Aurora supercomputer, an exascale Intel-HPE machine to be housed at the Argonne Leadership Computing Facility (ALCF), a U.S. Department of Energy (DOE) Office of Science User Facility located at Argonne National Laboratory.

Support from sponsors including DOE’s Exascale Computing Project (ECP) and the ALCF’s Aurora Early Science Program (ESP) provided the opportunity to restructure core functionality—including the elimination of longstanding bottlenecks associated with the generally successful NWChem code—concurrent with the production of sophisticated physics models intended to leverage the computing power promised by the exascale era. In accordance with this strategy, the developers have adopted the Aurora-supported DPC++ programming model.

From a design point-of-view, the development team gives equal weight and consideration to physics models, architecture, and software structure, in order to fully harness large-scale HPC systems. To this end, NWChemEx incorporates numerous modern software-engineering techniques for C++, while GPU compatibility and support have been planned since the project’s initial stages, thereby orienting the code to the demands of exascale as matter of constitution.

In order to overcome prior communication-related bottlenecks, the developers have localized communication to the greatest possible extent.

Maximal flexibility

The developers pursue multiple approaches to achieve general compatibility. At the core of their work is NVIDIA-based development using the CUDA model. Part of the impetus for this was the availability of and access to GPUs consonant with the team’s experience, thereby maximizing their chances for an efficient and effective development process while accelerating the path to milestone successes.

Today the NWChemEx project encompasses programming models such as CUDA, HIP, and DPC++ in order to target various hardware accelerators. Moreover, the portability of DPC++ potentially makes it a portable programming model for future architectures. With DPC++, explicit control of memory management and data transfers can be scheduled between host and device. The NWChemEx project uses the newly introduced Unified Shared Memory (USM) feature from the SYCL 2020 standards. USM enables developers to work with pointers over the traditional use of buffers and accessors. Work is in progress to transition existing DPC++ code to other SYCL 2020 standards.

Tracking code performance

The transition to GPU has fundamentally altered the ways in which the developers think about how to structure data. Combined with their decision to rewrite the code from the ground up, this has enabled greater creativity and more opportunities to optimize NWChemEx’s ability to seamlessly run on both CPUs and GPUs; while GPUs have opened up new scales of computing power, they still unavoidably have limits and finite memory. This invites a division of labor.

To help localize communication and thereby reduce related bottlenecks, NWChemEx is being geared such that CPUs handle communication protocols as well as any other non-intensive components (that is conditional-structure-based algorithms). Anything else—anything “embarrassingly parallel” or computationally expensive—is to be processed by GPU.

In order to understand the degree to which the application is utilizing experimental hardware, the developers implement a multitiered analysis for tracking code performance.

As a first step, the developers regularly perform roofline analysis to determine the disposition and dependencies of their algorithms: Are they compute-bound? memory-bound? both?

Second, the developers in actuality perform computations on the relevant experimental hardware and compare them against theoretical performance capabilities. This identifies precisely how efficiently processors are being utilized. Finally, the developers conduct a postmortem analysis to pinpoint the origin of errors and establish the scope of improvement that theoretically can be expected.

Intel’s compatibility tool

For Intel hardware, the developers employ Intel’s DPC++ Compatibility Tool to port any existing optimized CUDA code and translate it to DPC++. The Compatibility Tool is sophisticated enough that it reliably determines apposite syntax in translating abstractions from CUDA to SYCL, greatly reducing the developers’ burden. Subsequent to translation, the developers finetune the DPC++ code to remove any redundancies or inelegancies introduced by automation.

The most crucial aspect to using the Compatibility Tool is that it translates—on a timescale ranging from minutes to hours, depending on complexity—entire projects, not just mere source codes or specific functions.

This two-step process—automated translation followed by manual finetuning—generates, from old CUDA code, performant DPC++ code that specifically targets Intel architectures.

NWChemEX was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative. 

Additional support for this project was provided by the National Energy Research Scientific Computing Center (NERSC) Exascale Science Applications Program (NESAP) and the Oak Ridge Leadership Computing Facility’s (OLCF) Center for Accelerated Application Readiness (CAAR), as well as by NVIDIA, Intel, and HPE. NERSC and the OLCF are U.S. Department of Energy Office of Science User Facilities.