Each new generation of supercomputer at the U.S. Department of Energy’s (DOE) Argonne National Laboratory brings new technologies and capabilities designed to advance science. Taking advantage of these new architectures and features also comes with a learning curve.

Launched in August 2022, Polaris is the Argonne Leadership Computing Facility’s (ALCF) most powerful supercomputer to date. The machine, developed in collaboration with Hewlett Packard Enterprise, is a hybrid system equipped with NVIDIA GPUs (graphics processing units) and AMD CPUs (central processing units). The ALCF is a DOE Office of Science user facility at Argonne.

To help researchers jumpstart their science projects on Polaris, the ALCF recently hosted its third annual GPU Hackathon in partnership with NVIDIA and OpenACC. This year, the hackathon was focused on helping teams elevate their code performance to prepare for the 2024 call for proposals for DOE’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

“Our goal is to help teams port their codes to GPU-accelerated architectures and help teams to scale up to using large numbers of GPUs in their work,” says ALCF computational scientist Chris Knight, who served as a mentor at the hackathon. “ALCF training events like this become the start of a long relationship between ALCF staff and teams working together to enable new impactful science on large-scale supercomputers.”

One of the hackathon teams with members from Argonne and Washington University in St. Louis signed up for the hackathon to transition their computationally intensive physics problem to Polaris. They plan to use the system to gain a better understanding of how the interactions of protons and neutrons affect the properties of nuclei. Thanks to the event, the team was able to double the speed of their code.

“We took this sort of ‘lawn mower approach’ to improving our code, if you will,” says Patrick Fasano, a postdoctoral appointee in Argonne’s Physics division, “which is where you go and find the tallest thing, the thing that takes the most time and try to port it to the GPU. Then you whack it and whack it down again. You then find the next most time-consuming part and repeat again, and I think this is how we got a surprisingly large speedup.”

“The hackathon was really valuable to us,” adds Fasano, “You can’t discount the importance of having a structure and an impetus for your work, and the value of getting your research running on GPUs. In the hackathon setting, our team worked hard to have something valuable to report to our mentors and peers.” 

The 12 participating teams were paired with experts to port, accelerate and optimize HPC and artificial intelligence (AI) applications on Polaris using diverse programming models, libraries and tools. The teams’ applications spanned a wide range of research areas including weather research and forecasting models, colon cancer research and methods to reconstruct large biomolecular structures.

“Rather than developers searching around the internet for code examples and studies that might be related to their project, these hackathons create an opportunity for teams to discuss their specific code questions and issues with HPC experts,” Knight says. “An important aspect of this event is prioritizing one-on-one interactions with teams to understand their science application, the challenges they’re facing, what they would like to achieve, and collaborating on a strategy to use the large-scale resources at ALCF.”

During the course of the hackathon, Fasano and his collaborators spent most of their time using profiling tools to assess and improve their code’s performance.

“We were really trying to determine where the performance bottlenecks were in the code, which ended up being very interesting,” says Fasano, “We decided to do a little bit of profiling, just to make sure that we truly understood what we have, and after we did an initial profile, it was not at all what we expected.”

Using profiling tools, the team found that the issue was a single OpenMP loop that was about 10 lines of code. Working closely with mentors from the ALCF and NVIDIA, the team was able to achieve a 20x speedup in one kernel, which led to the 2x speedup overall.

“This experience was super helpful,” says Fasano, “it was vital having someone who knew all weird edge cases related to the codes you are working on.”

The Argonne-Washington University team wasn’t the only team that had success in improving code performance. All 12 teams were able to make strides toward demonstrating computational readiness on Polaris – a requirement for INCITE proposals.

A team of researchers from the University of Illinois Chicago, for example, was able to achieve a 10x speedup of their code used to study 3D-chromatin structures. Mentored by Argonne computational scientist Wei Jiang, the team’s goal was to convert their entire source code from CPU to GPU. Through collaborative efforts with ALCF staff and by changing an important loop in their code, they were able to achieve significant performance improvements when they shifted to GPU.

“This hackathon was the one of the most productive events I’ve participated in,” says Jiang. “All the mentors involved in the event were very knowledgeable, yet able to provide low-level and detailed technical assistance, which was valuable to everyone."

The facility’s next hands-on training event, the ALCF Hands-on HPC Workshop, will be held October 10-12, 2023. To register, visit: https://www.alcf.anl.gov/events/alcf-hands-hpc-workshop 

==========

The Argonne Leadership Computing Facility provides supercomputing capabilities to the scientific and engineering community to advance fundamental discovery and understanding in a broad range of disciplines. Supported by the U.S. Department of Energy’s (DOE’s) Office of Science, Advanced Scientific Computing Research (ASCR) program, the ALCF is one of two DOE Leadership Computing Facilities in the nation dedicated to open science.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.

The U.S. Department of Energy’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://​ener​gy​.gov/​s​c​ience.