Much like personal computers, the average lifespan of a supercomputer is four to five years. In 2013, Intrepid, the Argonne Leadership Computing Facility’s (ALCF) IBM Blue Gene/P supercomputer, hit the five-year mark, ending an impressive run of innovative computational research.

Ranked as the third fastest computer in the world when it debuted in 2008, Intrepid enabled researchers to accelerate studies of everything from advanced battery materials and climate models to supernovae and Parkinson’s disease. The 557-teraflops system was decommissioned on Dec. 31, 2013, but its legacy lives on in many of the ALCF’s ongoing research projects.

“Intrepid was a transformative system that consistently demonstrated how supercomputers are changing the way we solve scientific and engineering problems,” said ALCF Director Mike Papka. “As the ALCF’s first production machine, it helped blaze new trails for computational research in several disciplines, opening the door to many of the projects that are currently running on Mira.”

Intrepid’s successor, Mira, entered production mode in April 2013 and began handling the full ALCF workload in 2014. At 10 petaflops, the IBM Blue Gene/Q supercomputer has a peak speed 20 times faster than Intrepid.

With a vast majority of large-scale ALCF projects selected through the INCITE or ALCC allocation programs, many of the projects are multiyear efforts. Researchers often seek allocation renewals to continue their work from year to year, allowing research projects to evolve along with the facility and its increasingly powerful computing resources. In addition, many projects are so innovative and complex that it takes years to fully develop the software and achieve their goals.

From Protein Structure Prediction to Protein Design

David Baker, head of the Institute of Protein Design at the University of Washington, has been tapping ALCF supercomputers for more than five years to advance the development of engineered proteins by creating and studying 3D models of protein structures at atomic-level resolution.

With his initial INCITE allocation on Intrepid in 2008, Baker gained access to a machine that would take his protein structure prediction work to new heights.

“Up to that point, the computing resources we’d used had all been embarrassingly parallel. We basically had access to Linux cluster with nodes that pretty much weren’t communicating,” Baker said. “It was really when we started using ALCF resources that we were able to take advantage of more and more sophisticated algorithms that were essentially able to dynamically relocate where we were focusing our research based on what we were finding.”

“A couple of years ago when we were able to use 32,000 cores simultaneously for these large-scale searches, it was really kind of a milestone for us,” he added.

While Intrepid has allowed Baker’s team to successfully determine the structure of many proteins of biological interest, Mira is enabling them to break new ground in protein design. With increased processing power and more memory per core, the researchers are able to model larger systems and store thousands of different possible conformations, both of which are necessary for protein design. Currently, the team is working on designing peptides with future applications in possible protein therapeutics. 

“Mira is allowing us to design peptides for the first time,” said Vikram Mulligan, a Senior Fellow at the University of Washington and a member of Baker’s research team. “We’ve started to design some peptides that will hopefully be able to bind to an enzyme involved in antibiotic resistance. If we can inhibit this enzyme, our research on Mira could be applied to conventional antibiotics so they work against antibiotic-resistant bacteria.”

Simulating Hydrogen Combustion in Industrial-Sized Pipes

University of Chicago professor Alexei Khokhlov began using Intrepid in 2010 to advance the design of safer hydrogen fuel systems by studying the deflagration-to-detonation transition (DDT) process for hydrogen-oxygen mixtures.

DDT involves several physical processes, including chemical reactions, microscopic transport, turbulent fluid flows, and a wide range of temporal and spatial scales, which makes numerical modeling of these phenomena extremely computationally intensive.

“Intrepid was the first machine we could use to develop a new approach for direct numerical simulations of DDT in long pipes,” said Khokhlov, who has received computing time at the ALCF through the INCITE program and the ALCF’s Early Science Program.

His first three years on ALCF systems were devoted to code development and validation on Intrepid, with early simulations showing excellent agreement with non-reactive experiments using carbon dioxide.

With the groundwork completed on Intrepid, Khokhlov’s research team has been able to use Mira to study reactive hydrogen-oxygen mixtures, demonstrating the feasibility of first-principles DDT simulations in industrial-sized, meter-long pipes.

“These simulations are giving us an opportunity to study phenomena not observable in experiments,” Khokhlov said. “Without Intrepid and Mira, this research would not exist.”

Nuclear Reactor Simulations Capture Industry Attention

Paul Fischer, senior computational scientist at Argonne, has also seen his research into nuclear reactor hydrodynamics take leaps and bounds during his time using ALCF resources through the INCITE program. He started with the ALCF’s Blue Gene/L machine in 2005 before moving on to Intrepid in 2008.

The upward trajectory of the ALCF’s computational capabilities can be seen in the progression of Fischer’s simulations of nuclear reactor fuel assemblies. Prior to Intrepid, his research team was able to simulate a 7-pin fuel assembly. With Intrepid, they successfully simulated 19-pin, 37-pin, and, finally, full-scale 217-pin bundles.

“These simulations involve fairly complex geometries,” Fischer said. “It was a learning process. We worked our way up, gaining experience as we performed simulations of larger and larger pin assemblies.”

Intrepid also enabled the first high-fidelity simulations of wire-wrapped fuel pins, which added to the accuracy of the simulations and helped to capture the attention of the nuclear engineering community. The nuclear industry began looking to Fischer’s large eddy simulations as an alternative to experiments in validating the Reynolds-Averaged Navier-Stokes (RANS) models used to design advanced nuclear reactors.

Fischer’s work continues to improve and evolve on Mira, with his research team now collaborating with four industrial partners to simulate different kinds of fuel assemblies.

“Industry started to realize that they can accelerate research and development by doing things with high-performance computing that would be extremely expensive through experiment,” Fischer said. “It’s starting to become a game changer.”

In addition to the groundbreaking nuclear reactor simulations, Fischer’s team leveraged Intrepid to perform many so-called “hero calculations” that tested the limits of high-performance computing. This allowed them to achieve many firsts for Fischer’s Nek5000 code, including running on 1 million cores and performing simulations that exceeded 1 billion grid points.

“What was considered a hero calculation in 2009 is now being done routinely on Mira to address real-world engineering questions,” Fischer said. “It really shows you how the evolution of high-performance computing will continue to allow us to go into new places that we haven’t been able to go before.”