To digitally reconstruct the universe, scientists use massive computer simulations that capture the full sweep of cosmic evolution across billions of years. These simulations are so large, they cover distances stretching across billions of light-years — the scale of the entire visible universe.

In a milestone achievement, a research team led by the U.S. Department of Energy’s (DOE) Argonne National Laboratory and Oak Ridge National Laboratory (ORNL) has harnessed the power of one of the world’s most advanced supercomputers, ORNL’s Frontier, to run the biggest hydrodynamic cosmological simulation ever achieved. 

The Frontier Exascale (Frontier-E) simulation sets a new benchmark in cosmological modeling — combining unprecedented scale with high-fidelity physics and complexity to simulate the universe with a level of realism not previously achievable at these scales. 

For their work, Argonne and ORNL researchers have been selected as finalists for the prestigious Gordon Bell Prize. Presented by the Association of Computing Machinery, the annual prize recognizes breakthroughs in using high performance computing to solve complex science problems.

Frontier-E is the culmination of an ambitious effort launched in 2016 under the DOE’s Exascale Computing Project (ECP), which concluded in 2024.

“The effort involved adding new physics capabilities to an already highly performant code, improving performance on new architectures, and scaling up to a size that would use all the power of exascale systems,” said Salman Habib, director of Argonne’s Computational Science division and an Argonne Distinguished Fellow. ​“Frontier-E is the first major scientific simulation to fully meet the goals of a Challenge Problem set by the ECP.”

An ECP Challenge Problem is a large-scale scientific or engineering task designed to push the limits of exascale computing.

Frontier-E is the first trillion-particle cosmological hydrodynamic simulation of its kind, combining gravity with detailed modeling of baryonic matter such as gas and stars. The application uses a staggering 4 trillion particles to model cosmic evolution, incorporating detailed astrophysical processes. Each particle represents a portion of cosmic material, across a volume comparable in scale to modern astronomical surveys.

Frontier-E was run with more than 15 times the number of particles used in previous simulations, allowing scientists to capture finer structures, cover larger regions of space and model the complex physics of the universe with greater fidelity. 

Scientists use Frontier-E to simulate a ​“digital universe,” modeling cosmic evolution with dark and normal (baryonic) matter — the two components in the universe. By creating a detailed synthetic universe, scientists can compare the results with real telescope observations.

“By combining trillions of particles with realistic physics in a dynamic, expanding space, the Frontier-E simulation can follow the evolution of the universe from its earliest cosmic structures to the present day. It is like watching the rapid expansion of the cosmos unfold, revealing billions of distinct galaxies forming across time,” said Nicholas Frontiere, a computational scientist at Argonne and a lead researcher on the project. 

The code can also perform targeted simulations that model smaller regions of the cosmos, allowing scientists to rigorously test its physics and modeling assumptions. 

Powering the simulation: CRK-HACC and Frontier

Frontier-E was run with a code called CRK-HACC (Conservative Reproducing Kernel Hardware/Hybrid Accelerated Cosmology Code), a high-performance extension of the HACC framework that was developed more than a decade ago. The framework has been continuously enhanced under the leadership of Argonne researchers.

While the core HACC code models the gravitational evolution of cosmic structures, CRK-HACC extends its capabilities to include full hydrodynamics, adding gas and astrophysical feedback physics while preserving the scalability and performance optimizations of the gravity-only code.

Graphics processing units (GPUs) — specialized chips built for handling massive parallel computations — are central to the performance achieved by CRK-HACC in Frontier-E’s run. The CRK-HACC framework is optimized for GPU acceleration, enabling it to scale seamlessly across all 9,000 nodes of the Frontier supercomputer.

By also using GPUs to process and analyze massive datasets during the simulation, the team cut run times dramatically — from about a year on a central processing unit (CPU)-based system to just one week of machine run-time. CRK-HACC also runs at scale on Argonne’s exascale-class Aurora supercomputer.

“Making the full CRK-HACC code run efficiently on GPUs was a major breakthrough,” said Frontiere.

Striving for new heights in simulating the universe

Among its groundbreaking results, Frontier-E reached a peak speed of 513 petaflops, equivalent to about 513 quadrillion calculations per second. 

The simulation produced over 100 petabytes of data yet spent less than 3% of its total runtime on saving and storing that information. A petabyte is a unit of digital information storage equal to 1 million gigabytes.

Altogether, Frontier-E established new records for performance, data handling and scalability in large-scale cosmological simulations.

“Frontier-E is exactly the sort of impossible-anywhere-else computational science that the Leadership Computing Facility is designed to enable,” said Bronson Messer, ORNL director of science. ​“The addition of baryonic physics to a trillion-particle dark matter simulation and carrying out that work at the very edge of performance in every important component is a real tour de force.” 

While the results set new milestones, Frontier-E also highlights areas ripe for advancement. Scientists aim to push resolution even further, incorporate more complex physics and refine models to better match observational data. 

“As telescopes become more powerful and datasets grow, simulations like Frontier-E will evolve to meet the challenge — bringing us closer to answering fundamental questions about dark matter, dark energy and the origins of cosmic structure,” said Frontiere. 

HACC is a core component of the ExaSky project, which was part of DOE’s Exascale Computing Project. In both 2012 and 2013, the HACC team was a finalist for the Gordon Bell Prize in computing for their accomplishments in extreme-scale gravity-only simulations.

The precursory model parameter scans necessary to run Frontier-E were conducted on the Perlmutter supercomputer at Lawrence Berkeley National Laboratory. 

In addition to Frontiere and Habib, the team includes Argonne researchers J.D. Emberson, Michael Buehlmann, Esteban M. Rangel, Katrin Heitmann, Patricia Larsen, Vitali Morozov and Adrian Pope; Northwestern University researcher Claude-Andre Faucher-Giguere, and ORNL’s Antigoni Georgiadou, Damien Lebrun-Grandie and Andrey Prokopenko.

This research was supported by the Exascale Computing Project, a collaborative effort of the DOE Office of Science and National Nuclear Security Administration, and by DOE’s Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through the Advanced Computing (SciDAC) program.