Simulations of large-scale structure formation that can simultaneously encompass a representative volume of the universe and resolve the dark matter halos that host galaxies are required for both planning and analyzing current and future astronomical surveys of galaxies across the sky. In order to harness the power of modern supercomputing systems for running such simulations the Hardware/Hybrid Accelerated Cosmology Code (HACC) has been developed to address issues of massive concurrency and heterogeneity. HACC uses N-body methods and splits the calculation of the gravitational force into a long-range component that is highly portable and a short-range component that is tuned to specific compute node architectures. Variants of HACC have been developed for x86, IBM Cell (LANL/Roadrunner), IBM Blue Gene (ANL/Mira), and GPGPU (ORNL/Titan) systems. In this talk I will focus on how experiences with various memory hierarchies and potential performance bottlenecks has influenced ite rations of code design in order to achieve better load-balancing and higher performance. In addition I will review the evolution of the in-situ and post-processing analysis strategies employed in HACC as its simulations have grown beyond the petascale. Finally, I will speculate briefly on the convergence of high performance computing and data intensive computing that will be necessary to fully leverage observations from future galaxy surveys for cosmological inference.