Tier 1 Science Project

Science

This ESP project will be used to advance four critical scientific applications that have a very diverse set of scientific objectives. The first scientific application will consist of performing simulations on the order of a few neocortical columns, each including 31,000 neurons and about 37 million synapses. The particular focus will be to study microcircuit plasticity, the phenomena of experience-dependent changes in synaptic connectivity, which is thought to be a substrate for learning and memory. The second and third scientific applications will rely on the simulation of the electrical activity of the rodent somatosensory cortex with morphologically detailed neurons to refine, on the one hand, the model and on the other hand develop a neurobotics application. The last scientific application will consist of simulating on the electrical activity of the largest possible brain model for several seconds of biological time.

Impact: While much is known about the brain, little is understood. Integration through data-driven brain tissue models is a novel method in the scientific repertoire. With this project, researchers will advance simulation tools, widen the ability for scientific questions on brain plasticity, and trigger valuable discussions with the HPC community.

Problem Scale

For the first science goal, researchers will simulate of the order of a few neocortical columns, each with 31,000 neurons and 36 million synapses. For the second and third science goals, they will simulate on the order of 20 million neurons and 20 billion synapses. The exact scales for the fourth science goal will be determined based on progress in draft rodent brain models.

Numerical Methods/Algorithms

CoreNeuron is an application used to simulate the electrical activity of large-scale neuronal circuits using detailed cellular morphologies, multi-compartment models, Hodgkin-Huxley channel and probabilistic Tsodyks-Markram synapse models. Time discretization is backward Euler; spatial discretization defines cylindrical elements referred as compartments. The bulk of the work is solving a large number of 1D nonlinear ODEs. A quasi-tridiagonal linear system is solved using the Hines algorithm.

Parallelization

The parallelization of CoreNeuron is implemented using MPI and OpenMP. A single MPI process is used per node with 64 threads on IBM Blue Gene/Q and 120 on a single node of Intel KNC-based Phi system (which was enough to saturate the memory bandwidth).

Application Development

• Expose more parallelism from the CoreNeuron application: investigate a variant of the code where parallelism at the compartment rather than the single neuron level can be exposed.
• Try using the MPI distributed graph and neighborhood collective functionalities to replace MPI_Allgather to support close and faster than real time use cases.
• Above improvements may put pressure on spike exchange protocol. If so, investigate using low-level networking interface to reduce communication latency
• Store as many neurons as possible in the available memory: For one case, make use of available non-volatile memory/persistent data store.
• MCDRAM: Try using as cache. Also experiment with flat and hybrid models, where data used in memory-bound kernels preferably put into MCDRAM while compute kernel and neuron topological data remain in DRAM.

Portability

The Blue Brain Project has an active effort on portability based on code generation using a computational neuroscience DSL. This currently supports Intel x86 and Xeon Phi using OpenMP directives, and GPUs using OpenACC directives. There is also a C++ library, which encapsulates machine-specific math implementations based on lower-level approaches such as vector intrinsics.