Among the key challenges of computing today are the emergence of many-core architectures and the resulting need to effectively exploit explicit parallelism. Indeed, programmers are striving to exploit parallelism across virtually all platforms and application domains. The shared memory programming model effectively addresses the parallelism needs of mainstream computing (e.g., portable devices, laptops, desktop, servers), giving rise to a growing ecosystem of shared memory parallel techniques, tools, and design practices. However, to meet the extreme demands for processing and memory of critical problem domains, including scientific computation and data intensive computing, computing researchers continue to innovate in the high-end distributed memory architecture space to create cost-effective and scalable solutions. The emerging distributed memory architectures are both highly parallel and increasingly heterogeneous. As a result, they do not present the programmer with a cache-coherent view of shared memory at the level of the individual node. Furthermore, it remains an open research question which programming model would be ideal for the heterogeneous platforms that feature multiple traditional processors along with accelerators or co-processors. Hence, we have two contradicting trends. On the one hand, programming convenience and the presence of shared memory call for a shared memory programming model across the entire heterogeneous node.  On the other hand, increasingly parallel and heterogeneous nodes lacking cache-coherent shared memory call for a message passing model. In this talk, I will present the architecture of Samhita, a distributed shared memory system that addresses the challenge of providing virtual shared memory for non-cache-coherent systems. I will then talk about regional consistency (RegC), the consistency model implemented by Samhita. I will also present a performance evaluation of Samhita using data from both cluster supercomputers and heterogeneous systems. Finally, I will outline my vision for creating effective solutions for emerging parallel architectures.