Automatic differentiation (AD) is a method to compute gradients of numerical software with a long history in science, engineering, and recently machine learning. Due to the ongoing stall in processor clock speeds and the growing demand for energy-efficient computing, applications are increasingly using multi-threading, SIMD vectorisation, or GPU offloading. Current AD tools lack support for all these techniques, particularly in the adjoint or reverse mode, which is the basis for many real-world applications.

In this talk, I will discuss some of the technical difficulties that prevent AD tools from adapting to these user needs, particularly for unstructured programs whose memory access pattern is not known statically. Following this, I will present early success stories of code analysis and transformation methods that preserve a given parallelisation during differentiation. The talk will conclude with a roadmap towards AD tools that can handle structured and unstructured multithreaded and vectorised programs in practice.