We will discuss some exciting new developments in the studies of high-order tensors and their coordinate-dependent form, hypermatrices. We argue that a subject that parallels matrix theory and numerical linear algebra is emerging from these recent developments. Examples will be drawn from areas of interest to this audience: physics (elementary particles, Yang-Baxter equations), chemistry (fluorescence spectroscopy, density matrix renormalization group), and optimization (self-concordance, higher-order optimality conditions).