NDAK14008U Programming Massively Parallel Hardware (PMPH)

In simple words, the aim of the course is to teach students how to write programs that run fast on highly-parallel hardware, such as general-purpose graphics processing units (GPGPUs), which are now mainstream. Such architectures are however capricious; unlocking their power requires understanding their design principles and also specialized knowledge of code transformations, for example aimed at optimising locality of reference, the degree of parallelism, etc. As such, this course is organized on three tracks: hardware, software, and lab.
 
The Software Track teaches how to think parallel. We introduce the map-reduce functional programming model, which builds programs naturally, like puzzles, from a nested composition of implicitly-parallel array operators, which are rooted in the mathematical structure of list homomorphisms. We reason about the asymptotic (work and depth) properties of such programs, and discuss the flattening transformation, which converts (all) arbitrarily-nested parallelism to a more-restricted form that can be directly mapped to the hardware.   We then turn our attention to legacy-sequential code written in programming languages such as C.  In this context we study dependence analysis, as a tools for reasoning about loop-based optimizations (e.g., Is it safe to execute a given loop in parallel, or to  interchange two loops?). As time permits, we may cover more advanced topics, for example related to dynamic analysis for optimising locality of reference.

The Hardware Track studies the design space of the critical components of parallel hardware: processor, memory hierarchy and interconnect networks.  We will find out that modern hardware design is governed by old ideas, which are merely adjusted or combined in different ways.

The Lab Track applies the theory learned in the other tracks. We will review the fundamental ideas that govern the GPGPU design and potential performance bottlenecks.  We will quickly learn several parallel-programming models, and we will get our hands dirty by putting in practice the optimizations learned in the software track.   We will use (the in-house developed) Futhark to write nested-parallel programs, to demonstrate flattening, and as a baseline. We will use OpenMP and CUDA to write "parallel-assembly" code for multi-core and GPGPU execution, respectively.