This page collects material for my part of the course F21DP --- Parallel and Distributed Technology. Check this outline of the entire course and these learning outcomes for my part of it.
Material for Phil Trinder's part of it can be found here.
Purpose and Learning Objectives
Software is no longer sequential. Many programs are now concurrent, parallel, distributed or mobile. This module explores technologies that enable, the design issues that concern, and the hierarchy of architectures that deliver, distributed and parallel systems.
The parallel strand starts by reviewing basic concepts including brief overviews of the need for parallelism, parallel architectures and programming approaches. The main focus is then on parallel programming models, namely
- MPI with C,
- Glasgow parallel Haskell,
- OpenMP on multicores,
- GPU processing.
Following a review of C and an introduction to essential MPI concepts and constructs, parallel design patterns and their realisations as algorithmic skeletons are explored. The distributed strand also starts by reviewing basic concepts, including concurrency, threads and sockets. Existing technologies covered include fault tolerance, remote evaluation (RPC/RMI), and distributed object-based systems including object request brokers like CORBA. Several emerging technologies are covered, and compared to existing technologies. These include high-level coordination languages, mobility, Clouds and Grids.
Pre-requisites:
- F29OC: Operating Systems & Concurrency
- Solid C programming skills (there will be a quick C revision)
Course Structure
See this course outline.
- Week 1 (HWL): Introducing Parallelism (HWL/PWT); C Revision I; C Revision II;
- Week 2 (HWL): C Tutorial; Communication Libraries (MPI) I; Communication Libraries II;
- Week 3 (HWL): MPI Tutorial I; Parallel Performance Tuning (HWL); Haskell I (PWT);
- Week 4: MPI Tutorial II (HWL); Haskell II (PWT); Glasgow parallel Haskell (GpH) I (PWT);
- Week 5 (PWT): Haskell Lab; GpH II; GpH III;
- Week 6: Programming Week and GpH Lab;
- Week 7 (PWT): OpenMP I; OpenMP Lab;
- Week 8 (PWT): Introducing Distribution & Sockets; Remote Evaluation (RPC/RMI); Concurrent & Distributed Haskell Coursework due
- Week 9 (HWL): Higher-order Parallelism (Skeletons) I; Higher-order Parallelism (Skeletons) II; Warehouse Computing Frameworks I;
- Week 10 (HWL): Hadoop Lab; Warehouse Computing Frameworks II; Vector Processing;
- Week 11 (PWT): Cloud & Grid Computing (GridGpH) Distributed Object-based systems (CORBA) Mobile & Global Computation (mHaskell)
- Week 12 (PWT/HWL): Revision
Assessment:
- Assessed Coursework: 20% (Deadline: Week 8)
- Exam: 80%
Learning Material
Slides (HWL):
- Week 1: Lecture 1: Overview, Lecture 2: C Revision (Part 1) Lecture 3: C Revision (Part 2)
- Week 2: Lecture 4: Communication Libraries: MPI (Part 1) Lecture 5: Communication Libraries: MPI (Part 2)
- Week 3: Lecture 6: Parallel Performance Tuning
- Week 4: slides for Haskell and GpH part of the course are here
- Week 5: slides for Haskell and GpH part of the course are here
- Week 6: Programming week (see coursework below)
- Week 7: slides for Haskell and GpH part of the course are here
- Week 8: slides for Haskell and GpH part of the course are here
- Week 9: Lecture 8: Parallel Program Design Guest Lecture: The PGAS model and UPC, by Michéle Weiland, EPCC
- Week 10: Lecture 9: Algorithmic Skeletons Lecture 10: Data Center and Warehouse Computing
Sample Sources:
- Sources from Lecture 2: C Revision (Part 1)
- Sources from Lecture 3: C Revision (Part 2)
- MPI-style Hello world (same with hostname)
- Sequential matrix multiplication (2D arrays)
- Sequential matrix multiplication (array of vectors)
- Sequential matrix multiplication (using transpose)
- Input matrices: 5x5, 80%, max value 99
- Input matrices: 2000x2000, density 10%, max value 65536
- Parallel matrix multiplication (using transpose)
- Parallel matrix multiplication (send rows once)
- Parallel matrix multiplication (send result by rows)
- Parallel matrix multiplication (arbitrary number of processors)
- Parallel quick-sort
- Parallel bubble-sort
- Parallel bubble-sort (with instrumentation)
- Input: 10000 element list
- Input: 4141382 element list (density: 80%, max value: 65535)
- UPC exercises: (1) start with the hello world example (solutions: 1, 2, 3); (2) next implement the third version of vecadd from slide 33 (solution); (3) implement the maximum from slide 30; (4) if you are ambitious, do exercise 2. See UPC FAQ for more help.
- Hadoop Lab: follow the steps in this tutorial, using one of the bwlf?? machines; for a much more detailed tutorial check this on-line Hadoop tutorial
Coursework
The assessed coursework for 2011/12 can be found here. Submission deadline is Week 8.
Reading List
No single book or web resource covers the course, but recommended reading is associated with many lectures. Some useful links are on my web page on parallel computing.
Specific topics:
- C programming: The standard textbook on C progamming is the one by Kernighan and Ritchie, available from the library. The C book, by Mike Banahan, Declan Brady and Mark Doran, is a completely on-line version of another C introduction.
Other Resources