14:332:451 Introduction to Parallel and Distributed Programming

Course catalog description: Parallel and distributed architectures, fundamentals of parallel/distributed data structures, algorithms, programming paradigms, introduction to parallel/distributed application development using current technologies.

Credits and contact hours: 3 credits; 1 hour and 20-minute session twice a week, every week

Pre-Requisite courses: 14:332:331, 14:332:351

Co-Requisite courses: None

Topics Covered:

  • Introduction to Parallel and Distributed Programming (definitions, taxonomies, trends)
  • Parallel Computing Architectures, Paradigms, Issues, & Technologies (architectures, topologies, organizations)
  • Parallel Programming (performance, programming paradigms, applications) 
  • Parallel Programming Using Shared Memory I (basics of shared memory programming, memory coherence, race conditions and deadlock detection, synchronization)
  • Parallel Programming Using Shared Memory II (multithreaded programming, OpenMP, pthreads, Java threads) 
  • Parallel Programming using Message Passing - I (basics of message passing techniques, synchronous/asynchronous messaging, partitioning and load-balancing)
  • Parallel Programming using Message Passing - II (MPI)
  • Parallel Programming – Advanced Topics (accelerators, CUDA, OpenCL, PGAS) 
  • Introduction to Distributed Programming (architectures, programming models)
  • Distributed Programming Issues/Algorithms (fundamental issues and concepts - synchronization, mutual exclusion, termination detection, clocks, event ordering, locking)
  • Distributed Computing Tools & Technologies I (CORBA, JavaRMI)
  • Distributed Computing Tools & Technologies II (Web Services, shared spaces)
  • Distributed Computing Tools & Technologies III (Map-Reduce, Hadoop)
  • Parallel and Distributed Computing – Trends and Visions (Cloud and Grid Computing, P2P Computing, Autonomic Computing)            

Textbook: Peter Pacheco, An Introduction to Parallel Programming, Morgan Kaufmann.

 Other supplemental material: 

  • Hariri and Parashar (Ed.), Tools and Environments for Parallel & Distributed Computing, John Wiley.
  • David Kirk, Wen-Mei W. Hwu, Wen-mei Hwu, Programming massively parallel processors: a hands-on approach, Morgan Kaufmann.
  • Kay Hwang, Jack Dongarra and Geoffrey C. Fox (Ed.), Distributed and Cloud Computing, Morgan Kaufmann.