Parallel Programming and C++

Question

I've been writing a lot recently about Parallel computing and programming and I do notice that there are a lot of patterns that come up when it comes to parallel computing. Noting that Microsoft already has released a library along with the Microsoft Visual C++ 2010 Community Technical Preview (named Parallel Patterns Library) I'm wondering what are the common parallel programming patterns you have been using and encountering that may be worth remembering? Do you have any idioms you follow and patterns that you seem to keep popping up as you write parallel programs with C++?

Answer 1

First you have to chose between shared-memory computing, and shared-nothing computing. Shared memory is easier, but doesn't scale that well - you will use shared-nothing if you either

a) have a cluster, rather than a multiprocessor system, or

b) if you have many CPUs (say, > 60), and a high degree of non-uniform memory

For shared-memory, the common solution is to use threads; they are easy to understand as a concept, and easy to use in the API (but difficult to debug).

For shared-nothing, you use some kind of messaging. In high-performance computing, MPI is established as the messaging middleware.

You then also need to design an architecture for the parallel activities. The most common approach (again because it's easy to understand) is the farmer-worker-pattern (a.k.a. master-slave).

Answer 2

Patterns:

• Produce/Consumer

  • One Thread produces data
  • One Thread consumes the data

• Loop parallelism

  • If you can show that each loop is independent
    each iteration can be done in a sperate thread

• Re-Draw Thread

  • Other threads do work and update data structures but one thread re-draws screen.

• Main-Event Thread

  • Multiple threads can be generating events
  • One thread has to processes the events (as order is important)
  • Should try separate the Event Thread/Re-Draw Thread
    This (helps) prevents the UI from freezing
    But may cause excessive re-draws if not done carefully.

• Work Group

  • A set of threads waits for jobs on a que.
  • Thread extract one work item from queue (waiting if none is available).
    Thread works on one work item until complete
    Once completed thread returns to queue.