How can one learn multi-threaded parallel programming?

Question

No matter how proficient you are at programming C or Java, you should consider adding multi-threaded programming to your set of skills.

This is NOT something you should try to learn on your own. It is MUCH harder to learn than sequential programming. If you are a Technical Manager, you SHOULD invest in retraining your key staff in multi-threaded programming. You might also monitor the research activities in concurrent programming languages (like those listed above). You can be sure that your competitors will.

This is a quote from this article. I imagine most of us here are very proficient in teaching ourselves different languages, data structures, algorithms, etc, and I do recognize the mental shift that needs to occur to do parallel programming right.

I reject the idea that one cannot learn parallel programming "right" on their own. So what's the most responsible way to teach oneself parallel programming? What books and other resources are recommended?

Edit: Here are some more details. I would be mostly applying these to scientific computing, but I was looking for general, language-agnostic material/advice. I am also looking for a heathy dose of practical theory. Imagine you have an excellent developer who loves math and computer science, but never took a course on parallel programming. Now imagine he has a deadline for a problem (say 1 year), and you have to give him the materials to figure out whether parallelization would be helpful, and how to implement it right. What resources would you give him? That is how I (and I hope other developers) would be interested in learning parallelization/multi-threading.

Answer 1

For Microsoft technology, there is a wealth of information at the MSDN Parallel Computing portal here. You could start out with the Getting Started links.

Answer 2

That article is basically an advertisement for training services. You should treat a salesman's opinion of the value of his own products with a degree of circumspection.

I've no idea how you learned everything else you already know about computers, but if that worked for you I'd stick with the same approach for the next thing you want to learn.

I can't recommend any language/platform agnostic books - I suspect they'd be very academic anyway. If you're actually on .NET, then Jeff Richter writes a lot of good stuff about threading, and I believe the 3rd edition of his C#/CLR book (earlier editions were excellent) has a great deal about parallel programming.

Answer 3

If you pitched up in my workplace and asked that question I'd throw a couple of books at you:

Introduction to Parallel Computing and Parallel Scientific Computing

Your response may well be 'that's not what I want to learn about !' so come back and be a bit more specific in your question and we'll be able to be a bit more specific in our answers.

But the most 'responsible' way to teach yourself this stuff is the same as the most responsible way to teach yourself any programming stuff: get a problem, get a toolbag, get a deadline, and get cracking.

Answer 4

if you read everything Google finds for the stuff below, you'll have a pretty good start. Assuming general IT background, etc. These are not language or OS specific:

Peterson's algorithm atomic test-and-set critical section rendezvous memory barriers lock-free algorithms

The mathematical background of this is probably Petri net.

Read Dijkstra.