Hi,
From my previous posts, I come to know that today software industry is moving towards more and more of parallelization and I want to learn and understand more about this technology and what are some of the recommended books/resources for Parallelization ?
Thanks.
I think a great resource is studying at college (Programming), where teachers can guide you in your parallelization tryouts :) and torture you with incomprehensible math verifications :)
if you believe google and their awesome technologies, there is a paper discussing MapReduce http://code.google.com/intl/en-US/edu/parallel/mapreduce-tutorial.html
understanding parallelization mostly means understanding OS resources or your Language's api for threading, for complex structures a lot of math. So maybe books discussing specific languages and their multitheading support?
I'm very fond of Design patterns, and this book looks as what I would like to read: Patterns for Parallel Programming
The Art of Concurrency, by Clay Breshears is well written and digs into a lot of details.
If you are developing on Windows, Concurrent Programming on Windows, by Joe Duffy, is well worth reading.
The Little Book of Semaphores, by Allen B. Downey (free PDF download!) is a very practical and useful book containing many concurrency patterns and how to implement them correctly. If nothing else, it has some great reference material. Note that this book doesn't cover the basics; it's almost entirely dedicated to explaining various concurrency patterns via short code examples. It's also quite funny.
Technical Books for Multi-Core Software Developers is a great list of books on parallel programming that are recommended by Intel. The list includes:
It's available at http://softwarecommunity.intel.com/articles/eng/1567.htm with a short description of each book.
You don't say what language you're working in. If it's Java, I highly recommend Java Concurrency In Practice. Not only is it great at covering the differences between the techniques, but it tells you what are the advantages and traps in real world applications of each.
It really depends on what kind of parallel programming you would like to learn about. It is a vast topic and difficult to master. Parallel programs are difficult to debug and even if you implement one correctly, odds are that the program is slower than a sequential version.
Learning about "parallel programming" in general is almost impossible, although Patterns for Parallel Programming gives a nice introduction. Reading that book will not be enough to learn how to actually write parallel programs though. I would recommend that you pick a sub-topic and stick to that, at least for a while.
These are some interesting sub-topics:
GPU-programming
Message-passing parallel programming in Erlang or using MPI.
Shared memory parallel programming using OpenMP or something similar.
Low-level shared-memory parallel programming using threads and locks. I would not recommend this for production code though, since you are most likely better off using a higher level library. It can teach you a lot though.
For a theoretical mathematical angle, take a look at Principles of Model Checking.
Parallel programming is also a very active research topic, with work being done on Fortress, software transactional memory and many other interesting things. The Glasgow Haskell Compiler includes some very interesting extensions, but I don't think parallel Haskell is something you will see in your average project any time soon.
A word of warning: I decided to learn about parallel programming because I too had heard a lot of hype about it. I have spent a lot of time on the subject and I am still nowhere near an expert. What I do know is that most of your previous programming experience is pretty useless in this domain. Parallelisation is no quick fix and if you are just looking for a quick way to speed up your code, you are probably better off learning more about sequential optimisation techniques.
This is the list of books I have read about parallelism/concurrency ordered by (personal) preference:
I have not read (but I will try ASAP):