Parallel programming, are we not learning from history again?

Question

I started programming because I was a hardware guy that got bored, I thought the problems being solved in the software side of things were much more interesting than those in hardware. At that time, most of the electrical buses I dealt with were serial, some moving data as fast as 1.5 megabit!! ;)

Over the years these evolved into parallel buses in order to speed communication up, after all, transferring 8/16/32/64, whatever bits at a time incredibly speeds up the transfer. Well, our ability to create and detect state changes got faster and faster, to the point where we could push data so fast that interference between parallel traces or cable wires made cleaning the signal too expensive to continue, and we still got reasonable performance from serial interfaces, heck some graphics interfaces are even happening over USB for a while now.

I think I'm seeing a like trend in software now, our processors were getting faster and faster, so we got good at building "serial" software. Now we've hit a speed bump in raw processor speed, so we're adding cores, or "traces" to the mix, and spending a lot of time and effort on learning how to properly use those. But I'm also seeing what I feel are advances in things like optical switching and even quantum computing that could take us far more quickly that I was expecting back to the point where "serial programming" again makes the most sense.

What are your thoughts?

Answer 1

Things like optical switching and quantum computing have been in the wings for years; hopefully, they will eventually come to fruition, but progress is usually a gradual process. There's no reason for research to go in only one direction when it can go in many, and as processors have hit their limits, parallelism seems like one promising approach to increasing computing efficiency.

Answer 2

I think your point is very interesting.

However it seems to me that it boils down to to "eventually we manage to make fast serial hardware faster". That might be true. Nevertheless, in my experience the cost and time to improve a technology by adding multiple units of something rather than moving to the next generation is still much lower. So I believe that with every "generation change" we would still go through a period where advancement is achieved by the "multi" approach rather than the "neo" approach. In fact, more time is spent in the multi approach that investing in parallelization makes sense.

Furthermore, we have learned with time. Look at graphics cards. They are essentially a bulk set of cheaper number crunchers. Graphic programming is already making use of this. Eventually, more calculations will be done this way for regular development.

Also, if in the past many developers never understood threading or it was some advanced concept, today most C# and Java programmers encounter it early in the education. It's becoming more important early, and that means that with time we will have more people able to write software that makes use of this. Fifteen years ago, very few job interviews asked developers any questions about concurrency. Today, most screening interviews include at least rudimentary threading questions, and almost every developer knows what a deadlock is and why the philosophers might starve.

Beyond all this, we can't ignore the cloud. Even if we can change the way that a single machine is implemented, we can't deny that the network, and the ability to draw in additional resources as we need, create an inherent need for parallel software. More and more software, and the whole model of Software as a Service, is heading that way.

On an unrelated note, I'm not sure I share your optimism on quantum computing. We haven't managed to make reliable software in this universe with deterministic means :)

Answer 3

I Think its a hardware software cycle, At one point the development in hardware stops for sometime and then the world looks towards the software developers to get better performance out of the existing hardware.The software engineers become important at this point. However after sometime the intelligent electrical engineers find a way around the stalling factors in hardware development and then hardware development starts again with a faster pace allowing the software engineers to relax a bit.

Answer 4

I think we are a long ways off from quantum computing or optical switching being relevant.

I see multi-core programming as a very important problem that needs to be solved. These new hybrid functional languages like scala and clojure are a big step in the right direction, but we still have a lot of work to do.