I know a lot of developers that struggle to fluently write multi threaded code. They have to spend a lot of time just thinking about the parallelism and what not. It really racks their brains. And even then they don't usually choose the correct lock for the given situation. I.e. a simple Monitor instead of a Reader-Writer...
Is it just a question of time or is it that not everyone will just "get it"?
Is it a matter of getting it to "click" and then you're fine?
It really depends on the tools you're using.
Some languages or tools (like erlang and it's environment) make multi-threading feel relatively intuitive and easy.
In some other languages, it's really hard.
No matter what your tools, you will have to think about your problem in a slightly different way as soon as things can run at the same time. Just like when you're new to GUI programming, you need to think about the problem differently than you do for command-line scripts.
I believe, like almost anything, the more you do it the easier it is. My first multithreaded app took me forever to write and was about as pretty as a pig, but after alot of refining, recoding, reading and asking a thousand and one questions I've become alot more familiar with it.
It's a different mode of thought as opposed to single threaded app, there's alot more to consider and your mind gets tugged in alot more directions. Some people just don't have that train of thought. It's hard to wrap your mind around mutex, locks, joins etc. but with practice it sinks in eventually.
A simple monitor will often perform better than a ReaderWriterLock in .NET. ReaderWriterLockSlim in .NET 3.5 is a lot more efficient, but I'd usually still go with a plain monitor most of the time, for three reasons:
Threading is hard. I'm reasonably competent at it, but if I ever try to write lock-free code or do anything "clever" I have to think really hard - and it's rarely worth it, IMO.
Parallel Extensions in .NET 4.0 should make things a lot easier, and LINQ is at least encouraging people somewhat to write side-effect free code, but I suspect it will never be really simple - at least not in traditional OO languages like C#. Functional languages are more "naturally" parallelizable due to the emphasis on immutability and side-effect free operation.
No offense, but if someone said they found threading in Java or C# natural and easy, I'd still review their code just as thoroughly as someone who struggled with it. The subtleties in things like the memory model are really tough nuts to crack.
I think it's like any other programming paradigm. Some people find some of them easy to understand and others harder
For instance primitive object types are a piece of cake. But other paradigms are a little more difficult to understand like pointers and the differences between ByVal and ByRef. Then maybe you get to linked lists and recursion design or delegates and threading...and so on.
Some paradigms immediately click for some people while not for others. A friend of mine at university couldn't get his head around using boolean variables as flags, which just made sense to me immediately and I'm sure to most people is quite basic, but he grasped pointers in a split second and it took me a few different angles before I found one I understood.
If you have problems with a paradigm find information on it from different authors - how it is explained by one style of writer may make absolutely no sense to you, yet you might stumble across one that makes complete sense in an instant. It was like this for me with delegates...it took me ages to wrap my head around it, but one day someone explained it to me in a way that just made sense and I thought why could nobody have explained it that way the first time? Now it's a piece of cake.
Threading for a C/C++ programmer, much like pointers will probably come far easier than to a VB6 programmer. If you ask a VB6 programmer about pointers or how you spin your process off to a separate thread, they're likely just to stare at you blankly as I would've back then. So the origins of your programming background can help/hinder a lot too.
I don't find multithreading easy at all, and when I try it I usually screw it up royally. Fortunately, I rarely encounter programming situations where multithreading is necessary or even desirable. It's far more common to see threads used where they aren't necessary than it is to see threads not used where they are necessary.
Threading is one of those things where you can tell whether you're talking to an expert on the subject, because they say "It's really hard." This is something Herb Sutter has often alluded to. He estimates that the majority of threading code running in the world today is riddled with subtle bugs with a low-but-non-zero probability of causing a deadlock or data corruption, and I think he's probably right.
The biggest area of difficulty is where "experts" start to wonder if there's a way to avoid locking by taking advantage of atomic operations to write "lock free" code. It's possible, but it's a totally black art. See this - and bear in mind that Andrei Alexandrescu is a God of C++, and he didn't spot the problems.
Also check out the saga of double-checked locking in Java.
Multi-threading is a great way to massively increase the effective complexity of your program, instantly making it hundreds of times harder to see all your mistakes. C. A. R. Hoare said something that I think applies here: "There are two ways of constructing a software design; one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." Although I worry sometimes that the sarcasm-impared may misunderstand it... he's saying that the second method makes it easy to fool yourself that you've got it right. Multi-threaded programs are an extreme case of this, because a totally broken multi-threaded program looks a lot like a correct single-threaded program.
As they say, practice makes perfect. I don't think I have a 'click' moment. But after a few years of writing/debugging MT code, I find it's quite intuitive as life itself, which is definitely multi-threaded :) The quality (less debugging necessary) of my MT code got markedly better when I adopted RAII (scoped_lock/monitor and friends) religiously and understood conditional variables better.
My current related interest is lock-free data structures, which I find very clever after dealing with mutex all these years.
Well I'm very much a n00b when it comes to multi-threading, and I'm only using it with the .NET BackgroundWorker class, which does most of the difficult stuff for you.
But already it's clear that there's more to think about.
Once upon a time, I wrote batch programs. No events, everything happened in a nice clear order.
Then I progressed to writing Windows UIs, where you have to think about things happening in response to events. It's harder, but at least only one thing happens at a time.
Now I'm doing that, but with two threads executing (the UI thread, and the background one - only one of those, fortunately!).
Inevitably, it takes the complexity level a bit higher, but gets easier with practice.
Multithreading is really hard. I find two things help:
Push as much synchronization as possible into atomic data structures. Two examples: an unbounded queue with atomic put and get operations, and Arvind's MVar which is a single cell (empty or full) also with atomic put and get operations.
Validate your synchronization code with a tool like SPIN.
These aren't magic solutions but they're often better than trying to program directly with locks or semaphores.
Shared-memory imperative multithreading is hard, very hard. So hard that the only way to do it right is to avoid it as much as possible, i.e. religiously separate the threads, have as little contact between them as possible and make very, very sure you got those points of contact right.
Anyone who says that multithreading is easy is either deluding himself, or working in a pure functionaly and/or shared-nothing environment like Erlang, where threads are completely separate per default and communication between them is through explicit message queues.