After struggling with a particular problem or bug in some part of my code for hours, without getting anywhere, I often get a sudden revelation as soon as I try to explain the problem to one of my coworkers, or while formulating it in writing for posting to some forum.
Does this kind of experience have a name?
Where can I read more about it and how to train it? Do any of you use this consciously in your day-to-day work?
I would say the nearest is "Serendipity", although it is open to discussion. There is no way to train it but to actually use it, and it's not surprising at all. Discussion is central to clarify a point, and analyzing the issue from a different angle is the key to understand critical points.
It's why my first resort is pencil and paper, it usually falls into place there. Then my "rubber duck" is a basic process description in comments in code.
To be honest, I think it partially works because you're not actually 100% thinking about it, you're thinking about talking, you're not looking at the screen any more. It's a break. It's like thinking up solutions on the toilet to something you've been fighting with for a couple of hours.
Considering this happens to people in all sorts of fields, referring to it with a programming term seems quite silly to me.
Explaining something forces you to think about it. You may THINK you were thinking about everything before, but until you've explained it, you may be glossing over something.
I don't see any reason to believe this is serendipitous or not trainable. You've already seen it work, and there are countless stories of it working for other people. It sounds proven to me, so just use it. As with everything else, I imagine practice will make perfect.
Some of the people I've been explaining to when the moment happens would not appreciate being considered Rubber Ducks, but thank you for bringing the term to my attention.
Next time it's some Sr. VP that I'm explaining to, I'll think of him as a rubber duck.
Another effective (debugging) technique is to explain your code to someone else. This will often cause you to explain the bug to yourself. Sometimes it takes no more than a few sentences, followed by an embarrassed "Never mind, I see what's wrong. Sorry to bother you." This works remarkably well; you can even use non-programmers as listeners. One university computer center kept a teddy bear near the help desk. Students with mysterious bugs were required to explain them to the bear before they could speak to a human counselor.
B - Kernighan & D. Pike (in "The Practice of Programming" pp. 123)
I always liked that passage. It lead to me talking often in several teams about "teddy bear management". One boss even brought in a large stuffed animal. As silly as you might feel explaining the bug to a soft toy... it does work.
I've found that you never really understand something until you have to teach it to someone else. I've never thought of giving this a name before so I'm happy to jump on the rubber duck bandwagon. I wouldn't call it serendipity (or similar) though. I don't think it's accidental or lucky. It seems to be a reasonably repeatable phenomenon.
Cardboad cutout programmer. Have one by your desk at all times.
The act of explaining with no actual input from the listener is sufficient.
This is different from rubber ducking, and from taking a shower, though both work too!
Stackoverflow also works nicely as a Rubber Duck. I've had numerous times where I had a problem, started writing a question here and thought of the solution while writing it. And if you haven't found the solution by the time you've written your question, just submit it :)
To answer the second part, yes I think you can learn this process. I often find myself stuck on a problem. It seems I've read over every line of code and verified every line is doing exactly what it's supposed to do but I can't solve the problem. It's kind of like proof reading grammar. You tend to read over the mistakes because you know what it's supposed to say in your head.
I work from home so my co-worker includes my unfortunate girlfriend and I do this to her all the time. Usually one of two things happens. Either her simple questions lead me to dig deeper while explaining and I eventually figure out whats wrong or just the process of actually talking about it and explaining every step out loud helps to find what I was glossing over in my head. Then in mid sentence I go OOH! and stop talking while she rolls her eyes and goes back to what she was doing originally.
The hardest part about working this method into your thought process is that, for me anyway, when I'm stuck on something I tend to go over what I'm looking at over and over again the same way without realizing how long I've been sitting there staring at the screen. You have to recognize when you're stuck and learn to step back and work through it slowly from beginning to end.
One of my lecturers at University called this the "nodding bear". He said that every development team should have a nodding bear because it will solve 90% of your problems very quickly.
The idea is simple. Somewhere in the office you should install a stuffed toy (the bear) that does nothing but nod when someone talks to it. When you have a problem you consult the nodding bear before disturbing any of your colleagues - and hopefully before you spend hours staring at your code. Nine times out of ten the bear will reveal the answer to you without even saying a word!
I believe the Nodding Bear pre-dates the publication of the Rubber Duck in The Pragmatic Programmer. My memory of Uni isn't that good, but I think Terry Woodings should be credited with the Nodding Bear.
I suggest you practice this technique often. I like the form where you start an email to other developers describing the problem - if you find the solution by writing the email, then complete and send the email with the solution as this is useful (or at least entertaining!) for everyone to read.
In Code Complete, it's also referred as “confessional debugging”.