I used it for a while, but generally didn't get very good results for moderate sized projects (things that take less than 5 minutes to build on a 4-core Mac Pro). The most noteworthy gotcha is that when I set up Bonjour discovery of available machines, the current machine (i.e. the one I'm kicking off the build from), didn't get included as one of the available build machines. Since I generally sit at a 4-core Mac Pro and was trying to leverage smaller boxes (Macbook Pro, Mac Mini, that sort of thing), it dramatically slowed my build time because the biggest box wasn't helping. The solution was to manually add all the machines, but then I ran into some slow downs if any of the machines weren't online (which was common for me).
Most frustrating was that iPhone projects couldn't be compiled in on Xgrid at all when I last checked (haven't checked in the 3.0 SDK time frame, so this may have changed). Worse, requesting distributed compilation actually caused errors, rather than just being ignored. So I had to switch my configuration back and forth when switching between Mac and iPhone. I do this a lot, so this became a serious headache.
In the end it was much more trouble than it was worth for me, particularly because my primary machine is so much bigger than all the other machines, and the overall build time for most of my projects is small on a Mac Pro. I might feel different if my primary machine were a laptop.