I've had a few cases where I had to remote onto a machine, usually for debugging something that I can't do on my local machine like having 2 websites with Windows 2000 Professional or there is some software that has to be installed that is more of a pain to install locally than work with on the server like a CMS or ran the middleware of the software.
For myself, the biggest problem is that things may be in a different place on the server than my local machine which has its good and bad advantages. Another point is having different IIS versions between the machines. The Web Server 2008 has some nice features that make it similar to Vista in explorer which is a bit of good and bad, IMO. Also the wallpaper, quick launch buttons, and a few other personal touches are usually minimized in this situation.
I'd say it was OK to do for a day or two, but much more than that and it gets ugly. I remember a story of a former co-worker where multiple developers had to share the same server for development. I haven't seen anything that bad in my years.