I'd say one of the big disadvantages of using In_Proc is that session state can be lost if the app pool or domain is recycled. This can happen any time, for instance if the server is low on memory etc. I personally never rely on In_Proc session for anything you don't want to lose. I've spent hours debugging sites with sporadic problems only to find it was because session state was being lost due to a server that was low on resources recycling (and, of course, the more you store in session the lower the more server resources you use up. Remember, if it can go wrong then it probably will go wrong at some point!
That's why I now normally use State Server for anything but trivial session data. The only real disadvantage I've found is you need to mark classes as serializable, but this is usually trivial. It's also a bit slower, too, but that is negligible in most cases.
There's a good article about this on the IIS MSDN blog.