Two things:
1 - You should not be storing so much in session that managing memory is a concern. I.e., don't store objects in session, store pointers to things, e.g., rather than an instance of User just store the UserID. If you need to, you can add caching of retrieval of info, but that should be in a caching layer separate from sessions.
2 - Use database sessions, so that there is no concern about server memory, and so that you can add more web servers easily if desired (note, StateServer gives you this capability as well). Additionally, this lets you recycle the app pool without users losing their session. This is the main reason I do it - it lets me deploy on the fly.
The reason to treat sessions so delicately, is that they hang around after a user's last request, typically for 10 to 20 minutes. So, requests from many different sessions can use up large amounts of server memory if you are storing large objects in session. Doing even crazier things, like storing database connections in session, can result in you using up all available database connections, just because so many are hanging around in memory waiting for the session to expire.
Ideally, there is no "managing" of sessions required.