When you experience this behavior, how recently have you done a deployment of your files to the server?
There is a nasty configuration option called numRecompilesBeforeAppRestart on the compilation tag:
<system.web>
<compilation debug="true" numRecompilesBeforeAppRestart="15">
http://msdn.microsoft.com/en-us/library/system.web.configuration.compilationsection.numrecompilesbeforeapprestart.aspx
This value defaults to 15. I've been through an application killing all user sessions before and this was the culprit for me. For roughly a day after a lightly used web application was updated (new files copied to the server; this ended up overwriting EVERY file, numbering into the hundreds), we'd get constant AppDomain restarts evidenced by all Session values for all users disappearing.
I found this bug report listing the behavior I'm experiencing:
http://support.microsoft.com/kb/319947
Here is the really important relevant text to my scenario:
However, this problem occurs when you
load many new .aspx or .ascx files to
the server (for example, 61 files).
The server unloads the application
when the first 15 files are recompiled
and every time another 15 files are
recompiled until the server reaches
61. This results in four application restarts even though only one is
required.
I switched the value to 99999 and the problem went away. This means more memory will build up in my worker process, so I added a daily recycle (3am when my site has no users) to the IIS AppPool recycle settings.