Why do I get "Access is denied" when re-loading an allready running XBAP?

Question

I'd like to embedd a XBAP in a SharePoint Web Part. My approach is to load the XBAP (on the same server) using an IFrame. If I load the Web Part for the first time, it works. However if I navigate to some other page within SharePoint and return back to my Web Part, I get

"An error occurred in the application you were using"

and

"Access is denied."

To track down the cause of this I created an empty XBAP Project, loaded it in IE and tried to re-load it by focusing the adress bar and pressing enter (NOT with "Refresh(F5)"). By doing this I get the same error message. So it seems to me that this issue is not SharePoint related and you simply cannot load an XBAP more than once during a browser session. If I clear the cache or restart the browser, the XBAP is running again.

Please note that this has something to do with the browser cache, not the ClickOnce application cache (I checked that).

I wonder if someone...

1. is able to reproduce the error.
2. has an explanation what the cause of this error is.
3. has a workaround/solution for this.

BTW: Silverlight is no option. I need full access to the file system.