+1  A: 

My gut feeling is some sort of file locking issue. Have you tried running Process Monitor and having a look at the trace when this error occurs? Be warned that running it may sap some resources on your server.

To use, follow these steps. (You may also find Mark Russinovich's blog useful.)

  1. Set the filter so that it only displays the w3wp.exe process. This will reduce noise. The simplest way is to right-click on 'w3wp.exe' in the list and select Include 'w3wp.exe'.
  2. The monitor should have automatically started so now just wait until you receive the ASP.NET error. Make a note of the time the error occurs! It may also be reported in the SharePoint ULS log.
  3. Stop Process Monitor from capturing events (File, Capture Events) and narrow down to the time of the error.
  4. Look for the word 'FAIL' or 'ERROR' (you can use the program's find feature) and see if you can work out what's happening. You may need to remove the filter as the problem might have occurred with another process besides w3wp.exe.

There may also be interference with an anti-virus program as I've seen these lock files and cause problems on SharePoint before. Can you try disabling it to check if the error disappears? Alternatively, here are details of locations that need to be excluded so that SharePoint can operate correctly with a file-level virus scanner. You may need to add others, I don't believe that list is comprehensive.

Alex Angas
There is antivirus running on this machine, although looking at it's config it seems to be explicit in the folders it blocks. If i run process monitor what type of event should i be looking for?
Rob
I've added a lot more detail to the answer now. Hopefully this helps.
Alex Angas
excellent stuff, thanks. The error usually happens overnight, so i'll start in this evening.
Rob
A: 

I can see why the app pool recycle works, the dll's get recopied to the Temporary ASP.NET files folder.

The question here is, why are they not there anymore? It could be that the files were locked when the app pool recycled overnight, the files were not overwritten with a new version and remain in a locked state. Maybe search crawl was running when the app pool tried to recycle, visiting that page at the time?

P.S. Is your custom page build as part of a "Visual Studio Web Site" or a "Visual Studio Web Site Project"?

Colin
The visual studio project is a class library copied to the bin folder. I created the pages manually in sharepoint desinger and 'hooked' them up to the code behind in the assembly. (I'd love to change this but have now run out of time).I've also removed the content source from the crawl
Rob
Could you try adding the dll to the GAC?
Colin