My guess is that the application pool's worker processes are set to recycle at a certain time every night in IIS, which causes the delay first thing in the morning. Right-click the application pool in IIS -> Properties -> and look under where it says "Recycle worker processes at the following times:" to see for yourself.
You could disable this option to fix your problem but I don't recommend it because having the worker processes recycle every night will reclaim any memory that may have leaked. Memory leaks are especially possible during SharePoint development because as you may know many of the objects in the SharePoint object model perform most of their work in unmanged memory. If these objects are not properly disposed of a large amount of memory may be occupied while adding little memory pressure on the .NET garbage collector, delaying garbage collection.
Best Practices: Using Disposable Windows SharePoint Services Objects
I agree with seanb's suggestion of setting up a scheduled task to load the site in the morning to solve the delay problem. Just make sure you schedule it for a time after the worker processes have recycled obviously.