Local ASP.NET MVC Suddenly Very Slow; Load times > 1 minute

Question

Over the last few weeks I've been subject to a sudden and significant performance deterioration when browsing locally hosted ASP.NET 3.5 MVC web applications (C#). Load times for a given page are on average 20 seconds (regardless of content); start up is usually over a minute. These applications run fast on production and even test systems (Test system is comparable to my development environment).

I am running IIS 6.0, VS2008, Vista Ultimate, SQL2005, .NET 3.5, MVC 1.0, and we use VisualSVN 1.7.

My SQL DB is local and IPv6 does not seem to be the cause. I browse in Firefox and IE8 outside of Debug mode using loopback, machine name, and 'localhost' and get the exact same results every time (hence DNS doesn't seem to be the issue either).

Below are screen shots of my dotTrace output.

http://www.glowfoto.com/static_image/28-100108L/3123/jpg/06/2010/img4/glowfoto

This issue has made it near impossible to debug/test any web app. Any suggestions very much appreciated!

Answer 1

Try creating a new, default MVC2 application in a new web folder. Build and browse it. If your load times are okay with the new app, then there's something up with your application. If not, it's outside of the context of the app and you should start looking at IIS config, extensions, hardware, network, etc.

In your app, back up your web config and start with a new, default web.config. That should disable any extensions or handlers you've installed. If that fixes your load times, start adding stuff from the old web.config into the new one in small blocks until the problem reappears, and in that way isolate the offending item.

I call this "binary search" debugging. It's tedious, but actually works pretty quickly and will most likely identify the problem when we get stuck in one of those "BUT IT SHOULD WORK!!!" modes.

Update Just a thought: to rule out IIS config, try running the site under Cassini/built-in dev server.

Answer 2

You could download Fidler to measure how long each call takes and get some measurements.

http://blogs.msdn.com/b/tess/archive/2009/11/06/recap-of-oredev-and-some-net-debugging-videos.aspx

This video might help...

Answer 3

Surely the big red flag on that profiler output is the fact that AddDirectory is called 408 times and AddExistingFile is called 66,914 times?

Can you just confirm that there's not just a shed load of directories and files underneath your MVC app's root folder? Because it looks like the framework is busying itself trying to work out what files it needs to build (or add watches to) on startup.

[I am not au fait with MVC and so maybe this is not what is happening but 67k calls to a function with a name like "AddExistingFile" does smell wrong].

Answer 4

I've learnt that it's usually a "smell" when things fail near a power of two ...

Given

Over the last few weeks I've been subject to a sudden and significant performance deterioration

and

AddExistingFile is called 66,914 times

I'm wondering if the poor performance hit at about the time as the number of files exceeded 65,535 ...

Other possibilities to consider ...

• Are all 66,914 files in the same directory? If so, that's a lot of directory blocks to access ... try a hard drive defrag. In fact, it's even more directory blocks if they're distributed across a bunch of directories.

• Are you storing all the files in the same list? Are you preseting the capacity of that list, or allowing it to "grow" naturally and slowly?

• Are you scanning for files depth first or breadth first? Caching by the OS will favor the performance of depth first.

Update 14/7

Clarification of Are you storing all the files in the same list?

Naive code like this first example doesn't perform ideally well because it needs to reallocate storage space as the list grows.

var myList = new List<int>();
for (int i=0; i<10000; i++)
{
    myList.Add(i);
}

It's more efficient, if you know it, to initialize the list with a specific capacity to avoid the reallocation overhead:

var myList = new List<int>(10000);  // Capacity is 10000
for (int i=0; i<10000; i++)
{
    myList.Add(i);
}

Update 15/7

Comment by OP:

These web apps are not programmatically probing files on my hard disk, at least not by my hand. If there is any recursive file scanning, its by VS 2008.

It's not Visual Studio that's doing the file scanning - it is your web application. This can clearly be seen in the first profiler trace you posted - the call to System.Web.Hosting.HostingEnvironment.Initialize() is taking 49 seconds, largely because of 66,914 calls to AddExistingFile(). In particular, the read of the property CreationTimeUTC is taking almost all the time.

This scanning won't be random - it's either the result of your configuration of the application, or the files are in your web applications file tree. Find those files and you'll know the reason for your performance problems.

Answer 5

The solution was to format and do a clean install of Vista, SQL Server 2005, Visual Studio 2008, IIS6 and the whole lot. I am now able to debug, without consequence, the very same webapp(s) I was experiencing the problems with initially. This leads me to believe the problem lay within one of the installations above and must have been aggravated by a software update or by the addition of software.