robots.txt in ASP.NET MVC app returning 404

Question

I've deployed an ASP.NET MVC site with a robots.txt file in the root but when I try and access this url it's returning 404. I have an .xml and a .html file in the root as well and they return valid content.

I'm running this on IIS 6. I've tried adding .txt as "text/plain" to the MIME Types but no luck.

Suggestions?

Answer 1

Did you look the extension mapping inside IIS? File extension can be associated with handlers and it might be possible that it's mapped to ASP.NET dll. Make sure that robots.txt is also located at the root of your website.

Answer 2

Check your web.config. It could be blocking access to it, in the same way the web.config in the /Views folder blocks direct access to all the View Pages.