Deny bots to download my files

Question

I have a asp.net download page which send a file to client but I want to deny robots download this file because the file is large and as I can see from the records a bot downloads this file about 20 times. This is slowing down the server and causes bandwidth consumption.

I coded this page to count downloads and detect .net framework of the client so I can post a setup file containing .net framework or not.

I need somehow to deny Google and other bots to reach this page.

At this page is my download link. download.aspx?pack=msp

Answer 1

You want a robots.txt file. For example:

User-agent: *
Disallow: /download.aspx

This doesn't forcibly block search engines, but most (including Google) will check for a robots.txt file and follow its instructions

Answer 2

Yes, add a robots.txt file to your site. It should contain a list of rules (suggestions really) how spiders should behave.

Check out this article for more info. Also for kicks, this is the robot.txt file used by Google.

Answer 3

The correct answer, as noted by the other two people, is to created a robots.txt file to make well-behaved robots not download things.

However, it is important to know that not all robots are well-behaved, and that robots.txt is only advisory. If you have pages which are not publicly linked, do not list them in robots.txt to "protect" them as some particularly badly-behaved robots actually scan the file to see what interesting URLs there may be that they don't already know about.

Answer 4

In lieu of a robots.txt file, where it isn't possible you can decorate your pages with a <meta name="robots" content="noindex"> tag.

• Again, as Donnie mentioned, this is just a recommendation for bots and there is no requirement to follow it.

• Implement a CAPTCHA method that provides a login mechanism to allow desirable users to access a protected folder where you keep your biggest files.

• Instead of providing direct links to content that is easily parsed by bots, use Javascript on your download link to redirect your users. Many bots won't execute javascript, though bot obfuscation is often a moving target.