What does robots.txt file do in PHP project?
+2
A:
Not really PHP, see:
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
Sarfraz
2010-05-26 08:48:30
+2
A:
robots.txt is related to search engines robots and not specific to PHP projects. it contains instructions for robots that crawl you website such as which urls you dont want to be crawled by the search engines and so on.
see http://www.webconfs.com/what-is-robots-txt-article-12.php
Sourabh
2010-05-26 08:49:19
A:
robots.txt lives in the / of your website.
When a search engine bot finds your website it first requests the robots.txt to find out which URIs it CAN and CANNOT index.
This is to prevent (it's a CONVENTION, so robots CAN ignore it) certain parts of your website from popping up in Google / Yahoo (for example your shoutbox...)
More info http://en.wikipedia.org/wiki/Robots_exclusion_standard
Snake
2010-05-26 08:49:25
Why on earth is this voted down?
Snake
2010-05-26 08:54:25
-1 because it doesn't really allow or forbid robots to do anything.It's equivalent to hanging a sign on your door with the words "robots should die" on your front door. This will of course have horrible repercussions once the robot revolution starts, but meh...
ivans
2010-05-26 08:55:00
+2
A:
It is used to instruct webcrawlers like google and yahoo to crawl only parts of the page or not at all. See http://www.robotstxt.org for more information.
Rune Hammerskov
2010-05-26 08:54:06