can i use robots.txt while handling my site with htaccess

Question

I am using htaccess in my site, such that all the request to my site will be redirected to index page in my root directory. No other file in my site can be accessed because my htaccess will restrict it. My doubt is, when I use robots.txt file, will the search engines be able to reach the robots.txt file in my domain?. Or must i modify my htaccess file to allow the search engines to read the robots.txt file. If so help me in finding that specific code for htaccess.

Answer 1

I suppose you're using some sort of rewriting. You can exclude files from beeing processed by mod_rewrite with the following rule:

RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^(.*) $1 [L]

In this case all files that do exist in the webroot and are called directly will not be processed any further. Replace _%{REQUEST_FILENAME} with robots.txt and you should be fine.

Best wishes,
Fabian

Answer 2

How about trying if you can yourself access the robots.txt via a web browser? If you can, then the search engines can, an vice versa (unless you're using some IP- or browser-specific redirections or such). That is, http://yoursite.com/robots.txt