How do you dynamically edit robots.txt in a load balanced environment?

Question

Looks like we are going to have to start load balancing our webservers here soon.

We have a feature request to edit robots.txt dynamically which is not a problem for one host -- however once we get our load balancer up and going -- it sounds like I will have to scp the file over to the other host(s).

This sounds extremely 'bad'. How would you handle this situation?

I already let the client edit the meta tag 'robots' which (imo) should effectively do the same thing as he wants from the robots.txt editing but I really don't know that much about SEO.

Maybe there is a completely different way of handling this?

UPDATE

looks like we will store it in s3 for now and memcache it frontside...

HOW WE ARE DOING IT NOW

so we are using merb..I mapped a route to our robots.txt like so:

match('/robots.txt').to(:controller => 'welcome', :action => 'robots')

then that relevant code looks like this:

def robots
  @cache = MMCACHE.clone
  begin
    robot = @cache.get("/robots/robots.txt")
  rescue
    robot = S3.get('robots', "robots.txt")
    @cache.set("/robots/robots.txt", robot, 0)
  end
  @cache.quit
  return robot
end

Answer 1

I might have the app edit the contents of robots.txt and have the user input saved to a database. Then at certain intervals, have a background process pull the latest from the DB and push to your servers.

Answer 2

An alternative would be to have the reverse proxy that is doing your load balancing treat robots.txt differently. You could serve it directly from the reverse-proxy or have all requests for that file go to a single server. It makes a lot of sense since robots.txt is going to be required relatively infrequently.

Answer 3

I'm not sure if you're home on this yet. If so ignore. (UPDATE: I see a note to your original post, but this may be useful reagrdless.)

If you mapped a call to robots.txt to an http-handler or similar, you can generate the response from say a dB.

Answer 4

serve it via whatever dynamic content generation you are using. its just a file . nothing special.