I have changed the structure of the URLs of my site more than 6 months ago. I detect the use of legacy URLs and redirect to the new URL with a 301 status code. I verified with flidder that the status code is correctly returned upon the request. But bots (yahoo slurps, googlebot, etc.) are still hitting the old URLs. Is there something I am missing?
+2
A:
No, just it takes a very, very long time for crawlers to get the message. I have bots crawling addresses that have not existed since 2005 - when folk harp on with addresses being permanent, they really are.
Additionally, depending on how your URL's are structured, you can disallow the old addresses with robots.txt
squeeks
2009-10-16 13:48:04
+1
A:
If external sites have linked to your old pages and those links are still accessible for bots, the bots will keep coming and try to access the content.
innaM
2009-10-16 13:55:09