Ok, bare with me here as I explain my convoluted problem...
I run a video site, and I have setup custom SEO friendly permalinks to to something like this:
/tv/view/352/title-of-video
The number is the ID of the video and that's all the PHP script needs to fetch it. The "title-of-video" is completely superfluous and is just there for extra SEO points. It can be anything and the same video will load as long as the URL is correct.
A few weeks ago, I updated my design and in my nav I forgot to set a few relative paths.
Seeing as though I have over 4000 videos, that means for every video, MSNBot crawled an extra 10 non-existent pages which all loaded the same video page. This resulted in incredibly high CPU usage.
I have tracked the problem and I am trying to block MSNBot. I have updated my robots.txt, and I have even tried to block the IP in htaccess but I am still seeing hits to these fake pages from MSNBot. I'm assuming it's using a cached version of my robots.txt file. Is there any way for me to force to stop?