I am trying to deter bots from surfing my pages. So I want to force a login from all Ip Address that are not the top 4 search engines. Is this possible?
+4
A:
Have you considered using a robots.txt file to minimize unwanted traffic from automated spidering tools?
You can have multiple Disallow lines for each user agent (ie, for each spider). Here is an example of a longer robots.txt file:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
User-agent: Googlebot-Image
Disallow: /
Here is an example that disallows everything except google
User-agent: *
Disallow: /
User-agent: Googlebot
allow: /
A word of warning: This method isn't guaranteed to stop disallowed agents from going through your site, it just asks them nicely in a standardized way that most of these tools understand.
JohnFx
2009-10-07 19:11:50
He is looking to ALLOW robots and DENY others. This will not help.
Mike Daniels
2009-10-07 19:13:56
@Mike: Read again. He clearly states " force a login from all Ip Address that are NOT the top 4 search engines." Basically, he wants to only allow certain "bots"; which is covered in the robots.txt file
Chris Lively
2009-10-07 19:23:44
I think the question here is ambiguous as to the aim. I read it as saying he wants to limit access to bots that are NOT the top 4 search engines, but it could just as easily be read as saying any access that is not from the top 4 search engines should require a login.
Paul Whitehurst
2009-10-07 19:24:45
Just trying to give him some helpful information that might solve the underlying problem, even if it doesn't address the question exactly the way he asked it.
JohnFx
2009-10-07 19:30:38
Question is clear. He wants login to access his pages for clients from any ip address except 4 know.
rjoshi
2009-10-07 20:49:40
I have asked the question here in a clear way here: http://stackoverflow.com/questions/1534565/how-to-monitor-genuine-page-hits
Luke101
2009-10-07 22:28:39
You guys can close this question.
Luke101
2009-10-07 22:35:05
A:
Yes. You can force login for all your pages except for few ip addresses. It depends on what web server are you using and what kind of access control allowed?
rjoshi
2009-10-07 19:18:16
A:
if you want forced logind for everyone except the top 4 search engines, you could ask for login from everyone except the top 4 search engines or auto login them
sergiogx
2009-10-07 19:21:43