I have just stumbled upon Bad Behavior - a plugin for PHP that promises to detect spam and malicious crawlers by preventing them from accessing the site at all. Does something similar exist for ASP.NET and ASP.NET MVC? I am interested in blocking access to the site altogether, not in detecting spam after it was posted. EDIT: I am int...
What .htaccess rewriterule should i use to detect known bots, for example the big ones: altavista, google, bing, yahoo I know i can check for their ips, or hosts, but is there a better way? ...
Hi, I have a site where users can enter their profile and password-protect certain details. I would like search engines to crawl the 'unprotected' parts of the profile (which varies from user to user). Similar to how if you enter a user's name in facebook, their Facebook profile comes up in the search results. Do I have to do anything ...
we have a situation where we log visits and visitors on page hits and bots are clogging up our database. We can't use captcha or other techniques like that because this is before we even ask for human input, basically we are logging page hits and we would like to only log page hits by humans. Is there a list of known bot IP out there? ...
Hi all, I'm building an application that is a kind of registry. Think about the dictionary: you lookup for a word and it return something if the word is found. Now, that registry is going to store valuable informations about companies, and some could be tempted to get the complete listing. My application use EJB 3.0 that replies to W...
I would like to ask the proactive (or paranoid;) among us: What are you looking for, and how? I'm thinking mainly about things that can be watched for programaticaly, rather than manually inspecting logs. For example: Manual/automated hack attempts Data skimming Bot registrations (that have evaded captcha etc.) Other unwanted beh...
This is a general question about writing web apps. I have an application that counts page views of articles as well as a url shortner script that I've installed for a client of mine. The problem is that, whenever bots hit the site, they tend to inflate the page views. Does anyone have an idea on how to go about eliminating bot views ...
Ok, I understand the Title didn't make any sense so here I've tried to explain it in detail. I'm using a hosting that gives me space for my domain and lets me "add on" other domains on it. So lets say I have a domain A, and I add on a domain B. Basically my hosting gives me a public_html where I can put stuff that shows when someone vis...
I created a simple xmpp bot in PHP which connects to google talk server. I basically modified the cli_longrun example. When I run the script in browser the bot comes online and stays online for a while even after I close the tab on which the script was running (as it is just an infinite loop listening for events on the stream). But after...
We have a Business Listings directory hosted on IIS 6 Windows 2003. Our competitors crawl and steal our content and customers. We have tried IP blocking using honeypot URLs and log parsing without much success. Is anyone aware of a network device or a proxy server that I can run in front of my web server to minimize this issue? All su...
I am doing my own visitor tracking with special features that Google Analytics (nor any other) can provide me as it is customized. I was calling this function near the end of my script, but quickly ran into our clients running into thousands of pages being called from bots (I assume Google), and my table filled up with around 1,000,000 u...
what can mirc bots do and not do give some examples ...
I am a web developer for a web site that is occassionally plagued by form bots. Recently I received an error notification of a problem with the form submission that should be impossible for a human user. You cannot submit the form without JavaScript enabled but the server side script received a form field value that the JavaScript valida...
The idea is to develop my own bot to test the game's behavioral rules, in my case I have set on using Unreal Tournament (1999 or 2004 version) as a proof of concept. Initially, I would like to test the CTF type of mode. What I set to achieve is to have a bot, preferably in Java, that is fully controllable, that is, there is no AI but mec...
I'm going to block all bots except the big search engines. One of my blocking methods will be to check for "language": Accept-Language: If it has no Accept-Language the bot's IP address will be blocked until 2037. Googlebot does not have Accept-Language, I want to verify it with DNS lookup <?php gethostbyaddr($_SERVER['REMOTE_ADDR']); ?...
I have a asp.net download page which send a file to client but I want to deny robots download this file because the file is large and as I can see from the records a bot downloads this file about 20 times. This is slowing down the server and causes bandwidth consumption. I coded this page to count downloads and detect .net framework of ...
Hi, I have a web hosting package with 2 domains pointing to it. I've noticed on Google that it has indexed the directory of one of the domains for the other domain. Is there a way of preventing this from happening. ...
I'm a second Admin on a forum and have noticed that the number of View of threads varies from thread to thread quite a bit. I was told that a lot of those Views are from the many Bots that come in and go thru the forums. My question is, do they up the View number every time they come in a check a new post in a thread, or are all of tho...
I want to write bot for photo hunt game like this one, so I need to read every pixel and see if two pictures have difference color code. But don't have any experienced about flash before. Is it possible to write something like this and how ? Thanks, ...
Hi, Is there any library (java/python) that allows to make msn bots that support set/get display images. So far i've jml-java but it fails to set display image in wlm clients. I've checked twisted msn but it supports only msnp8. Is there somebody who's already implemented the protocol or could merely guide me so that i may implement th...