Is a visitor to my website coming from a search engine? What were their search terms?

Question

I'd like to display my page differently for the visitor based on if they came from a search engine or not. I guess I'm detecting search engine referrals? I'd also like to know the search terms used to arrive at my site.

My solution so far is this:

1) Filter on the HTTP request referers which contain common search engine URLs, i.e. http://www.google.com/search, http://www.bing.com/search, or http://search.yahoo.com/search

2) Parse the query string parameter of the referer for the search terms, e.g. "q=search+terms+for+my+website+go+here"

I feel this isn't the most robust solution, because it misses lesser known search engines and each search engine may have it's own query string parameter for the search terms. For example, Google's and Bing's search term parameter are both "q" but Yahoo's is "p" as far as I can tell. And what about special operators like +, -, etc.?

Is there a more general way to do this? Or is there a library that would help me handle more search engines? I'm working with Java running on Tomcat, but ideas from any language and server setup are welcome.

Answer 1

Your approach is about the best you can do. I ran a similar module on Drupal,

http://drupal.org/project/search%5Fengine%5Freferers

If you look at the code, it does exactly what you said. I don't think they have a version supporting Bing.com yet.

Considering where the search market share is going, you get well over 90% of the searches if you can handle Google (AOL uses Google), Bing and Yahoo. Do you really need to worry about other search sites? If you are in China like I am, add baidu.com to the list.

There are weird small search sites that you just can't handle. They change their parameter once a while. Some even use POST to send the query so it's not available in referrer.

Answer 2

Have a look here: http://www.gdargaud.net/Hack/Searches.html

and here: http://search.cpan.org/perldoc?URI::ParseSearchString

2 slightly different approaches but they cover a much wider range of search engines and their referer strings.

Answer 3

In case of PHP, you request the $_REQUEST["HTTP_REFERRER"] and parse the search-string with some pregs or eregs.