How to determine search query forwarding user to my website?

Question

Greetings,

I'm trying to work out what query is being used to forward people to my website. I'd appreciate it if anyone could tell me what api call I should be looking into. I'm sure this is possible with javascript as well as ruby and php so any technology is fine.

Just for learning sake I don't mind know what I should be using for all three :)

Answer 1

Try Google Analytics. It will give you the search engines, search terms and referring sites in a lot of detail.

Answer 2

Check the HTTP request header Referer of your visitors ( $_SERVER['HTTP_REFERER'] in PHP ). This will, in 90% of all cases, give you the page your visitor came from. If it's a search engine page, you then need to decode the referer querystring and parse the data out of it (note that the query string format may change).

It's more work than Google Analytics, and the simple example I gave doesn't get you as much data; but could be done if you aren't comfortable with Google doing the data collection for you.

Answer 3

In ruby, you can get the referrer by doing

request.referrer

The request params are available in your views and controllers. If you want to see other options available on request then just add

<%= request.inspect %>

in one of your views and access the page to see them.

Answer 4

Having worked with search engines for more than 5 years, I can tell you there's no standard way to retrieve the query value.

As other answers already told you, the first step is to inspect the HTTP_REFERER header. Assuming you are using Rails, you can get it from the request

request.referrer

Otherwise, you need to extract it from request headers in an other way.

Once you have the referrer, then you are in front of 3 main possibilities:

1. variable is empty. sorry, you can't do nothing
2. variable is not empty, it's a search engine
3. variable is not empty, it's not a search engine

The first option is simple. What you want to know is if the referrer is a search engine. If so, then you need to extract the query.

The most common way to do this is using a checklist. The checklist is usually a list of key/value where the key is the search engine domain and the value the name of the query string parameter that holds the query value.

google.com,q
yahoo.com,p
...

This is the same approach used by Google Analytics. From the ga.js file

g.T=l("daum:q,eniro:search_word,naver:query,images.google:q,google:q,yahoo:p,msn:q,bing:q,aol:query,aol:encquery,lycos:query,ask:q,altavista:q,netscape:query,cnn:query,about:terms,mamma:query,alltheweb:q,voila:rdata,virgilio:qs,live:q,baidu:wd,alice:qs,yandex:text,najdi:q,aol:q,mama:query,seznam:q,search:q,wp:szukaj,onet:qt,szukacz:q,yam:k,pchome:q,kvasir:q,sesam:q,ozu:q,terra:query,mynet:q,ekolay:q,rambler:words");

First host matches both key and value, first wins.

Answer 5

I think this will be very useful for you to check out. It explains in detail how to do exactly what you are trying to do:

http://www.alistapart.com/articles/searchhighlight/