Finding HTTP_REFERER empty/blank when using URL shortener site

Question

I have a client who is trying to determine traffic metrics, to his site by way of those "URL shortened" sites: (tinyurl.com, bit.ly and x.co) to be specific, and to take action based on that traffic...

We would have thought that the HTTP_REFERER variable would have held the referring resource name, the shortened URL from that service. Instead the HTTP_REFERER field is empty and if read in some browsers its actually not even there (I guess NULL?).

Here is an example of an attempt to enter a shortened URL that goes to my client's site and the name of the page is x.php:

HTTP_CONNECTION:keep-alive HTTP_KEEP_ALIVE:115 HTTP_ACCEPT:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,application/json HTTP_ACCEPT_CHARSET:ISO-8859-1,utf-8;q=0.7,*;q=0.7 HTTP_ACCEPT_ENCODING:gzip,deflate HTTP_ACCEPT_LANGUAGE:en-us,en;q=0.5 HTTP_COOKIE:ASPSESSIONDQADBDABT=HAEFPIOBONKMOIJFDGNHHEM HTTP_HOST:www.<myclientswebsite>.com HTTP_USER_AGENT:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729) 

Why wouldnt the http://x.co shortened URL show up in the header info OR am I not looking in the correct place?

My client would ultimately like to redirect incoming traffic to the appropriate resource within his website AND/OR out to other sites he owns.

UPDATE: I've looked through his raw traffic logs and I cant find a specific referrer other than x.co or bit.ly, etc I do not see the "/" part.. Is there something I can change in his IIS6 settings, on his web server, that would allow us to see and utilize the information he's looking for, or are we just out of luck due to the design of the HTTP redirect?

Answer 1

HTTP_REFERER is an optional header. In some cases they are stripped out (security software, proxies, etc).

In the case of URL shortening services, they probably do a header redirect and simply don't include HTTP_REFERER.

Answer 2

Because the shortening services use response 301 or 302 and Location - there is no referer. Your browser does not pass it.