What are the most common reasons for requests without referrer?

Question

I have a service with a large share of requests with an empty value for HTTP_REFERER. I'd like to interpret this correctly and wonder about the most common reasons for that.

I understand that HTTP_REFERER is an optional header field, but most browsers with default setting seem to send them.

Common reasons I have found so far:

• proxies
• robots
• JavaScript links (All of them? Is this browser dependant?)
• request from bookmarks or as browser startup page
• user entered URL manually
• Flash links
• link from a different app like email client
• browser settings or privacy browser add-ons
• some personal firewalls filter referrers
• no referrer is sent by most browsers if the redirect happens via semi-official Refresh http header
• referrer fakers like this

What's missing|irrelevant|wrong?

Is it possible to put percentages behind these items? Or at maybe sort the list and point out the proportions?

Answer 1

Basically, all page requests that does not involve user clicking on a link on a webpage.

Answer 2

A percentage will depend on what your website is and why people may want to fake their referrer .. Also some people just crack open a new-tab without a homepage. Or land via something other than the browser (such as an addon or chat link, whatever).

If your functionality relies on the referrer use a cookie or rethink the design. Because you can't rely on it.

Answer 3

All depends, and we don't have enough information to say which of the causes is most likely. I'd say robots, but you have to analyse the data (assuming you have server logs) and interpret it. I have no idea how popular your site is or what is its purpose, so robots may not be the number one reason.