Which browsers/plugins block HttpReferer from being sent?

Question

I am trying to interpret HttpReferer strings in our server logs. It seems like there is quite a high number of empty values.

I am wondering how many of these empty values are due to direct hits from people entering our URL directly into a browser and how many might be due to some kind of blocking utility that prevents the Referer from being sent.

I really have no idea how many people are using tools or browsers or 'anonymizers' that might block the refer. Any input?

Answer 1

How many of those are spiders? Are you looking at the user agent, too?

Answer 2

I think a large proportion may actually be caused by ISPs' restrictions. I know my ISP (BT, in the UK) filters it out (probably at the router) which is bloody annoying at times.

As it turns out, the block is actually put in place by Zone Alarm, a software firewall, which is often supplied by ISPs.

Answer 3

Some antivirus software is retarded and also started doing this for "security" reasons.

We had an email form that used referrer tracking to eliminate the gist of the random bot-spam an some people moaned that it didn't work.

Not entirely wonderful, but there are far more good uses of the referrer header than for just 'lets be evil and watch where people came from' to legitimise it.

( Some antivirus packages have been known to stop email working altogether for instance, and the clients will ring you and tell you its your fault until you tell them to get rid of their rubbish i've never heard of that company before' antivirus for the 40th time and they listen and their problem magically resolves )

Answer 4

I personally disable it using "Web Developer" extension of Firefox, only because of some "helpful" sites that highlight the search terms that I used to get to that page.

Thanks, I am fully capable of installing a highlighter plugin, or search for the words inside your page.

Answer 5

Opera has a quick toggle in the F12 menu that you can switch on "Send Referrer Information" or not to the site(s) you're surfing around.

Answer 6

I used to log all this stuff in my blogging app - pretty much all bots never send referrer info.

You should be able to make an educated guess as to whether it's down to it being filtered out or just people entering the URL.

If the first hit has no referrer but the loading of images/CSS etc has referrer info then they just entered the URL directly.
If they only ever pull down HTML with no images or CSS they are most likely a bot (or using Lynx perhaps).
If they pull down HTML, images and CSS with no referrer then it's being filtered out.