In our quest to provide the most accurate reporting to our clients, the monthly figures play an essential role. However, this is not quite as easy as we would like it to (or believe it should) be.
For many years we have used a combination of Google Analytics (reports produced from Google) results and Awstats (visitor details pulled directly from the server) to produce a comprehensive overview for our clients.
The world of Google can be a very cloak and dagger place on occasion though and a previously Google announced some big news for website owners. Going forward, when a search is made through the Google search engine (which in February 2014 was approximately 89% of all searches) and the result clicked, the search term would no longer be passed to the destination website. Instead, these visits would simply be grouped together in Google Analytics under the keyword “(not provided)”.
Whilst this does not affect the tracking of visitor numbers to a website (a key indication of growth and the success of an SEO campaign) it now makes it impossible to track which term a user typed in to find the website.
For example if you found our website by searching for SEO Oxted on Google, when you clicked on the link to visit our website, our unique visitor numbers would increase by 1 but we would not be able to identify how you found our website. Annoying isn’t it!!
What Made Google Do This?
A great question! On the surface, the official reason was that this was done in the name of privacy. Google is suggesting that those searching using their general services would not want their search term to be passed on to the destination site.
However, some people have other ideas regarding this……
“This is unfortunately a known issue with Google not passing referrer data from searches under certain circumstances.
Awstats parses information directly from Apache’s domain logs- which are the raw web server access logs. These logs contain various pieces of information (see the samples below), one of which is the referrer- i.e. the site the user was visiting prior to coming to yours. If the referrer is a search term, Awstats is intelligent enough to detect this, and also detect the search terms the user had entered in.
Unfortunately Google have been taking steps for a few years now (in the name of “privacy”) to restrict the visibility of these referrers. Initially this was only for users who were signed into google when they performed a search. Now, it seems to be that any search done over SSL (i.e. https) doesn’t pass the referrer over to the site. I’ve provided an example below based on my own testing:
To test this I searched for “footprint web design” on Google and Bing and then clicked through to your site. Here are what the raw access logs recorded:
18.104.22.168 – – [26/Jun/2014:10:27:47 +0100] “GET / HTTP/1.1” 200 96724 “https://www.google.co.uk/” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36”
22.214.171.124 – – [26/Jun/2014:10:29:46 +0100] “GET / HTTP/1.1” 200 96724 “http://www.bing.com/search?q=footprint+web+design&go=&qs=n&form=QBLH&filt=all&pq=footprint+web+design&sc=1-20&sp=-1&sk=&cvid=738cf01f67bd4d7b8f8f3e0d2b52f6d0” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36”
Notice the referrer (the sixth field) in the first line only reports “https://www.google.co.uk“. The actual referrer should be:
Because this is the URL of the page I was performing the search on. If this was the referrer Google sent, Awstats would pick up on the “q=footprint+web+design” bit at the end of the URL and record a search term of “footprint web design”. However, as Google doesn’t send the actual referring URL, Awstats can’t report it.
I hope this makes sense- unfortunately because this is a problem with the way Google are passing referrers there isn’t anything we can do at our end. I understand this will affect the SEO work you are doing as you won’t have visibility of many search terms.”
As you can see, whilst we are continually looking for an alternative method of tracking how visitors find our customer’s websites, this information is currently unattainable.
Some Good News Though!
Whilst the information provided by Awstats pay be incomplete (through no fault of the host server), the visitor numbers to your website are guaranteed to be accurate because these figures are generated directly from the server and cannot be altered in any way.