The problem of stats

No, not statistics in itself. The problem I am writing about is website statistics, and it started a long time ago.

Back in the day we simply used web server logs to analyse website traffic. One could see an incoming IP address and see where the associated browser went in the website. This worked well back then as websites were simple affairs and essentially all one big lump. Of course, this was an era when web servers were run almost in the spare time of those few IT (and indeed non-IT) that had any interest in the web. Back then I was not in the central IT team but I was afforded some latitude for experimenting with new things, especially when redundant hardware could be used. It was 1992 and the IMG tag was still in the realm of fantasy.

Later, there were two open source packages that became very popular, one called Analog and the other Linklint. The former produced statistics about website visitors and the latter could be used to check for errors, missing pages for example. Analog could, when provided with valid data estimate which countries visitors were coming from, very useful when your organisation markets itself globally.

Of course, the marketeers desired more. I was once asked to find out where everyone who only looked at our home page went next. Ok, where they visited another of our own web servers this was do-able, but the question was expanded to ask which of our competitors they visited next. This was new thinking, by which I mean thinking that one could not associate with any other media. For example, if the publisher of one newspaper wanted to know which other newspaper a person took after only glancing at their own it would need some form of physical surveillance, or perhaps a questionnaire. Neither would be particularly reliable, the questionnaire in particular.

Enter, stage left, Google Analytics. I had attended a launch event – well of a sort anyway – where a new product was described which would enable one to search all across the web. The name? Google. We had rudimentary search products by this time but nothing like what was being described. Bells were ringing, but rather quietly. I think we could see back then that all of a sudden content has value, just not to us. But, Google search aside we later got wind of Google Analytics ad the bells got louder amongst those of us who could already see future issues.

Google Analytics arrived with two quite major advantages. First, IT people no longer had to do anything, and second, the marketeers would have access to easy to understand graphs. But those of us who had this nagging voice about global surveillance and the fact that a corporate entity would effectively have access to data indicating where everyone browsed were ignored. Fast forward to the later times of the GDPR and the coming soon and already years late PECR replacement, cookie laws and all that and I resist shouting we told you so but we did and it was back in 1994.

Of course, there was still an issue. Ok, we have this useful global search facility now but how do we include local content which is not accessible from outside? Google again to the rescue. I had a pair of Google Search Appliances (GSA) installed, one in each of our main data centres and fronted by a NetScaler appliance. This provided resilience to the loss of a single GSA. Being on our LAN the GSAs were able to spider content that was restricted to local access and which therefore could not be spidered by Big Google. It also provided a useful facility whereby we could rank, to some extent, content and could apply keyword and key phrase matching to direct searches to specific content which would then appear top in the list of results. This little Google was far more friendly, not being bloated by the desire of the mothership to know all things of all people. Perhaps no surprise then that Google eventually retired the GSA product in favour of a cloud based provision. You guessed it, they wanted to know who was accessing all your secret stuff too.

Are we really where we are because marketing people wanted to know everything about everyone and companies, not just Google cashed in on it? Yes, I think so, and you can see just how far by those invasive adverts that themselves continually leverage new technologies to further invade. Remember pop-ups? And then pop-up blockers? And of course the whole cookie debate where a really quite useful facility enabling shopping carts among other things was hijacked in order to track us across webspace. Yeah, those. Remember the good old doubleclick cookie, adware, ad blockers, layers upon layers of this stuff. It is almost all because of marketing.

Advertising is here to stay and I have absolutely no issue with it. Although I generally ignore it I will admit to having seen something advertised that I was unaware of and which actually filled a need. But there is a constant battle between the marketeers and the techies which will continue because all of this, the Internet, the web, email is designed to help us and  be easy to use and to access. And that’s where it all went wrong but it could not really exist any other way.