Devil’s in the detail for big data

ANALYSIS: The audience at the WWW2017 conference held in Perth in early April was definitely at the geekier end of the spectrum. But for those with the patience to listen, it was fascinating.

One thing that struck me during the talks was the sheer volume of data about people that the big tech companies are collecting. The social media platforms are collecting data on every single action or motion for all their billion-plus users. That volume of data gives them an outlook on humanity that is unique in our history.

For example, at one of the talks I attended the audience heard that Facebook predicts how long you will spend browsing your news feed. Based on your age and gender, the Facebook algorithm knows to within minutes when you’re going to stop browsing.

So Facebook experiments with serving you different content towards the end of your session. The results of that experiment provide the company with knowledge of what will keep you browsing. This is the goal – Facebook knows how to manipulate you into doing or not doing something, using statistics and software, with no human intervention, to change your behaviour.

This produces a different mindset in these researchers; the way they talk about the experiments is revealing. Facebook got into trouble a while back for its experimental attitude. It turns out that people don’t like being experimental subjects. The attitude is still there, but now Facebook doesn’t talk about the experiments in public. 

This experimental mindset is a central tenet of ‘lean’ startup methodology. It is often referred to as ‘startup science’. The business idea is a theory, composed of a set of hypotheses about the market. Founders need to test these hypotheses against actual target customers to validate them. We do this by performing experiments that aim to test customer behaviour. We test behaviour rather than attitudes, because people don’t think they do what they do.

The WWW2017 conference showed that the tech giants are taking this startup science to the next level.

For years, they have collected trillions of data points from hundreds of millions of customers. This gives them an unmatched level of insight and ability to validate hypotheses. They perform hundreds of experiments every day, discovering how to make their product more addictive, more viral, more able to mediate all your personal relationships.

This is unique in history. The old truth that ‘half your marketing spend is wasted, but you never know which half’, is no longer true. Facebook can tell you within minutes whether you’re wasting your marketing spend, because it knows and understands its users so well. There’s a lot of benefit to that, but it’s also very creepy. We’re going to have to be very careful about how we walk this line.

Consensus politics

The most-heckled talk I saw was the World Wide Web Consortium (W3C) talk on upcoming standards. (W3C is the body charged with ‘developing protocols and guidelines that ensure the long-term growth of the Web’.)

They solicited feedback during the talk, and were actively heckled by members of the audience with comments. This correlates completely with what I’ve heard of the work of the group. To say they are unsung heroes is an understatement; vilified saints would be a better description.

Attempting to get any two techies to agree on the best way of doing anything is difficult. For example, There’s a ridiculous 40-year-old war about whether to use tabs or spaces to indent code. It is ridiculous because the people who use spaces still won’t see sense, of course. Getting the entire industry to agree to a standard is impossible. Yet the W3C manages it, somehow.

Future historians will refer to our time as the years ‘when we built the internet’. The most important work of our generation is happening online. The internet is what our great-great-grandchildren will remember us for, and how they will remember us.

The last time the WWW conference was in Australia, two geeky students presented the PageRank algorithm. That algorithm is now the core of a global business empire that is one of the most valuable companies in the world. Those geeky students are multi-billionaires.

One of the papers presented this month in Perth could possibly change the world that much, although I couldn’t see it. But that’s normal; trying to predict the future is hard. It was exciting to sense the possibilities in the air, though.

Finally, AI is beginning to become a real thing. Computers understand images, speech, written text, human relationships. The future as seen from that conference was bright and utopian. I could almost hear the 1950s-era voiceover of confidence and optimism about science during it. I’m glad I went.