The Digital Age is changing the context of everything that we do – and that includes how we use information for improvement.
Historically we have used relatively small, but carefully collected, samples of data and we subjected these to rigorous statistical analysis. Or rather the statisticians did. Statistics is a dark and mysterious art to most people.
As the digital age ramped up in the 1980’s the data storage, data transmission and data processing power became cheap and plentiful. The World Wide Web appeared; desktop computers with graphical user interfaces appeared; data warehouses appeared, and very quickly we were all drowning in the data ocean.
Our natural reaction was to centralise but it became quickly obvious that even an army of analysts and statisticians could not keep up.
So our next step was to automate and Business Intelligence was born; along with its beguiling puppy-faced friend, the Performance Dashboard.
The ocean of data could now be boiled down into a dazzling collection of animated histograms, pie-charts, trend-lines, dials and winking indicators. We could slice-and-dice, we could zoom in-and-out, and we could drill up-and-down until our brains ached.
And none of it has helped very much in making wiser decisions that lead to effective actions that lead to improved outcomes.
Why?
The reason is that the missing link was not a lack of data processing power … it was a lack of an effective data processing paradigm.
The BI systems are rooted in the closed, linear, static, descriptive statistics of the past … trend lines, associations, correlations, p-values and so on.
Real systems are open, non-linear and dynamic; they are eternally co-evolving. Nothing stays still.
And it is real systems that we live in … so we need a new data processing paradigm that suits our current reality.
Some are starting to call this the Big Data Era and it is very different.
- Business Intelligence uses descriptive statistics and data with high information density to measure things, detect trends etc.;
- Big Data uses inductive statistics and concepts from non-linear system identification to infer laws (regressions, non-linear relationships, and causal effects) from large data sets to reveal relationships, dependencies and perform predictions of outcomes and behaviours.
And each of us already has a powerful Big Data processor … the 1.3 kg of caveman wet-ware sitting between our ears.
Our brain processes billions of bits of data every second and looks for spatio-temporal relationships to identify patterns, to derive models, to create action options, to predict short-term outcomes and to make wise survival decisions.
The problem is that our Brainy Big Data Processor is easily tricked when we start looking at time-dependent systems … data from multiple simultaneous flows that are interacting dynamically with each other.
It did not evolve to do that … it evolved to help us to survive in the Wild – as individuals.
And it has been very successful … as the burgeoning human population illustrates.
But now we have a new collective survival challenge and we need new tools … and the out-of-date Business Intelligence Performance Dashboard is just not going to cut the mustard!