Is Big Data the New Six Sigma?

Why Big Data Needs Process Professionals

Every day the world creates 2.5 quintillion bytes of data, according to figures from IBM. That’s a lot of data. It means that 90 percent of the data present today simply did not exist two years ago.

Big data is not simply defined by its volume, however (although volume does play a major role). It is also characterised by its variety – from email and audio to traditional databases – and its velocity – how fast it is being produced and must be analysed.

Of course, data without analysis is not real knowledge. It is in the analysis of Big Data where its value truly lies. It is the next big thing in setting apart companies from their competitors, in the same way that Six Sigma once brought about the process improvements that allowed Motorola to outstrip its rivals.

Drowning in data?

According to strategy consultancy McKinsey, there are several ways in which Big Data creates value. Information is made transparent and usable at a must greater frequency, unlocking opportunities for growth.

More accuracy can be gained in everything from transactions to employee absences, identifying areas that need to be targeted and boosting performance. The data collected is then being used to "conduct controlled experiments to make better management decisions".

Big data is also translating to better insight into small segments of consumers, allowing for more accurate tailoring of products and promotions.

The sophisticated analytics associated with Big Data are also boosting decision making capabilities, while, finally, helping to shape the products of the future, with McKinsey offering the example of providing after-sales services in the form of proactive maintenance.

Talking in terms of numbers, this means a retailer using Big Data to its full potential could boost its operating margin by over 60 percent, the report claimed.

This opportunity does not come without its challenges, however. Challenges that require so-called Big Process to be solved and skills not yet present in high enough volumes to leverage Big Data to its full potential.

"I keep saying the sexy job in the next ten years will be statisticians," said Hal Varian, Google's chief economist, in an interview with the McKinsey Quarterly.

According to McKinsey, by 2018, the US alone could be faced with a shortage of between 140,000 and 190,000 people possessing the deep analytical skills needed to make sense of the volume, variety and velocity of Big Data.

These data scientists – the new term coined for these professionals – will find themselves in high demand. Research from EMC suggests 83 percent of people believe new technology will increase the need for data scientists in the future.

But, writing for the Forrester Research Blog, James Kobielus argues that the role of data scientist is more like a "very useful new term for referring to a wide range of advanced analytics functions that heretofore have had no consensus category label".

"The term recognizes that advanced analytics developers, like scientists generally, spend their careers exploring new data for powerful insights that may not be obvious on first glance," he explained.

And the above skills are another point at which the similarities between Six Sigma and Big Data can be noted. Although it has moved on from a strict interpretation of 3.4 defects per million opportunities, Six Sigma requires a high level of analysis.

As a management discipline, it brings with it an innate knowledge of process, which will prove vital as companies look to make sense and deliver on the insights provided by greater volumes of information. A further 1.5 million managers with the knowledge of how to effectively use the insights of Big Data to make better decisions will also be needed, according to the Mckinsey report.

Gartner has already warned too many IT managers are focusing on dealing with the huge volumes associated with Big Data at the expense of other aspects, which will leave them with challenges down the line – further promoting the case for other professionals to be involved in the process.

Mark Beyer, research vice president at Gartner, warned: "Today's information management disciplines and technologies are simply not up to the task of handling all these dynamics. Information managers must fundamentally rethink their approach to data by planning for all the dimensions of information management."

Without this, Big Data may not reach its full potential as an engine for adding value to almost all business sectors.