Big Data: 8 Trends Shaping the Future of Data-Driven Businesses

Posted: 10/03/2016


The year 2016 has been an important one in the world of big data. What used to be hype has largely become the norm as more businesses have realized that data, in all forms and sizes, is critical to making the best possible business decisions. As we head into 2017, we’ll see continued growth of systems that support non-relational or unstructured forms of data as well as massive volumes of data.

Big data is only getting bigger, and with so many viewpoints in the industry, it can be challenging to get a bird's-eye view of the various trends affecting the landscape. Here's a roundup some of the hottest trends in big data and hypotheses on how they might grow in the near future.

1.   Hadoop Is Still #1

Apache Hadoop, one of the most popular software frameworks for storage and processing of big data, seems poised to get even bigger in years to come. Major areas of focus in the near future include Hadoop's user experience, which some customers have described as still being too difficult to navigate. Improvements in the speed of Hadoop's data exploration, by leveraging technologies such as Cloudera Impala and AtScale, help accelerate indexing, processing and retrieval and are key to continued success. The growth of the Apache Sentry module, which enforces authorization and access to data and metadata on a Hadoop cluster, will also help spur adoption among big enterprise customers.

  • In a 2015 survey by software company Tableau, 76 percent of Hadoop customers said that they plan to use it more within the next three months, while only three percent said that they would use it less in the next year. Nearly half of companies who had yet to deploy Hadoop said that they planned to do so in the next year.
  • A 2016 report by Allied Market Research estimates that Hadoop will reach global revenue of $84.6 billion by 2021, growing by more than 63 percent in the next five years.

2.   The Internet of Things Takes Over

As people move more of their lives online, their devices migrate with them. The Internet of Things (IoT) is the term used to describe this vast technological network—which includes anything from medical implants and weather sensors to vehicles and power grids—and the endless streams of data that they collect and exchange while connected to the Internet. This constant churn of information will require advances in real-time analytics to be properly processed, and companies will need to address the challenges that IoT presents to data governance and security.

  • IBM expects IoT to grow 30 percent in the next five years.
  • Business analytics company Pentaho estimates that IoT will create economic value between $3.9 trillion and $11.1 trillion by 2025.
  • IT research firm Gartner projects that by 2017, more than 20 percent of customer-facing analytic deployments will use the IoT to provide tracking of product location and performance.

3.   The Rise of Cognitive Computing

Cognitive computing relies on technologies like artificial intelligence, natural language processing and machine learning in order to let humans and machines interact more fluidly, giving people easier access to the knowledge computers possess. Think of IBM's Watson, which arrived in the public consciousness by defeating two Jeopardy champions and is now analyzing vast quantities of medical data, or Facebook's M, a virtual assistant that helps users find nearby points of interest or place orders online. With advances in cognitive computing, even non-technical employees will eventually be able to crunch data by interacting with an application much like they would interact with another person. Companies are also looking to build customer-facing platforms that leverage cognitive computing to help users with technical issues or walk them through the buying process.

  • Intelligence firm IDC predicts that by 2020, half of business analytics software will rely on cognitive computing technologies.
  • Allied Market Research expects that the cognitive computing market will generate $13.7 billion in global revenue by 2020, growing by more than 33 percent every year.

4.   Big Data Matures

As further evidence to the growing trend of Hadoop becoming a core part of the enterprise IT landscape, investment continues to grow in the components surrounding it such as security. Apache Sentry provides a system for enforcing engrained role based authorization to data and metadata stored on a Hadoop cluster. This is just one example of the types of capabilities that customers expect from their enterprise-grade platforms and are now coming to the forefront of emerging big data technologies, thus eliminating one more barrier to enterprise adoption.

5.   The Cloud Remains a Constant

Big data and cloud computing are two of the IT sector's hottest buzzwords right now, and current trends suggest that the pair will be even further intertwined in coming years. Major players are migrating their massively parallel processing (MPP) data warehouses into the cloud, hoping to kickstart the slower growth that they've seen as of late. Services such as Amazon Redshift, Google BigQuery and Microsoft's Azure SQL Data Warehouse are all fully reliant on cloud computing, allowing customers to vary the amount of storage and processing for which they rely on the data warehouse. Cloud-based analytics engines are also expected to absorb and process the massive amounts of IoT data, becoming must-have additions for the cloud.

  • Cisco estimates that the amount of information stored in data centers will increase from 1.4 zettabytes to 3.5 zettabytes between 2014 and 2019. The proportion of data stored in data centers, instead of on client devices, will also increase from 12 percent to 16 percent.
  • IDC predicts that up to 2020, spending for cloud-based big data and analytics (BDA) technologies will grow 4.5 times faster than spending for on-premises BDA solutions.

6.   NoSQL is Key
NoSQL technologies are commonly associated with unstructured data. Going forward, the shift to NoSQL databases will become a leading piece of the Enterprise IT Landscape, particularly as the benefits of schema-less database concepts become more pronounced. Nothing shows that picture more blatantly than looking at Gartner’s Magic Quadrant for Database Management Systems, which has historically been dominated by Oracle, IBM, Microsoft and SAP. In contrast, the most recent Magic Quadrant shows several NoSQL companies, including MongoDB, DataStax, Redis Labs, MarkLogic and Amazon Web Services (with DynamoDB), outnumbering the traditional database vendors in Gartner’s Leaders quadrant of the report.

7.   Self-Service Becomes Essential

Self-service is becoming more and more of a need across all areas of technology, and big data is no exception. Data preparation tools are exploding in popularity, in part due to the shift toward business user generated discovery tools such as Tableau that reduce time to analyze data. Business users also want to be able to reduce the time and complexity of preparing data for analysis, something that is especially important in the world of big data when dealing with a variety of data types and formats. Companies focused on end user data preparation for Big Data such as Alteryx, Trifacta, Paxata and Lavastorm are investing heavily here, and even long established ETL leaders such as Informatica with their Rev product have followed this trend.

8.   More Robust Visualizations

What's really going to make big data go mainstream is not just its ability to connect with data scientists and technologists but business people in general. One of the core tenants in doing so will be visualization, being able to show people—not just tell people, or simply show numbers or charts—but to have those charts and graphs and visualizations come alive and interact.


The potential impact that big data will have on businesses' agility and drive for performance is hard to ignore. Gartner bets that by 2020, big data and the Internet of Things will be used to reinvent, digitize or eliminate 80 percent of business processes. However, it's less important that companies get caught up in the hottest big data trends, and more important that they begin leveraging big data in the first place to drive innovation and discovery for their business. To take advantage of the truly revolutionary power of big data, first identify how you can use your organization's master data and strategic data to build reporting and analytics that appeal to your operations and core strengths.

About the Author

Tristan Boutros is the award winning author of The Process Improvement Handbook: A Blueprint for Managing Change and Increasing Organizational Performance and The Basics of Process Improvement, both available on In order to read his future blog posts on Process Improvement, Program Management as well as general management and technology issues and trends, please click the “Follow” button on his profile.
Follow him via Twitter @TristanBoutros

Posted: 10/03/2016


Nashville Airport Marriott, Nashville, TN
May 28 - 29, 2018
Hilton London Kensington, London, United Kingdom
June 18 - 19, 2018
The Hurlingham Club, London, United Kingdom
July 8 - 10, 2018