What is the Relationship Between IoT and Big Data?
Written by: Cody Lirette
Published: September 17, 2019
Though they are often mentioned together, the Internet of Things (IoT) and big data represent distinct, if complementary technical fields. IoT describes the complex web of decentralized devices that collect information and route it back to a central point over the internet. This information can be used to create real-time analytics that are used in everything from industrial manufacturing to agricultural irrigation. Big data is a more general concept. Before we can understand the relationship between IoT connectivity and big data, let’s first define what big data is.
What is Big Data?
Put simply, big data describes very large sets of information. This information can come from anywhere and everywhere, from links clicked on social media to financial transactions worth millions of dollars. It’s estimated that there are now more than 44 zettabytes (or 352 trillion GB) of data comprising the digital universe, with the majority of that data having been created over the past two years. With roughly 1.145 trillion MB of information being created every day, businesses and other organizations are combing through mountains of data to create actionable insights that will shape the future.
Whether through modeling software or artificial intelligence (AI), like machine learning, these analyses are being used to predict outcomes, make recommendations and guide important decisions. For example, video streaming services such as Netflix and Amazon Prime collect data about your streaming patterns and recommend similar titles for future viewing. Elsewhere, online retailers will often use search histories and link usage to tailor web banner ads to your personal tastes.
The Four “V”s
To help understand such an enormous concept, data scientists at IBM popularized “The Four V’s” of big data: volume, variety, velocity, and veracity.
Volume
The incredible amount of data regularly collected through sensors, online transactions, social media, and other mediums cannot be processed or even stored using traditional methods. According to some estimates, people will generate around 463 exabytes of data every day by 2025. As you may imagine, some of these data sets can be too large to fit on a single server, and must instead be distributed between several storage locations. Data analytics frameworks such as Amazon S3, Hadoop, or Spark, are built to accommodate the need for distributed storage and aggregation.
Variety
Today’s data comes in many different types, from social media posts about your favorite food to readings from moisture sensors in a field. In past decades, data was more clearly defined—for example, phone numbers, addresses, or ledger amounts—and could be easily amassed into spreadsheets or tables. Today’s digital data often cannot be corralled into traditional structures. Powerful analytics software seeks to harness unstructured data, such as images and videos, and combine it with more straightforward data streams to provide additional insights.
Velocity
In 2020 alone, people created about 1.7MB of data every second. That’s more than 2.5 quintillion bytes of data per day. Considering that Google processes more than 3.5 billion searches in a day, around 60 hours of video are uploaded to YouTube every minute, and 6,000 tweets are sent every second, processing all of this data is a massive undertaking – and that’s without even mentioning the considerable insights generated by machine learning. As you may suspect, all of this accumulated data is streaming into servers at an unprecedented speed.
Veracity
Veracity refers to the truthfulness or accuracy of a particular set of data. That includes evaluation of the data source’s trustworthiness. IBM estimates that poor data quality costs the U.S. around $3.1 trillion per year, so pursuing veracity is important. It includes eliminating duplication, limiting bias, and processing data in ways that make sense for the particular application or vertical. This is an area where human analysts and traditional statistical methodologies are still of great value. While AI is becoming more sophisticated, it cannot yet match the discernment of a trained human brain.
IoT and Big Data
In one sense, IoT data represents one among many series of creeks and rivers that feed into an ocean of big data. The enormous collection of connected sensors, devices, and other “things” that represent IoT—an estimated 35.82 billion devices worldwide—makes a significant contribution to the volume of data collected. IoT has applications in just about every industry and sector, from agriculture to consumer smart devices to factory automation. Sensors can be used for asset management, fleet tracking, remote health monitoring, and more.
Data coming from IoT devices is only as useful as the analyses that can be drawn from it. Providers of IoT solutions like Soracom or Balena have created platforms, applications, and other products that enterprises and organizations can use to manage their IoT devices as well as feeding data securely to cloud services to analyze.
Distinct but Complementary
While both big data and IoT represent large collections of data, one of IoT’s main goals is to run analytics simultaneously to support real-time decisions. For example, an e-commerce company might track consumer habits over time and use that data to create tailored content and advertising for the customer. In the case of driverless cars, however, data cannot be put aside for later analysis. If all signs point to an impending accident, the machine needs to receive and interpret that information without delay so it can make a split-second decision.
Many IoT devices rely on remote servers accessed via the cloud, but in some verticals designers are embracing edge data processing to reduce the latency of data transfer. In this model, the device retains power to process some data locally, allowing for more immediate results for time-sensitive operations.
While much of IoT focuses on the immediate analysis and use of incoming data, big data tools can still aid some functions. Predictive analytics, for example, considers a machine’s performance and service alerts over time, building the library of data needed to anticipate upcoming problems. That allows companies to be proactive about servicing their equipment, avoiding the potentially costly downtime that can come with equipment failure.
The sources from which they draw data are another major distinction between the two. Big data analytics focuses mainly on human choices, especially in the online realm, in an effort to predict behavior and uncover patterns or trends. On the other hand, while IoT devices can certainly monitor and learn from user-generated data, there is a large number of projects built around machine-generated data, with primary goals that are machine-oriented. These goals include optimal equipment performance, predictive maintenance, and asset tracking, to name a few.
Common Goals
Big data and IoT are distinctive ideas, but that shouldn’t imply that they don’t overlap in many ways. Both convert data into tangible insights that can be acted upon, meaning that many applications for both will overlap.
One example of IoT working together with big data analytics comes from the energy market. In 2019, Japanese gas company NICIGAS installed IoT sensors into 850,000 gas meters across the country. These sensors record and transmit usage and event data that can be used to predict the amount of gas left in a household, or secure valves in the event of an earthquake. That data allows NICIGAS to better understand the demand for their product, and utilize their logistics and distribution channels more efficiently. Ultimately, this combination of immediate IoT insights and big data analytics results in a more satisfied clientele, improved efficiency, and better use of corporate and environmental resources.
IoT and big data have an important relationship that will continue to develop as technology advances. Companies wishing to harness the power of data should carefully consider the devices they choose to deploy and the types of information they collect. Making an effort at the front end to gather only useful, applicable data—and designing internal systems to process it in sector-specific ways—will make the process of analytics that much easier.
Editor’s Note: This post was originally published in September 2019 and has been updated for accuracy.
