Real-time data seems to be everywhere — in augmented reality, digital twins, 5G, IoT, AI, machine learning, wearables, and beacon technology. One can be forgiven for thinking that today’s enterprises are streaming real-time data across every vital task area. We’re getting there — thanks in large part to many open-source solutions such as Apache Flink, Kafka, Spark, and Storm, as well as cloud-based platforms. However, there is still a lot of work that needs to be done before we reach the point at which data moves through and between organizations at light speed or something close to it.
First, a level set, courtesy of IDC’s John Rydning: “Often, the terms streaming data and real-time data are used in conjunction with each other and sometimes interchangeably. While not all streaming data created is real time, and not all real-time data is streamed, organizations indicate that over two-thirds of streaming use cases require ultra-real-time or real-time data.”
Also: Every AI project begins as a data project, but it’s a long, winding road
There’s even ultra-real-time data in use — and companies are anxious to make this work. “Understanding the value of capturing and processing real-time data is growing at the fastest pace in recent times,” says Avtar Raikmo, director of engineering at Hazelcast. “With platforms taking complexity away from the individual user or engineer, it has accelerated adoption across the industry. Innovation such as SQL support, help make it democratized and provide ease of access to the vast majority rather than a select few.”
There is a wide range of use cases, including “compute at the edge for audio and video streaming, computer vision for AI and machine learning processing, or even active noise-canceling headphones,” Raikmo says. Another emerging use case is digital twins, especially for mobility. “Being able to capture real-time data and telemetry from cars, trucks or rockets enables organizations to model scenarios as they unfold. Digital twins can be used to optimize real-world routes taken, energy used or assisted driving improvements. In the world of sport, Formula 1 strategists determine the optimum pit-stop and tire compounds to maximize race performance.”
Still, there are many technical and organizational issues standing in the way of full real-time — or ultra-real-time data realities….