Data in Motion

Big data analysis can be applied to gain useful insights from data in-rest as well as data in-motion (INAP, 2019). Data in-rest is referred to data which is collected and analyzed later after the occurrence of event. For instance, analysis of sales data at the end of month for strategic purposes. On the other hand, data in-motion is referred to data which is collected and analyzed in real-time or near real-time. For instance, wristbands in theme parks are used to collect guest activity to improve customer experience by suggesting rides with less waiting time (2019). Other applications of data in-motion are ecommerce, fraud detection, network monitoring, financial trading markets, logistics, risk management, network management etc. (Watts, 2018).

For data in-rest, batch processing using bare metal servers are helpful to process and analyze when required (INAP, 2019). For data in-motion, real-time processing like streaming analysis is performed using cloud with less latency for analysis (2019). Combaneyre (2019) outlined that event stream processing analyze data is it flow from various organizational sources and outputs actions based on data. Data in-motion analysis can be broken into three key components, aggregation, correlation and temporal analysis. In aggregation step, continuous metrics can be calculated on varying time references like number of gift cards redemption with value of $2000 per hour and so on. Correlation is performed by say, comparing the redemption of number of gift cards with value greater than $2000 from various stores and generate alerts due to anomaly. Temporal analysis can be performed on the sales data of such gift card for the same store versus the historical trend to trigger fraud alerts (2019).

Some of the popular tools for real time data stream processing are (Jain, 2017):

Flink – streaming data processing over distributed computing architecture with support of several APIs. Flink has machine learning and graph processing libraries as well.

Storm – distributed real-time system with support of various languages and highly scalable and reliable.

Kinesis, Kafka, Samza etc. are some other stream processing tools with their own strengths and weaknesses (2017).

References:

Watts, S. (2018) What is stream processing? Event stream processing explained. Retrieved from https://www.bmc.com/blogs/event-stream-processing/

Jain, R. (2017) Real-time data streaming tools and technologies – an overview. Retrieved from https://www.algoworks.com/blog/real-time-data-streaming-tools-and-technologies/

Combaneyre, F. (2019) Understanding data in motion. Retrieved from https://www.sas.com/en_us/insights/articles/big-data/data-in-motion.html

INAP (2019) Data in motion vs. data at rest. Retrieved from https://www.inap.com/blog/data-in-motion-vs-data-at-rest/