AI Influence on big data

Traditional machine learning algorithms and systems were developed with the assumption that data will fit in memory however in the realm of big data it is not possible any more (L’Heureux et al., 2017). Although big data analytics encompass text analytics, business intelligence, data visualization, and statistical analysis but machine learning, which is a subset of artificial intelligence can serve as a core of data analytics by learning from the data, generate useful insights, help in decisions and present predictions.

Some of the ways machine learning is tackling the beast of big data is by following below steps:

1. Data extraction

2. Data pre-processing

3. Data transformation

4. Data storage

5. Data analysis

6. Decision making

In date pre-processing step, instance selection (selection of subset of data which represent the whole data set), dimensionality reduction (reduce the higher number of dimensions to fewer) and data cleaning (noise and outlier removal) is performed to bring the data to a manageable size without significant loss of information.

In data transformation step, vertical scaling (parallel computing) and horizontal scaling (map/reduce and graph based functions for batch oriented system while Apache storm and Spark are deployed for streaming real time systems) functions are performed depending upon the type of data set.

In data analysis step, machine learning algorithms like support vector machines (SVM) and regression models are tweaked for big data analysis.

Some key machine learning paradigms for big data are deep learning, online learning, local learning, transfer learning, lifelong learning and ensemble learning. These paradigms are developed to successfully handle the volume, variety and veracity of big data for efficient, robust and high quality data insights. With the evolution of machine learning techniques, tools and technologies for big data, it is now possible to analyze large data sets like genome decoding, opinion mining, speech recognition, etc.

Reference:

L’Heureux, A., Grolinger, K., Elyamany, H., & Capretz, M. (2017). Machine learning with bog data: challenges and approaches. 2017 IEEE Access Vol. 5, 7776 – 7797.