Big data analytics: Computational intelligence techniques and application areas

Highlights

• We highlight the importance of Big Data in modern life and economy.

• We investigate the benefits of computational intelligence techniques in big data analytics.

• We present a data modelling methodology called Hierarchical Spatial-Temporal State Machine

• We explore the potential of the powerful combination of Big Data and Computational intelligence

• We identify a number of areas where novel applications in real world problems can be developed.

Abstract

Big Data has significant impact in developing functional smart cities and supporting modern societies. In this paper, we investigate the importance of Big Data in modern life and economy, and discuss challenges arising from Big Data utilization. Different computational intelligence techniques have been considered as tools for Big Data analytics. We also explore the powerful combination of Big Data and Computational Intelligence (CI) and identify a number of areas, where novel applications in real world smart city problems can be developed by utilizing these powerful tools and techniques. We present a case study for intelligent transportation in the context of a smart city, and a novel data modelling methodology based on a biologically inspired universal generative modelling approach called Hierarchical Spatial-Temporal State Machine (HSTSM). We further discuss various implications of policy, protection, valuation and commercialization related to Big Data, its applications and deployment.

Introduction

The importance of data in our increasingly information driven economy and society can be summarized in the statement that “big data is the new oil” as was recently quoted by IBM's chief Executive officer “(Hirsch, 2013). The analogy of the importance data to that of the world's reliance on this natural resource has been identified and highlighted by various studies demonstrating the power and impact of Big Data to modern life. This new digital resource can provide the main driving force for building the smart cities of tomorrow. Big data can be defined as huge sets of data with a structure of increasing variety and complexity. The inherent difficulty in dealing with these large amounts of data results in major challenges concerning their storage and analysis as well as the cost and time efficient delivery of results. Moreover, information should be delivered in an interpretable and easily visualized way (Sagiroglu and Sinanc, 2013). Big Data analytics refers to the techniques utilized in order to examine, process, discover and expose hidden underlying patterns, interesting relations and other insights concerning the application context under investigation.

As pointed out by Hashem et al. Big Data have three main characteristics. Firstly, as the name reveals the data themselves are numerous. Secondly, it is not possible to categorize the data into regular relational databases, and finally data streams are created, captured, and analyzed rapidly (Hashem et al., 2015). As Gerhard mentions, Big Data is a revolutionary leap forward from traditional analysis which possesses three main characteristics volume, variety, and velocity (Gerhardt et al., 2012). Volume refers to the amount of data, which are created and stored. Variety is related to the various types of data collected, and velocity can be defined as the speed of data generation, streaming and aggregation (Kaisler et al., 2013). In Kaissler et al.'s study, data value, and complexity are also proposed as Big Data characteristics. Data value is a measure of the usefulness of data in decision making processes, while complexity is a measure of the degree of interdependence and interconnectedness in Big Data structures.

Nowadays, in modern smart cities, there is a growth in the amount of data, which can be captured and utilized. Recent advances in hardware and software technologies, such as social media, the internet of things, wearable sensors, mobile technologies, data storage and cloud computing, data mining techniques and machine learning algorithms have resulted in the ability to easily acquire, analyze and store large amounts of data from different kinds of quantitative and qualitative domain specific data sources. This data can be harvested from a large number of diverse sources including emails and online transactions, multimedia information such as video audio and pictures, large databases containing health records and other information, information captured during a user's interaction with social media such as posts, status updates etc., data derived from the search queries or click patterns of a user, physiological data such as heart rate, skin conductivity etc. as captured from wearable sensors, data derived and extracted from our interaction with our mobile devices or other devices inside a smart home, data from scientific research and others (Eaton et al., 2012). It can be easily concluded form the above that in the modern world, data is generated at a record rate (Villars et al., 2011). This fact has aided Big Data's emergence as a very important subject, which continually gained interest from both academia and industry.

The potential utilization of this huge amount of information to transform the way of life in modern smart cities, catapults Big Data and Big data analysis to the forefront of modern research communities, businesses, and governments (Hashem et al., 2015), delivering the promise of a countless new application areas and opportunities, which can emerge under completely diverse application contexts from transportation (Hashem et al., 2016) to health care (Murdoch and Detsky, 2013). As a result, the benefits arising from this wealth of knowledge and information can affect research in numerous ways. This includes for example promoting medical advances by providing evidence in the identification of symptoms and patterns concerning diseases, pandemics or modern health issues, or aiding in the creation of large ground truth databases for scientific fields such as emotion research, and affective computing which are in desperate need of vast amounts of data in order to successfully create emotion models and effective emotion recognition techniques. Modern economy and businesses in the context of a smart city, can also greatly benefit from Big Data and Big Data analytics since they can utilize the data generated from the interaction of users with social networks or smart devices in order to identify the users' preferences, or recognize unsatisfied needs of modern clients or understand the relationships between competitive and collaborating organizations, thus creating better and more appealing services and products, or improving already existing ones. Acquiring more nuanced insights of customer preferences and needs will provide modern organizations and businesses with a crucial advantage over competitors (Sagiroglu and Sinanc, 2013). As mentioned in (Hirsch, 2013) Big Data “is becoming a significant corporate asset, a vital economic input, and the foundation of new business models” (Hirsch, 2013). Fast evolution of ICT technologies, Big Data analytics, and energy efficient communication protocols has provided a new momentum to e-businesses and global connectivity (Qureshi et al., 2017). Big Data analytics can also facilitate government's and smart city local authorities' efforts towards delivering better services to their citizens. Big Data can aid governments in improving healthcare, public transport, education, and other fields of social life thus aiding in shaping a more efficient modern society. For example, data from traffic records can be utilized towards the improvement of public transport services delivered by the state to the population. Especially IoT related Big Data applications such as smart homes, combinations of wearable health devices, and smart cities are widely recognized as key factors which can contribute in the economic and social development of developing countries (Mital et al., 2017).

Utilizing Big Data poses a number of challenges such as security issues and robust handling and storage of data in emergency situations. Enterprises and organizations store, and analyze huge amounts of data, which should be processed in a secure way. This is a very challenging task in the modern environment. Traditional tools are not able to tackle this problem, and this fact is demonstrated in recent studies. This happens due to various reasons such as the centralised nature of Big Data stores or the enhanced capabilities of attackers to penetrate and nuteralise traditional security systems (Sagiroglu and Sinanc, 2013). In order to tackle these challenges companies should incorporate risk aware, contextual, and agile security models (Sagiroglu and Sinanc, 2013). As mentioned in Amin et al.'s paper, there is an identified need for lightweight intelligent security protocols that allows only legitimate users to access the sensitive information collected by various smart devices in the IoT environment (Amin et al., 2018).

In order to harvest the advantages of Big Data analytics in an increasingly knowledge driven society there is a need to develop solutions that reduce the complexity and cognitive burden on accessing and processing these large volumes of data in both embedded hardware and software based data analytics (Maniak et al., 2015; Iqbal et al., 2015a, Iqbal et al., 2015b). Big challenges stem from the utilization of Big Data in real world, since the implementation of real time applications is becoming increasingly complex. This complexity derives from a variety of data related factors. One factor is the high dimensionality degree which a dataset may possess increasing the difficulty of processing and analyzing the data. The interactions, co-relations and causal effects of these high dimensional data parameters in relation to the behaviours and specific outcomes of these systems are often too complex to be analyzed and understood by human users. Additionally, data can be accumulated from diverse sources and input channels, making the online processing very demanding due to the variety of signal input, which needs to be synchronized and diverse data types, which need to be analyzed simultaneously. Furthermore, the collected data is often comprised of multiple types of inputs that are also not always precise or complete due to various sources of imprecision, uncertainty together with missing data (e.g. malfunctioning or inaccurate sensors). Moreover, there is an inherent need in real life applications for high speed storage, processing of data and retrieval of the corresponding analysis results. Another factor that should be taken into account is that the method utilized for Big Data analytics, should extract knowledge from data in an interpretable way. The computational techniques deployed to perform this task should make the underlying patterns, which exist in the data, transparent to the person who tries to utilize and understand them. Finally, there is a need for techniques performing online adaptation, to incorporate contextual and user specific elements in their design, and decision making mechanism, in a user friendly and computationally feasible manner. All the above factors should be reflected in the computational and machine learning techniques utilized in order to process and analyze Big Data so that successful applications and models can be constructed (Suthaharan, 2014).

In the following section, we aim to explore the potential of applying Computational Intelligence techniques to Big Data analytics, and we present recent applications that utilize these techniques towards Big Data analytics tasks. In Section 3, we discuss potential areas where intelligent machine learning techniques can be applied on Big Data analytics resulting in novel and interesting smart city applications. In Section 4, we present our novel data modelling methodology approach. In Section 5, we consider different aspects of policy, protection, valuation and commercialization related to Big Data. Finally, in Section 6, we discuss the conclusions arising from this study.