Big data Deep learning applications and challenges

Contents● Introduction to big data● Big data challenges● Big data analytics● Applications of big data analytics● Deep learning● Application of deep learning in big

data analytics● Challenges in big data deep learning

Introduction to big data

Introduction

What is big data ?

Big data is loosely defined term

Data is said to be big data when:Volume is very large

Velocity is very high

Variety is broad

As per above definition question arises how do quantify them ?

What should be the limit of terms to make the data big data ?

Need to identify big data

Early detection of big data is necessary To enable proper provisioning of storage and retrieval

To deploy technology for data analytics

For acquisition of requisite expertise for data management and analytics

Big data challenges:

Data analysis is a big challenge due to variety of data which includes structured, semi-structured and unstructured data

Storage of such a large volume of data

Sharing and transfer of data is also a problem when volume and velocity of data generation is that high

Privacy and security of data is serious threat for big data

Big data analytics

Big data analytics:

Big data provides a great opportunity to analyse and extract new information from the data

Due to big data technologies we are able to handle such large volumes of ever increasing variety of data

Data which were considered to be worthless are now being used to exract information

Big data can large datasets to aid machine learning algorithms

Applications of big data analytics

Sensor data analysis

Trend analysis

Management information systems

Cyber physical systems

Network intrusion detection systems

Big data deep learning

Deep learningDeep learning algorithms extract high level complex abstractions

They develop layered hierarchical architecture of learning and data representation

Higher level abstractions are defined in terms of lower-level abstractions

When learning from large amount of unsupervised data in a greedy fashion deep learning algorithms perform quite well

Applications of deep learning in big data:

Extraction from Unlabeled and unsupervised data: Deep learning algorithms can be applied to large amount of unlabeled and unsupervised data for extracting meaningful representations and patterns

Extraction of non-local relationships and patterns:One of the attractive feature of deep learning algorithms is that they can extract non-local relationships and patterns in the data

Context extraction, relational and semantic knowledge extraction

Transfer learning

Practical applications of deep learning in big data analyticsSemantic indexing:

Semantic indexing for search engines: It presents the data in more efficient manner and makes it useful as a source for knowledge discovery and comprehension apart from increasing speed and efficiency

Example: semantic indexing can be used in search engines to make them work fast and more efficiently

In this scheme instead of using raw data for indexing, deep learning can be used to generate high-level abstract data representations for indexing

This semantic indexing works because it allow the data with relatively similar representations to be stored closer to one another in memory aiding in efficient information retrieval

Discriminative tasks and semantic tagging:

Deep learning can be used to extract complicated nonlinear features from the raw data and then use simple linear model to perform discriminative tasks using extracted features as input

MAVIS: Microsoft research audio video indexing system is an example that uses deep learning (ANN) based speech recognition to enable searching of audio and video files with speech

Transfer learning and domain adaptation:Transfer learning: Methods to transfer knowledge/features learned in one or more source

tasks and use it to improve learning in a other related target task

Deep learning can utilize high velocity and variety of big data for transfer learning and domain adaptation

Here training and test data are sampled from different distributions

Deep learning is capable of identifying shared factors present in the input by using transfer learning and domain adaptation

Deep learning challenges in big data analytics

challenges:

There are some problems associated with application of deep learning for big data analytics like:

Learning with fast moving and streaming data

High dimensionality of data

Scalability of the the model

Distributed computing of data

Future research challenges :

Adapting deep learning algorithms to address the analytics of big data

Making learning algorithms work efficiently while focussing on the scalability of the model

Finding criteria for extracting good data representations and domain adaptation

Improving the learning algorithms for distributed computing

References1.Maryam Najafabadi et. el., “deep learning applications and

challenges in big data analytics ”, Journal of Big Data, SpringerOpen Journal 2015.

2.Xue-wen Chen. “big data deep learning: challenges and perspectives”, IEEE, 2014.

3.Huihong He,”using object-oriented big data analytics to reveal server performance dead zone”, IEEE Annual comp. Soft. and applications conf., 2016.

4.Suthaharan Shan, ”big data classification: problems and challenges in network intrusion detection with machine learning”ACM, 2014.

5.Jose Lluis Berral_Garcia, “a quick view on current techniques and machine learning algorithms for big data analytics”, IEEE ICTON, 2016.

Thank you