mapping and analyzing data matrices in real time

7/27/2019 Mapping and Analyzing Data Matrices in Real Time

1/16

Mapping and Analyzing Data Matrices in Real Time

INTRODUCTION

Among other things, Security Information and Event Management (SIEM) is a

system capable of data aggregation, correlation, analysis and reporting of informationsecurity threats in an organization which deals with mammoth and labyrinthine data.One major function of the SIEM tool is to organize and log data collected from multiplesites and then to provide real time, actionable analysis of attacks. The Matrix Mapper(MM) entails the real time incorporation, enhancement and extension of SIEM tocomplex data streams across varying hardware and software platforms to providerelationships between events and entities coupled with predictive analytics.

The SIEM

SIEM is a combination of the following systems:

SIM: Securi ty Information Management - provides storage, powerful analysis andreporting of log data.

SEM: Security Event Management- provides real-time monitoring, link formation andcorrelation among events and generation of alerts.

SIEM covers two major functionalities:

1. Log Consolidation: An organisation receives numerous inputs in different formsfrom various sources creating a diffuse data alluvium which becomes difficult tohandle. The logs are the records of the activities performed by the softwarerunning on a system. These log records cover normal activities, errors,configuration changes, alerts, authorized and unauthorized user access,behavior patterns etc.

2. Threat intelligence and effective analysis: The log and other activities in thesystems are associated through a link and form a consolidated network whichmakes the analysis of occurrence of threats and their preventive measures thatcan be adopted. If such threats are repeated in the same fashion, the patternsare correlated to the previous incidents and preventive measures are eitherperformed or predicated.

Market Overview :

According to a1Gartner Report, 2012: Magic Quadrant for Securi ty Information andEvent Management

1Gartner Report, 2012: Magic Quadrant for Security Information and Event Management

http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sg


2/16

The SIEM market is mature and very competitive and thus SIEM technology demandsare increasing. According to Gartner, it is the fastest rising sub-section of the securitysector with a growing rate of 21% a year. During 2011 the SIEM market grew from $987million to $1.1 billion, achieving a growth rate of 15%. The multiple vendors meet the

basic log management, compliance and event monitoring requirements of a typicalcustomer. The greatest area of unmet need is effective targeted attack and breachdetection. Organizations are failing at early breach detection, with more than 85% ofbreaches undetected by the breached organization. The situation can be improved withbetter threat intelligence, the addition of behavior profiling and better analytics.

According to2Frost & Sull ivan Report

The Asia Pacific (APAC) security information and event management (SIEM) marketwitnessed a healthy growth momentum in 2010 and is expected to grow at a strongcompound annual growth rate (CAGR) of 27.0 percent during 2010-2014. Enterprises

have recognized the importance of SIEM in ramping up their security posture. The AsiaPacific Security Information and Event Management (SIEM) Market CY 2010, finds thatthe market earned revenues of $93.4 million in 2010 and estimates this to reach $242.7million in 2014.

According to an 3IDC Report

An IDC study examines the worldwide IT security products market for the period from2010 to 2015, with vendor revenue trends and market growth forecasts. Worldwidemarket sizing is provided for 2010, and a growth forecast for this market is shown for20112015. A vendor competitive analysis, with vendor revenue and the market sharesof the leading vendors, is provided for 2010.

"Following a difficult 2009, security revenue rebounded well in 2010 with an overallgrowth rate of 9.1%. This was a full percentage point higher than what had beenexpected. Organizations and enterprises upgraded their security in 2010 due to a muchmore difficult threat environment and technology innovation. In 2011, we expect securityspending to remain brisk with growth expected to be nearly 10%," said Charles Kolodgy,research vice president for Security Products. "Security is required for organizations thatwant to expand into cloud computing, increase their use of mobile and virtualtechnologies, and deal with increasing regulatory requirements.

2Asia Pacific SIEM Market to Reach $243 Million in 2014

http://www.frost.com/prod/servlet/press-release.pag?docid=226373777

3Worldwide IT Security Products 2011-2015 Forecast and 2010 Vendor Shares

http://www.idc.com/getdoc.jsp?containerId=232221
http://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.idc.com/getdoc.jsp?containerId=232221http://www.idc.com/getdoc.jsp?containerId=232221http://www.idc.com/getdoc.jsp?containerId=232221http://www.frost.com/prod/servlet/press-release.pag?docid=226373777


3/16

The MATRIX MAPPER (MM)

Matrix Mapperis under development as a strong, effective and advanced analytics toolhaving its application in sectors as diverse as mining, oil & gas, power (generation &distribution), airlines, land and water transport, shipping, chain stores, agriculture/ food

distribution, warehousing, courier concerns, accounting, banking and insuranceGovernment agencies/ departments, defence forces, police, forest conservation,counter terrorism units, municipalities, railways and a variety of Public SectorUndertakings would find Matrix Mapper to be a kind of force multiplier.

Matrix Mapper as an SIEM solution can be seen as one out of hundreds of applicationareas with area specific integrations and wider implementations in log management,analysis of data, threat intelligence, network forensics etc.

Theoretical Basis of the MM

1. The collation of enormous data which could be text, audio, video, images, graphicsetc. received by the police agency through diverse means and sources.

2. This data is then warehoused i.e. all the data is stored together, in once place, with ahomogenous structure which allows interactive analysis. In effect, the data is reduced to


4/16

the common denominator of a uniform database by way of what are callednormalization tables. This data is then ready to be analyzed.

3. The data is now to be mined i.e. knowledge is to be extracted from the warehouseddata through the application of appropriate algorithms which will be able to categorize

the data on the strength of defining parameters like say individuals, places, transportused, channel of monetary remittance etc., such categorization (or gradation orclustering or classification or association) depending on what the programme user islooking for. This exercise is also known in the cyber world as big data analytics.

4. Next come algorithms that subject the categorized data to network analysis viz. thelinkages, connectivities or relationships between elements in the various categories. Forinstance, who X is in touch with, how is he in touch with and how often is he in touchwith and then how are these relationships related to each other and to what degree andin what manner.

5. Finally comes the turn of predictive analysis. It includes the application of "what if"scenarios to the relationships that have been networked to plot the logical direction of"what next" viz for example what transport is 'X' likely to take to travel to which place,with whom and when.

Predictive analysis was earlier done using legacy data. Now it is also used to correlate

legacy data with real time event reporting applying SIEM (Security Information and

Event Management) technologies so as to be able to secure logical pointers for the

direction (or conclusion) towards which ongoing events are headed.

Types of data collected:

i) Business transactions including money movements.ii) Scientific data including RFID, technical surveillanceiii) Medical & personal data images, voice clips, personal profile & historyiv) Surveillance video and picturesv) Satellite sensingvi) Legacy datavii) Digital media scanned material, films, voice and video collectionsviii) Graphics data maps, drawings, schemes , sketchesix) Virtual data in cloud, digital repositories, mail, SMS

Imagery databases

The interpretation of images involves categorising or identifying data, in an image, bycorrelating it with domain knowledge. So the key to good image interpretation is domainknowledge.


5/16

Domain

Knowledge

Data Labelling

Tuples

Wrinkles

Dimples Jawline

Lip contour etc.

Feature

Extraction &

Algorithm

Interpretative

Conclusions &

Inferences

One way by which such correlation can be achieved is to identify the correspondenceand coincidence between captured image data and stored models.

In the case of sequential images, e.g. videos, each sequence has to be taken andexamined in terms of correspondence/ coincidence/ relationship with the stored models.

Another method can be to mimic the understanding/ interpretative process of the humaneye and the categories in which images, captured by the eye, are categorised andinterpreted by the brain.

All in all, image interpretation and analysis necessarily predicates the availability ofstored models or prior reference points regardless of the precise procedure adopted foridentification, categorization and/ or recognition


6/16

Audio Databases

This part of data mining works on simple principles. Firstly, there have to be sample

audios which already stand identified and labeled in relation to persons, things or

entities. Content analyzing software, part of the MM, uses a captive algorithm to analyze

the tempo, beat, amplitude and frequency of the audio rather than the encodinglanguage per se. Each of these factors, taken together, go to produce a complete

fingerprint for the audio sample which, in turn, makes it possible to match with other

audio files in the database that yield an identical fingerprint.

Spatial Databases

4Spatial data mining (SDM) looks for patterns rather than random features. The

common spatial features are location prediction, feature interaction and hot spots. SDM

is, therefore, the search for unexpected, interesting patterns in large databases.

Techniques used in SDM include classification, associations, clustering and outlier

detection.

It involves discovering the nuggets of useful, unexpected spatial patterns in large

databases; very like looking for a needle in a haystack. Examples of vast amounts of

spatial data are inputs through satellite imagery, sensors on highways, GPS tracks etc.

The basics of the probability calculus in SDM can be stated as:

4Spatial Databases a tour by Shashi Shekhar and Sanjay Chawla :

www.spatial.cs.umn.edu/Book/slides/


7/16

Given a set of events , the probability P is a function which satisfies thefollowing two axioms:

P() = 1

and If A and B are mutually exclusive events then P(AB) = P(A)P(B)

Conditional Probability:

Given that an event B has occurred the conditional probability that event Awill occur is P(A|B). A basic rule is:

P(AB) = P(A|B)P(B) = P(B|A)P(A)

5Associat ions, Spat ial Associat ions , Co-location

To determine patterns from the following dataset:

5Spatial Databases a tour by Shashi Shekhar and Sanjay Chawla :

www.spatial.cs.umn.edu/Book/slides/


8/16

Features of the MATRIX MAPPER

Matrix Mapper is a single tool with multiple technologies as its components. Thefeatures of Matrix Mapper are:

(i) Handling of big data by the MM

The tsunami of unstructured data such as contact details, financial transactions,remittances, video, audio, graphics, mails, logs, web files, records etc. is fed as input forMM. The MM supports data aggregation and collection from disparate sources such asmobile device, fixed line services, Geospatial information, relational database records,which is quickly sorted into structured form for further processing. It is capable ofhandling big colossal data, from gigabyte to petabyte and exabyte scales which has tobe efficaciously, reliably and quickly analysed to yield useful results.The major functions covered are as follow:

a) Collection of logs and events from different sources like security devices (firewall,

antivirus, and other UTMs), applications and software, access management

products etc.

b) Collection of network flow data from switches and routers.

(ii) Easy to use workspace IDE

Thousands of 2D and 3D icons to accommodate large number of different entities at

one time.

Drag and drop facility for quicker response and chart generation.

Information filtering and cluster analysis.

Option of searching maps and accessing charts simultaneously.

Zooming capability.

Timeline, charts, report etc. generation.

Automatic animation.

Different linkages and network options

Find links, Find path, Entity search and Visual search options are available.

Dynamic group association through SNA.

Highlight key players.

Flexible controls


9/16

(iii) Knowledge Discovery:

Matrix mapper then collates the big data for aggregation and correlation on whichknowledge discovery of entities and events is performed.

It is a highly domain specific tool where different knowledge discoverymethodologies are applied for intelligent data retrieval.

Various queries are operated to obtain simplified results from complex data.

The queries help in identifying trends and patterns by building links and relationsamong the entities. The highly efficient semantic capabilities of Matrix Mapper alsoensure the coverage of uncommon and duplicated data.

The analysis of trends and patterns provides the key players and entities fromnetwork for deeper forensic investigation.

(iv) Social Network Analysis (SNA):

Huge and nested complex networks of relationships among the different groupsof entities and events.

Powerful visualization by way of various techniques such as heat matrix wheresome bold lines represents strong or direct relationship or a lighter link showinginterconnections among contacts of contacts of contacts etc.

Matrix mapper would incorporate powerful visual tool to comprehend, decipherand derive intelligent conclusions from networks.

(v) Predictive Analysis:

Matrix Mapper is capable of predictive analysis whereby historical facts enable the

mapping out, in advance, the probability of future events. It would address issues like

what could happen next? so important to successful strategic management. Predictive

Analysis could be approached through a number of methodologies. Some of these are:

6The greatest number of incidents or heat in relationships or frequency of incidents areplotted on relevant area maps to show up kernels of intensity where the likelihood of

6Analytics in Policing: Predictive Policing & Location Intelligence

National Police Academy, Mahesh Narayan, May 2, 2013


10/16

recurrence would be high. Thus, geo spatial data is matched with incident and imagerydata. For instance, corridors and landmarks in a certain area can be associated withgreater risk. In such places, robbery risk can be a function of prior crimes plus disorderlyand suspicious activity calls.

The thumb rule of Hot Spot and Modus Operandi analysis is:

Future Crime ~= [past crime]

(vii) Customization:

The Matrix Mapper is highly customizable for the use for different businesses in sectors

as diverse as mining, oil & gas, power (generation & distribution), airlines, land and

water transport, shipping, chain stores, agriculture/ food distribution, warehousing,

courier concerns, accounting, banking and insurance. Government agencies/

departments which would find Matrix Mapper to be a kind of force multiplier include the

defence forces, police, forest conservation, counter terrorism units, municipalities,

railways and a variety of Public Sector Undertakings.

(viii) Database:

A flexible centralized database is used for effective analysis from pool of data. Fromhigher growth perspective Matrix Mapper can have capabilities that aid in targetedattack detection, including support for data access, user activity, applicationactivity monitoring, profiling and anomaly detection, threat intelligence andeffective analytics.

Connaught Place Delhi Crime Prediction based on event

data, local geography and high risk locations.


11/16

Architecture:

The architecture is comprised of following major entities:(a) Data Accumulator; (b) Log Manager; (c) Analytics Engine; (d) Predictive Results

1. Data Accumulator:

The data accumulator is an agent that collects data from firewalls,software applications, IDS/IPS, antivirus, UTMs, router information, IPaddresses, normal activities, errors, configuration changes, alerts,authorized and unauthorized user access etc.

It splits the data streams to form logs.

2. Log Manager:

The log manager is dedicated to the log related activities in database. There are twolog managers each for real time activities and long term activity analysis. The realtime log manager acts on the operation directly on the single log database and


12/16

expects real time results while the other log manager operated on centralizeddatabase pool.

3. Analytics Engine:

The analytics engine is responsible for

Detailed data access: The engine is capable of scanning and parsing the

data on which semantics are applied. It performs events and flow data

searches in near real-time streaming mode or on a historical basis to enhance

investigation.

Event identification: The engine identifies the potential events mined from

the database for correlation and network formation.

Incident and event correlation: It tracks significant incidents and threats,providing links to all supporting data and context.

Recognize trends and behavior patterns: It automatically discovers most

log source devices and monitors network traffic to find and classify hosts and

serverstracking the applications, protocols, services and ports they use. It

also includes a view to access near real-time analysis, incident management

and reporting.

Intuitive Report and chart generation: It produces visually dexterous

reports and charts for the real time as well as long term analysis results.

4. Predictive results:The results are based on the historical evidences available inthe log and the alerts, notification and updates are generated accordingly.

MATRIX MAPPER- STAR: COMING TOGETHER OF THE SIEM AND THE SBMS

Matrix Mapper- Star is a proposed Secure Messaging cum Analytics Tool to be

developed by Aarken Technologies (Aarktech). It is an integration of Aarktechs mostpromising and unique, secure messaging system - SBMS and the powerful analyticaltool, Matrix Mapper. SBMS ensures secure real time communication of complex data(text, graphics, audio and video) even in low bandwidth with low latency and MatrixMapper is a powerful analytics tool. Integration of the two produces a powerful tool thatimproves the scalability of an SIEM tool and can fit into organizations with globallyconnected IT infrastructure. The secure data transmission is the need of organizationsdealing with critical data and decisions are rely heavily upon analysis of that data. For


13/16

example, an oil company with global presence requires regular analysis of variousonshore and off shore activities but the communication has to be achieved in captiveand secure environment which signifies the need and importance of integration ofAdvanced Matrix Mapper with SBMS thus producing intuitive Matrix Mapper-Star. Thisinnovative concept would usher in a revolution in the SEIM and Big Data Analysis

industry.

Architecture:

The diagram below would throw more light on the design principles of this concept.


14/16

Suppose a company has several Regional/Zonal offices each having its own SIEM. ThisSIEM would spot and flag attacks that would otherwise go unnoticed. The flagged datain the log database at each region/zone is then sent to centralized database located inthe country office via SBMS that ensures low latency at low bandwidth transmission.The data streaming in is coalesced into the normalization table of the legacy database

and subjected to the analytics engine of Matrix Mapper which performs the analytics inreal time and extracts useful patterns The real time pattern identification and predictiveresults with directions for actions to be taken thereon are routed back to regional/zonaloffices securely with assured delivery that is achieved by SBMS.

SUMMARY CONCLUSIONS

As an incident management tool, SIEM can be highly effective at increasing

organisations security layer to identify and handle a large number of events while

simultaneously analyzing them to improve the accuracy of threat identification

thus increasing effectiveness in detecting and responding proactively to security

threats. The innovative concept of Secure messaging cum Analytics tool

reflected in proposed Matrix Mapper- Star is an intuitive need of Security and

Vulnerability Management domain to ensure internal as well as network security

for industries that are engaged in critical communication and exchange.


15/16

BIBLIOGRAPHY:

1. 1 Gartner Report, 2012: Magic Quadrant for Security Information and EventManagement

http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sg

2. 2 Asia Pacific SIEM Market to Reach $243 Million in 2014http://www.frost.com/prod/servlet/press-release.pag?docid=226373777

3. 3 Worldwide IT Security Products 2011-2015 Forecast and 2010 Vendor Shareshttp://www.idc.com/getdoc.jsp?containerId=232221

4. 4,5 Spatial Databases a tour by Shashi Shekhar and Sanjay Chawlawww.spatial.cs.umn.edu/Book/slides

5.

6

Analytics in Policing: Predictive Policing & Location IntelligenceNational Police Academy, Mahesh Narayan, May 2, 2013

6. http://en.wikipedia.org/wiki/Siem

7. http://en.wikipedia.org/wiki/Security_information_and_event_management

8. http://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEM

9. http://www.certconf.org/presentations/2005/files/WC4.pdf

10.http://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architect

11.http://www.slideshare.net/vikasraina/SIEM

12.http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781

13.http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-

quadrant/32874165

14.http://www.aarktech.net/solutions-news/matrix-mapper.html
http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sghttp://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sghttp://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.idc.com/getdoc.jsp?containerId=232221http://www.spatial.cs.umn.edu/Book/slideshttp://www.spatial.cs.umn.edu/Book/slideshttp://en.wikipedia.org/wiki/Siemhttp://en.wikipedia.org/wiki/Siemhttp://en.wikipedia.org/wiki/Security_information_and_event_managementhttp://en.wikipedia.org/wiki/Security_information_and_event_managementhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://www.certconf.org/presentations/2005/files/WC4.pdfhttp://www.certconf.org/presentations/2005/files/WC4.pdfhttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.slideshare.net/vikasraina/SIEMhttp://www.slideshare.net/vikasraina/SIEMhttp://www.slideshare.net/vikasraina/SIEMhttp://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.aarktech.net/solutions-news/matrix-mapper.htmlhttp://www.aarktech.net/solutions-news/matrix-mapper.htmlhttp://www.aarktech.net/solutions-news/matrix-mapper.htmlhttp://www.aarktech.net/solutions-news/matrix-mapper.htmlhttp://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.techrepublic.com/whitepapers/gartner-2012-siem-magic-quadrant/32874165http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.sans.org/reading_room/whitepapers/logging/practical-application-sim-sem-siem-automating-threat-identification_1781http://www.slideshare.net/vikasraina/SIEMhttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-security-architecthttp://www.certconf.org/presentations/2005/files/WC4.pdfhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://searchsecurity.techtarget.com/definition/security-information-and-event-management-SIEMhttp://en.wikipedia.org/wiki/Security_information_and_event_managementhttp://en.wikipedia.org/wiki/Siemhttp://www.spatial.cs.umn.edu/Book/slideshttp://www.idc.com/getdoc.jsp?containerId=232221http://www.frost.com/prod/servlet/press-release.pag?docid=226373777http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sg


16/16

mapping and analyzing data matrices in real time

Documents