time series analysis (tsa): patterns, algorithms, examples

‹#›© Cloudera, Inc. All rights reserved.

Mirko Kämpf | April 2016

Time Series Analysis (TSA): patterns, algorithms, examples


Warmup

• Correlation

• Short Term Correlation

• Long Range Correlation

• Return Interval Statistics


Agenda

• Recap: The Data Science Process (DSP) - Where are all that time series?• Similarity Graphs• Hadoop.TS and HDGS• Typical patterns in TSA

• Application of Hadoop.TS• HDGS: Current status• Practical hints


Recap: The Data Science Process

Application of Big-Data-Technology

Images from: http://semanticommunity.info/Data_Science/Doing_Data_Science

http://semanticommunity.info/Data_Science/Doing_Data_Science


Where are the time series?





Where are the time series?



- Events are collected, grouped and sorted

- Normalization of raw series

- quality inspection- derive new information

about the series

- Plot the right charts- Visualize system properties

as networks



“A time series describes a thing over time.” Many time series describes many things over time.

—Why should I care about time series analysis?


“A time series describes a thing over time.” Many time series describes many things over time.

Correlation networks are drived from time series. CNs describe the system.—Why should I care about time series analysis?


Similarity GraphsWhat is similar among nodes?

(a) static properties(b) dynamic properties


Inspection of system properties: data quality screening


Motivation for Hadoop.TS & HDGSOverview & Concepts


Uni-Variat TSAProperties per time series


Distribution of values (PDF) …


Fluctuation AnalysisReturn Interval Statistics

Detect Long Term Correlation in Time Series


Multi-Variat TSAProperties per time series


Similarity Measures

• Cross-Correlation• Event-Synchronization

• Cosine-Similarity

• Granger Causality • Mutual Information

Question: How can I identify significant links?

Modifications and variation lead tobetter results in special use cases.


Get Meaning out of Correlation Metrics …


Application of Hadoop.TS:Results


HDGS: Current statusData Flow, Prototype & Architecture Overview


Hadoop Ecosystem incl. Apache Spark

Spark is our time series workbench.Hadoop.TS brings in domain specific functions.HDGS exposes Metadata and data set properties as „linked data“ in RDF.

Hadoop.TS

HDGS


How to use your code in Apache Spark-Shell?

A. Interactively, by loading it into the spark-shell.B. Contribute to existing Spark projects.C. Create your module and use it in a spark-shell session.D. Build a data-product which uses Apache Spark.

For simple and reliable usage of Java classes and complete third-party libraries, we define a Spark Module as a self-contained artifact created by Maven.

http://blog.cloudera.com/blog/2015/03/how-to-build-re-usable-spark-programs-using-spark-shell-and-maven/

Such packages can be shared and added to any Spark Session via –package option and Maven coordinates.


Hadoop Distributed Graph Space (HDGS)


HDGS (2)

• Deploy a graph-meta-storeas parcel and activate via CSD

• Mediawiki or Fuseki/Jenaare available


Where is the Oscilloscope for Hadoop?


https://www.youtube.com/watch?v=BQ8NHHZ-SuA


Practical Tips


Data Management

• Think about typical access patterns: • random access to each event, record or field?• access to entire groups of records?• variable size or fixed size sets?

• In general, prepare for „full table scan“• OPTIMIZE FOR YOUR DOMINANT ACCESS PATTERN!• Select efficient storage formats: Avro, Parquet• Index your data in SOLR for random access and data exploration • Indexing can be done by just a few clicks in HUE …


Collecting Sensor Data with Spark Streaming …

• Spark Streaming works on fixed time slices only.

• Use the original time stamp? • Requires additional storage and bandwidth• Original system clock defines resolution

• Use „Spark-Time“ or a local time reference: • You may lose information!• You have a limited resolution, defined by batch size.


Thank you !Enjoy Apache Spark and all your data …

time series analysis (tsa): patterns, algorithms, examples

Data & Analytics