metricbase: iot, sensors and automation · 10/9/2018 · 12 © 2018 servicenow, inc. all rights...

MetricBase: IoT, sensors and automation

2 © 2018 ServiceNow, Inc. All Rights Reserved. Confidential.

Agenda

Introduction, Goals, and Context

Setup—Introducing the Thing Simulator

Data Modeling and Discovery

Metric Modeling

Data Acquisition

Visualization and Reporting

Triggers and Automated Actions

Wrapping Up


Safe Harbor

This presentation contains “forward‐looking” statements that are based on our management’s beliefs and assumptions and on information current ly available to management. We intend for such forward‐looking statements to be covered by the safe harbor provisions for forward‐looking statements contained in the U.S. Private Securit ies Lit igation Reform Act of 1995. Forward‐looking statements include information concerning our possible or assumed strategy, future operations, financing plans, operating model, financial posit ion, future revenues, projected costs, competit ive posit ion, industry environment, potential growth opportunit ies, potential market opportunit ies, plans and objectives of management and the effects of competit ion.

Forward‐looking statements include all statements that are not historical facts and can be identified by terms such as “anticipates,”“believes,” “could,” “seeks,” “est imates,” “expects,” “intends,” “may,” “plans,” “potential,” “predicts,” “projects,” “should,” “will,” “would” or similar expressions and the negatives of those terms, although not all forward‐looking statements contain these identifying words. Forward‐looking statements involve known and unknown risks, uncertaint ies and other factors that may cause our actual results, performance or achievements to be materially different from any future results, performance or achievements expressed or implied by the forward‐looking statements. We cannot guarantee that we actually will achieve the plans, intentions, or expectations disclosed in ourforward‐looking statements and you should not place undue reliance on our forward‐looking statements.

Forward-looking statements represent our management’s beliefs and assumptions only as of the date of this presentation. We under take no obligation, and do not intend to update these forward‐looking statements, to review or confirm analysts’ expectations, or to provide interim reports or updates on the progress of the current financial quarter. Further information on these and other factors t hat could affect our financial results are included our filings we make with the Securit ies and Exchange Commission.

This presentation includes certain non‐GAAP financial measures as defined by SEC rules. We have provided a reconciliat ion of those measures to the most directly comparable GAAP measures in the Appendix.

The information in this presentation on new products, features, or functionalit ies is intended to outline ServiceNow’s general product direction and should not be included in making a purchasing decision. The information on new products, features, functionalit ies is for informational purposes only and may not be incorporated into any contract. The information on new products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and t iming of any feat ures or functionality described for our products remains at ServiceNow’s sole discret ion.


What You Will Learn

HOW TO

VIEW AND ACT ON

TIME SERIES DATA

3

HOW TO PLAN AND DESIGN

A TIME SERIESDATA MODEL

THINGS TO CONSID ER WHEN PUSHING

TIME SERIES DATA TO SERVICENOW

21


MetricBase—Time Series in ServiceNow

Industry trends

• IoT—more and more ‘things’ in ServiceNow—both IT and non IT

• Monitoring—constant stream of measured data—time series

• Fast reaction—real time (or predictive) automated action on measured condition

• Analytics—long-term trends, automated modeling, and anomaly detection

Challenges

• Scale—speed and volume beyond relational database

• Heterogeneous environments—requiring complex integrations

• Complexity—how much data, and what to do with it?

MetricBase

• Time Series platform native to ServiceNow


MetricBase vs. ITOM Operational Intelligence

• OI is an out-of-box solution for IT ‘Things’

• It is built on top of MetricBase

• MetricBase features and APIs can be used directly

– With OI data, to complement it

– With non IT ’things’ where OI isn’t appropriate

• Facilities, consumer, manufacturing, energy, environment, transportation, …


Setup: GitHub Setup

1. Go to “Repositories”

2. Sign in– Your login

3. Fork and Import “MetricBaseThing Monitor”

4. Check out the first checkpoint:Exercise 1

Lab Setup/Exercise 1

1


IoT Demo Challenge

• Which things?

• The real things: cumbersome and messy

• Demo data: too predictable

• Something in between?

Exercise 1


The MetricBase Thing Simulator Exercise 1


The MetricBase Thing Simulator

https://clothosimulator.service-now.com/water

or from your Lab instance:

“Water Filtration Simulator”

Exercise 1

https://clothosimulator.service-now.com/water


The MetricBase Thing Simulator

• Tanks (18)–Monitoring: volume of water

• Pumps (32, +2 hidden)–Can be on, off, or bypass (flows by gravity)

– Flow depends on head (difference between water levels) and pump model

–Monitoring: flow and temperature

• Water flows from tank to tank

• Goal: Keep a continuous flow from left to right, using all tanks

Exercise 1


Chaos Monkeys

• That’s you

• Pumps can be turned on, off, bypassed

• Pipes can clog, tanks can leak

• Basic sensors (volume of water, flow, temperature)

• Built in safeguards (like any good system)

• We’ll show how to use ServiceNow to extra value from your data

Exercise 1


Exercise 1: Setup (continued)

Before you do anything: change scope to ’MetricBase Thing Monitor’

1

2

3

4

5

Exercise 1


Exercise 1: Keeping Track of Things

• Sensors are associated to records in ServiceNow

• We need a data model

• We need and inventory of things

Exercise 1



1. Create tables and columns to record tanks and pumps

2. Import inventory from the Thing Simulator

Once done, you should have:

1. A ServiceNow representation of all things in the Thing Simulator

Exercise 1



• This workshop is not about creating table and columns

• Everything is already there

Exercise 1


Exercise 2: Defining Metrics

• Time series period and retention are defined in the ‘Time Series Metrics’ table

Exercise:

1. Create metrics for measures you want to keep in ServiceNow

2. Display them in a list

Once you’re done, you should have:

1. Empty columns for metrics in a list view

Exercise 2


About Metrics and Metric Definitions

• Metric: a special field on a record capable of storing time series data

• Described in the ‘Time Series Metrics’ table

• Metrics have a label, name, and retention policies

– … and some other properties we’ll explore later

Exercise 2


About Retention Policies

• Each metric has a fixed retention policy

– Those are immutable, beware!

• Retention policies define how much data you want to keep, for how long

– E.g., ‘Dense’

• 8 days of data at 1 minute intervals

• 3 months at 10 minutes

• 1 year at 1h intervals

–Data is compacted by averaging

• Cost is (loosely) based on usage, so choose carefully

Exercise 2


Exercise 2: Defining Metrics

1. Go to the ‘Time Series Metrics’ table (navigator)

2. Create 3 new metrics

Table Label Name Retention Policy

Tanks Volume volume Dense

PumpsFlow flow Dense

Temperature temperature Dense

Exercise 2


Exercise 2: Configure List to Show Metric fields

1

2 34

Victory!

Exercise 2


GitHub Checkpoint

2

1

Exercise 3


IoT Design at Scale

• Collectors

– Device dependent

– Live in your infrastructure

– Basic data normalization

• Broker

– e.g., Kafka or RabbitMQ

• Pusher

– Broker dependent

– Allows batching

– Advanced normalization

– sys-id cache

Exercise 3


IoT Design: Thing Simulator

Collector:

– Converts pressure to volume

– Converts sensor id to tank id

Pusher:

– Batches measures

– Map tank id to sys_id

Exercise 3


IoT Design: Things Simulator, Simulated

Inverted model:

• Stable things

• Multiple transient instances

➔Simplified design

➔Pull instead of push

Exercise 3


• This is a cost optimization exercise

• Aim for deliberate collection

– Collect and store the data you need

– Avoid collecting and storing values just because you can measure it

• Collect in the ‘right’ form

– Optimize for processing

– E.g., disk space: %used, %free, vs bytes, vs MB, vs blocks

– Account for 5 significant digits (floats)

Cost

Performance

Valueof theData

About Series Selection Exercise 3


Exercise 3: Acquiring Time Series Data

• Measured Time Series data must be pushed to ServiceNow

Exercise:

1. Look at data coming in from the Thing Simulator


1. Recent data visible in the list view

Exercise 3


Exercise 3: List View and Drill Down

1

Exercise 3


Exercise 3: Time Machine

Exercise:

1. Click ‘load data’ in Navigator to accelerate time


1. Older data visible by clicking on a list view metric field

Exercise 3


Exercise 3: List View and Drill Down (with More Data)Exercise 3


Exercise 4: Visualizing Time Series Data

Time Series data can be displayed in list, details, and reports.

We’re going to:

1. Change pump temperature list visualization

– Show average temperature over the last 24h

2. Use reports:

– Aggregates, Group by, Filters

Once you’re done, you should:

• Be comfortable with reporting on MetricBase data

Exercise 4


Exercise 4.1: Change List Displayed Value

1. Go to “Time Series Metrics” table

2. Pick pump temperature

3. Change display from ‘last’ to ‘average of the last hour’

4. Go back to the Pumps list to verify the change

Exercise 4.1


Exercise 4.2: Where Did the Water Go?

There appear to be less water in the simulator than at the beginning.

Let’s find out!

1. Go to Tanks

2. Open chart designer

3. Chart the sum of all volumes over time

Exercise 4.2


Exercise 4.2: Go to Chart Designer

1 (right

click)2 pick “Time Series

Chart Designer”

Exercise 4.2


Exercise 4.2: Chart Leaks

1

2

3

Exercise 4.2


Exercise 4.3: Confirm the leak

• Tank 1-1 is suspect. To confirm:

1. Isolate tank 1-1

(Turn all the pumps off)

2. Create a new report for tank 1-1 with:

– Volume over time

– Flow of all pumps to and from this tank (stacked report)

3. Observe: flow is 0, however volume goes down

Exercise 4.3


Exercise 4.3: Confirm the Leak

1. Go back to “Time Series Chart Designer” for tank volume

2. Limit your chart to the relevant tank

3. Chart on the last few minutes (since we isolated the tank)

1

2 3

Exercise 4.3


Exercise 4.3: Double Check Flow (stacking charts)

Problem: we need to confirm that there is no incoming or outgoing flow

1. Save your report

2. Use “Show Report Structure” to add a stacked report

3. Add dataset showing flow of pumps linked to tank1-1

1. Pick “Pumps” table, “Flow” metric

2. Filter pumps to tank 1-1 or from tank1-1

4. “Save Dataset”

1

Exercise 4.3


Exercise 4.3: Leak Report Exercise 4.3


Exercise 4.4: Compare Pump Models

Which pump model is better?

1. Go to Pumps

2. Create a new report:

1. Group by Model

2. Display Temperature

3. Decide which model of pump you want to keep buying

Exercise 4.4


Exercise 4.4: Reporting Trick

Make it pretty!

2

1

Exercise 4.4


Exercise 4.4: Compare Pump Models Exercise 4.4


GitHub Checkpoint

2

1

Exercise 5


Exercise 5: Trigger and Automated Actions

MetricBase allows you to react to time series events.

IFTTT for time series

Let’s open an INT if a tank gets empty!

3 parts:

1. Trigger Definition

2. Trigger: trigger definition + record filter + [additional processing]

3. Flow

Exercise 5.1


Exercise 5.1: Open INT on Empty Tank (Definition)

Exercise:

1. Create a trigger definition

1

2

3

5

6

4

Exercise 5.1


Exercise 5.1: Open INT on Empty Tank (Levels)

Triggers transition between levels

Level 0 is the base level (everything is normal)

Other levels can be rising levels of alerting

1 2 3

4

Exercise 5.1


1. On to Flow Designer…

2. Create a new flow

3. Select a MetricBase type trigger

4. Pick your newly defined trigger

5. Add flow logic and an action to open an INT on level 1

6. Save and Activate

Exercise 5.1: Open INT on Empty Tank (Flow)

1

Exercise 5.1


Exercise 5.1: Verification

Let’s empty a tank!

You should see an INT appearing.

1

Exercise 5.1


Triggers React on Level Transition

Series already in an elevated trigger state on trigger creation (or registration) will not trigger.

Exercise 5.1


Exercise 5.2: Complex Triggers

Pumps can clog … but we don’t monitor that

However: clogged ➔ no flow, high temperature

Trigger on the least likely condition– Temperature in this case

–Use the additional script to confirm no flow

Exercise 5.2


Exercise 5.2: Complex Triggers

Exercise:

1. Create a trigger definition for high pump temperature (>62)

2. Use trigger definition in a flow– Ignore AnglerFish pumps

– Verify condition with additional filtering script

Exercise 5.2


Exercise 5.2: Verification

Wait for a clogged pump

You should see an INT appearing

Exercise 5.2


Triggers Design and Best Practices

• Beware transition spikes

• Use additional filter for deduplication

– Ideally: fast, stateless script

• Use flow logic when number of events isn’t a concern

• Try to keep record filter stable

• But exclude records you don’t care about

Exercise 5.2


Next Steps

• Implement advanced triggers

• Explore the JavaScript API

• Design complex reports

• Lab instance will shut down in a couple hours– … but talk to us if you want to experiment


Top Takeaways

VIEW AND ACT

USING REPOR TSAND TRIGGER S.

3

PLAN AND DESIGNDELIBER ATEL Y

FOR SCALE.

PUSHEFFICIENTLY .

21

October 9–10 • Sydney

metricbase: iot, sensors and automation · 10/9/2018 · 12 © 2018 servicenow, inc. all rights...

Documents