Monitoring in the era of Cloud Computing

SpeakersAttilio Broglio - abroglio@create-net.orgSilvio Cretti - scretti@create-net.org

Webinar on November 26, 2015 - 15:00 CET

Who are we and what is our offer ?

Create-Net is a research centre active in the ICT field and located in Trento (Italy)

Our Cloud Computing offer comprises:Research and Consultancy in the Cloud Computing field

cloud architectures, cloud deployment, cloud management, distributed cloud, energy efficiency in the cloud

Training on Cloud Computing/Openstack (authorized Mirantis training partner)

OpenStack free webinar list offered by Create-Net

Webinar Url

OpenStack fundamentals https://goo.gl/VRg5Nr

Platform as a Service (PaaS) and OpenStack https://goo.gl/VRg5Nr

OpenStack Networking https://goo.gl/Htcs2s

Monitoring in the era of Cloud Computing

Agenda

1. Introduction to Monitoring

2. Introduction to Ceilometer Architecture

Agents (Compute/Central)

Storage & API

Quick Demo

3. Introduction to Monasca Architecture

Events/Messages

Storage & API

Quick Demo

4. Conclusions and Trends

5. QA session

1.Introduction to Monitoring

...in the Cloud Computing era

Introduction to the Monitoring

Monitoring a cloud requires a unified view of physical and virtual resources and of cloud services focusing on the customer SLA

Two complementary topics:monitoring a cloud computing infrastructure (ref.

Ceilometer)Monitoring as a Service - xaaS - not limited to a

specific cloud and offering higher level functionalities above the simple data collection (ref. Monasca)

Tools of the trade

How to monitor a (federated) cloud?

What is on the market now?

Some examples:

OpenNMS

Nagios

Zabbix

IBM Tivoli - Netcool

Zenoss

Openstack Ceilometer

OpenStack Monasca

Modular (micro services)

Distributed and scalable

Fault tolerant

Open Source & Open API

Support for Big Data analytics

(trend)

2.Introduction to Ceilometer

“..quando un nuvol vada sovr’essa..”[Inf. XXXI ]

Introduction to Ceilometer (1/2)

OpenStack provides infrastructure as a service (IaaS) to end clients, it’s necessary to be able to meter its performance and utilization for:

billing benchmarking scalability statistics

Ceilometer aims to become the infrastructure to collect measurements within OpenStack. Main goals:

efficient collection of metering (CPU, network costs, ..) collecting data by monitoring notifications (from services or by polling the HW) accessing and inserting the metering data through the REST API expanding the framework to collect custom usage data (by plugins)

Introduction to Ceilometer (2/2)

What is a meter?Three type of meters are defined in ceilometer:

Cumulative: Increasing over time (i.e. instance hours) Gauge: Discrete items (i.e. floating IPs ...) and fluctuating values (such as disk I/O) Delta: Changing over time (i.e. bandwidth)

Each meter is collected from one or more samples, which are represented by counter objects:

counter_name: meter id (i.e. disk.ephemeral.size) counter_type: counter types (cumulative, gauge, delta). counter_volume: measured data (CPU, #cores)

Ceilometer-Architecture (1/2)

Agents:Rest API

central Agent compute Agent

collectors data Stores

AMQP

glancelibvirt

nova-compute

RPC RPC

hypervisor APIAPI

Rest API: access to metering data

central: Retrieve info not linked

to instances or compute node

compute: polls metering data and

instances statistics from the

compute node

AMQP: message broker (RabbitMQ)

data stores: mongo (read/write)

collectors: message queues for

metrics

Ceilometer-Architecture (2/2)

Ceilometer offers 5 core services divide in 3 main groups:

data agents collection alarming

The 5 blocks “daemons” are:1. polling agent: polls OpenStack services and build Meters.2. notification agent:

listens to notifications on message queue converts to Events and Samples apply pipeline actions

3. collector daemon: gathers and records event and metering data

4. api service: to query and view data recorded by collector service

5. alarming: alarming system based on rules

Data Agent Collection AlarmingPolling Notification Collector API Alarming

Ceilometer-Agents (1/3)

How can we collect meter/sample?Two methods available to collect data:

Bus listener agent: events on the notification bus transforms them into Ceilometer samples community guideline (Oslo library) compliant with Notification Agent

Polling agents, polls API or other tool to collect information

(Δt) local or remote hypervisor API

Ceilometer-Agents (2/3)

Notification Agent?..it listens for data

consumes messages from services (Nova, Glance, Cinder, Neutron, ..)

loads one or more listener pluginsthat listen to any topics (by namespace)

redirects to the proper endpoint(pipeline)

Event matching → wildcards e.g. compute.instance.*.

Notifications can be converted into Events

Ceilometer-Agents (3/3)

Polling Agent?.. it asks for data

compute node: Polling for compute resources is handled by a polling agent running on the compute node (where hypervisor is)

central node: Polling via service APIs for non-compute resources.

The polling agent daemon is configuredto run one or more pollsters

Δt for sample from pollsters (set on pipeline)

passes the samples to the pipeline for processing

Ceilometer-Storage and API (1/2)

Pipeline?..manipulates dataCeilometer uses pipeline for:

manipulate and publish samples it in various combinations

combined by historical or temporal context transform by a scaling factor publishes using 3 different transports:

1. notifier: a notification based that pushes to a message queue: (i.e. collector, external system)

2. udp: publishes samples using UDP packets;3. kafka: publishes to a Kafka message queue

Ceilometer-Storage and API (2/2)

Collector? gathers the processed event and metering data

(notification and polling agents) validates data writes the messages to: database, file, or http

From Juno, database is divided into 3 connections: alarm

event

metering

API?RESTFUL API (read/write data)

raw statistics/filter/group

Ceilometer-Demo

CC License - Photo by Amy Loves Yah

3.Introduction to Monasca

“intrammo a ritornar nel chiaro mondo..”[Inf. XXXIV ]

Monasca - Introduction (1/2)

MONitoring As a Service sCAlable: Highly-performant, scalable Fault-tolerant, capable of big data retention

process 100s of thousands metrics/sec data stored for ~ 1 year

Based on REST API (http not standard CollectID, NSCA ”Nagios”) storing querying (metrics/historical data)

Multi-tenant (by keystone) metrics submitted and authenticated by Keystone metrics stored with tenants reference

Monasca - Introduction (2/2)

MONitoring As a Service sCAlable: Extensible based on micro-services message bus architecture Real-time events processing

threshold/alarm ->Metrics threshold/alarm ->storage/retrieval/statistics

Notification system Real data:

Performances: ~ 25K metrics/sec on three HP Proliant SL390s G7 Apache 2.0 license Who is working on it: HP, IBM, Rackspace

Monasca - Architecture (1/4)

Monasca API (REST)

Horizon

Dashboard

Monasca

Client (CLI)System

Monasca

Agent

Config DB

mySQL

Metrics

Events DB

Alarms

Notification

Engine

Threshold

Engine

Anomaly and

Prediction

Engine

Events

Engine

Transform

EnginePersister

Message

Queue

Kafka

Monasca - Architecture (2/4)

Monasca API (REST)

Monasca

Client (CLI)

Notification

Engine

Threshold

Engine

Anomaly &

Prediction

Engine

Events

Engine

Transform

EnginePersister

Message

Queue

Kafka

Metrics

Events DB

Alarms

SystemMonasca

Agent

Horizon

Dashboard

Monasca

Agent

Monasca Agent: different subcomponents different metrics

statsd, nagios plugin, ceilometer

Monasca API: REST API post/get metrics and events

(JSON) query and get statistics and metrics, alarms,

alarm definitions, notificationsMonasca API (REST)

Message Queue “Kafka”: receives

metrics from API alarms from Threshold engine Publishes consumes events

RabbitMQ-> Kafka

Message

Queue

Kafka

POST

metrics

Publishes

metrics/events

Transform

Engine

Monasca - Architecture (3/4)

Monasca API (REST)

Monasca

Client (CLI)

Notification

Engine

Threshold

Engine

Anomaly &

Prediction

Engine

Events

Engine

Transform

EnginePersister

Message

Queue

Kafka

Metrics

Events DB

Alarms

SystemMonasca

Agent

Horizon

Dashboard

Monasca

Agent

Notification Engine: consumes alarms from kafka and notifies

(email, sms,.. )

Threshold engine: stream data processing (Apache Storm) computes threshold on metrics and publish

on kafka

Monasca API (REST)

Anomaly & Prediction engine: evaluates predictions and anomalies and

publish predicted metrics (or anomalies)

Message

Queue

Kafka

Events engine: manages events

Transform engine: transforms metrics into delta or time-based

derivative generates new metrics (Kafka) Notification

Engine

Threshold

Engine

Anomaly &

Prediction

Engine

Events

Engine

Monasca - Architecture (4/4)

Monasca API (REST)

Monasca

Client (CLI)

Notification

Engine

Threshold

Engine

Anomaly &

Prediction

Engine

Events

Engine

Transform

EnginePersister

Message

Queue

Kafka

Metrics

Events DB

Alarms

SystemMonasca

Agent

Horizon

Dashboard

Monasca

Agent

Persister: Consumes alarms and metrics from Kafka

and stores into nonRelational DB InfluxDB Vertica

Horizon Dashboard: Actions on alarms:

CRUD alarm definitions read/delete alarm history

Actions on notifications: CRUD notifications methods

Actions on metrics visualizations of metrics “Grafana”

Monasca API (REST)

Message

Queue

Kafka

Notification

Engine

Threshold

Engine

Anomaly &

Prediction

Engine

Events

EnginePersister

Horizon

Dashboard

Create alarms definitions

Query/Delete alarms

Create notifications method

Query Metrics

Monasca - Metrics flow (1/4)

Monasca

Agent

Metrics

Events DB

Alarms

Notification

Engine

Threshold

Engine

Transform

EnginePersister

Message

Queue

Kafka

POST metric

1.0 publish metric

1.1.0 consume metrics

1.1.1 store metric

1.2.0 consume metric

1.2.2 publish metric

1.2.1 aggregate metric

1.3.0 consume metric1.3.3 publish alarm

1.3.2 theshold metric

2.1.0 consume alarm

2.1.1 send notification

2.2.0 consume alarms

2.2.1 store alarm

1.0 API authenticates and validates the request and publishes the metric to Kafka

1.1 Persister consumes the metric from Kafka and stores in the Metrics Store

1.2 The Transform Engine consumes the metrics from kafka, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue

1.3 Threshold Engine consumes metrics from Kafka and evaluates alarms. If a state change occurs in an alarm, an alarm is published into kafka

2.1 The Notification Engine consumes "alarm" from kafka, evaluates whether they have a Notification Method associated with it, and notifies

2.2 Persister consumes the "alarm" from Kafka and stores it in the Alarm State History

Monasca - Events flow (1/4)

Monasca

Agent

Metrics

Events DB

Alarms

Event Engine

Transform

EnginePersister

Message

Queue

Kafka

POST event

1.0 publish event

1.1.0 consume raw event

1.1.1 store event

1.2.2 publish

tranformed event

1.2.1 transform raw event

into monasca event

1.3.0 consume

transformed event

2.0.0 fire event 1.0 API authenticates and validates the request and publishes the event to Kafka

1.1 Persister consumes the event from Kafka and stores in the Events Store

1.2 The Transform Engine consumes the event from kafka, performs transform and aggregation operations on events, and publishes events that it creates back to Message Queue

1.3 Event Engine consumes compliant events from Kafka. It converts the event into stream a “YAML”

2.0 Streams are processed by the Stream Engine Winchester. If needed it publishes Metric

Stack Distiller

StackTach

Winchester

1.3.0 add event

2.0.1 publish metric

1.2.0 consume raw event

Monasca - API

Keystone

Metrics

Transformations

Alarms

Notifications

Streams

EventsService Catalog provided by keystone

http://my_monasca

http://my_stacktach

/metrics//transform//alarm-definitions/ or /alarms//notification-methods/

/streams//events/

Monasca API StackTach API Your API * Anomaly API

Monasca client StackTach client Your client* Anomaly client

Monasca-Demo

"Back to the Future with Michael J. Fox as Marty McFly" by Source. Licensed under Fair use via Wikipedia - here

4.Conclusions

“E quindi uscimmo a riveder le stelle..”[Inf. XXXIV ]

OpenStack Bootcamp

The main topics covered are:• Overview on OpenStack and its

architecture,• OpenStack networking;• Swift;• Ceilometer and its architecture,• Heat Overview;• OpenStack deployment.

At the end of the Bootcamp each student will be able to:• Describe the architecture of an OpenStack

deployment;• Discuss the main functionalities of OpenStack;• Deploy, configure and use the Openstack

services;• Create and manage VMs and Virtual Networks;• Create and manage, suers, roles, and quotas;• Use the OpenStack CLI and Dashboard.

In partnership with Mirantis

For further information: http://openstack.create-net.org; openstack@create-net.org

Attilio Broglio

Software architect @ Create-net

abroglio@create-net.org

Tanks for your attention!!

Silvio Cretti

Deputy Area Head @ Create-net

scretti@create-net.org

Webinar on November 26th, 2015 - 15:00 CET