nosql databases and monitoring - it monitoring working...

PES

CERN IT DepartmentCH-1211 Geneve 23

Switzerlandwww.cern.ch/it

CERNITDepartment

NoSQL Databases and MonitoringIT Monitoring Working Group

Jerome Belleman Christos MargiolasCERN – IT-PES

19 September 2011

http://www.cern.ch/it

http://www.cern.ch

PES



CERNITDepartment

2 – NoSQL Databasesand Monitoring

Outline

1 Monitoring Context

2 NoSQL Databases for Monitoring Purposes

3 Examples

4 Outlook


http://www.cern.ch

PES



CERNITDepartment

3 – MonitoringContext

Section 1

Monitoring Context


http://www.cern.ch

PES



CERNITDepartment


CERN Batch System Monitoring

Investigate more advanced features of the batch system

Reduce problem identification time

Lego-like, interchangeable set of building blocks

Batch System

Collection

Transport

Processing StorageAnalysis

Live Historical


http://www.cern.ch

PES



CERNITDepartment


What to Monitor?

Jobs (number, CPU usage, I/O, . . . )

Users

Hosts

Slots

Queues

. . .

Ñ Correlations & heavy analytics

Live representations

Historical representations

Ñ Browse metrics over time


http://www.cern.ch

PES



CERNITDepartment


How Much to Monitor?

ą 200 000 jobs/day

ą 3 700 nodes

ą 30 000 cores


http://www.cern.ch

PES



CERNITDepartment


What Do We Need?

2 Scalability

2 Data collection Ñ Many small writes, very often

2 Heavy analytics Ñ Massive reads, quite often

2 Horizontal & vertical aggregations

2 Time views

2 Flexible schemas

Ñ Let’s investigate the use of a NoSQL database


http://www.cern.ch

PES



CERNITDepartment

8 – NoSQL Databasesfor Monitoring

Purposes

Section 2

NoSQL Databases for Monitoring Purposes


http://www.cern.ch

PES



CERNITDepartment

9 – NoSQL Databasesfor Monitoring

Purposes

General Concepts

NoSQL ù Non Relational

A broader set of structured storage

Different consistency levels

Different classes of NoSQL databases


http://www.cern.ch

PES



CERNITDepartment

10 – NoSQLDatabases for

Monitoring Purposes

Different Classes of NoSQL Databases

Column-oriented databases (e.g. Google BigTable,Apache HBase, Apache Cassandra)

Key-value stores (e.g. Amazon Dynamo, ProjectVoldemort, MongoDB, Riak, Google BigTable,Apache Cassandra)

Document stores (e.g. Apache CouchDB, MongoDB)

We’ve been looking at Cassandra.


http://www.cern.ch

http://labs.google.com/papers/bigtable.html

http://hbase.apache.org

http://cassandra.apache.org

http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

http://project-voldemort.com

http://project-voldemort.com

http://www.mongodb.org

http://wiki.basho.com

http://labs.google.com/papers/bigtable.html

http://cassandra.apache.org

http://couchdb.apache.org

http://www.mongodb.org

PES



CERNITDepartment


Monitoring Purposes

Key-Value Stores such as Cassandra

k0 c0,0 c0,1 c0,2 c0,3 c0,4

k1 c1,0 c1,1 c1,2

k2 c2,0 c2,1 c2,2 c2,3 c2,4 c2,5

k3 c3,0 c3,1

k4 c4,0 c4,1 c4,2 c4,3

k5 c5,0 c5,1 c5,2


http://www.cern.ch

PES



CERNITDepartment


Monitoring Purposes

How Fast?

(Quite) Simple-minded

High throughput

Horizontal scalability

No need for object-relational mapping


http://www.cern.ch

PES



CERNITDepartment


Monitoring Purposes

What Do We Need?

2 Scalability

2 Data collection Ñ Many small writes, very often

2 Heavy analytics Ñ Massive reads, quite often

2 Horizontal & vertical aggregations

2 Time views

2 Flexible schemas



http://www.cern.ch

PES



CERNITDepartment


Monitoring Purposes

What Do We Need?

2� Scalability

2� Data collection Ñ Many small writes, very often

2� Heavy analytics Ñ Massive reads, quite often

2� Horizontal & vertical aggregations

2� Time views

2� Flexible schemas



http://www.cern.ch

PES



CERNITDepartment

14 – Examples

Section 3

Examples


http://www.cern.ch

PES



CERNITDepartment

15 – Examples

A Time Series Database: OpenTSDB

On top of Apache HBase

Horizontal aggregation Ñ Free time browsing

Vertical aggregation Ñ Drill down details


http://www.cern.ch

http://opentsdb.net

http://hbase.apache.org

PES



CERNITDepartment

16 – Examples

Job Performance Analysis: Cassandra

Batch

Cassandra

Batch

Cassandra

Batch

Cassandra

Batch

Cassandra

ActiveMQQueues

Consumer

ActiveMQQueues

Consumer


http://www.cern.ch

PES



CERNITDepartment

17 – Examples

Fairshare Monitoring: Cassandra

Reading from bhpart output

ą 30 000 groups/users every 5 minutes

malberti

ebye2

collazug

emicelot

SHARE

u_GEANT4

kmassri

andreev

michal

pimia

cerri

kresan

awojtasz

sreia

mcortesi

kulikov

gentile

gear049 gear048

vormc

sbianucc

gear043

gear042gear041

gear040

gear047gear046

gear045kunin

fantechi

jhakola

jcs

smele

mchizhov

msando

kuptsov

na49

palim

legou

jsmolik

unger

ams

atonazz

na48cdr

na48u010

bourquin

korolko

ccasella

torimoto

ams

sgiani

glamanna

wislicki

marusov

rondio

dangelop

na48wien

ecoucemo

abaas

pmatak

luckey

scprod

aelagin

wangqin

blochp

kperl

emonnier

na48plot

na48u005na48u004

na48u007

na48u006

na48u001

na48u003

na48u002

ldjambaz

markytanna48u008

gabri

tvrba

janot

ifisk

denisd

runna49

polenkev

berdugo

karpukhn

dgibin

lferreia

caragxv

malgeri

sgallori

lourenco

ecortina

kwosinsk

mdoucet

karev

emarinov

mthomann

zrelov

acecchet

dumitriu

u_THEORY

pfilips

fedine

dikolev

muijs

malahov

miliescu

norton

na48uk

silviac

fischerg

xma

karyotak

barr

mvaldata

dschmitz

jurajb

gchen

mbigliet

ajago

eperez

mveltri

mion

u_ILC

wotton

rbatley

gear010

gear011 gear012

na48kab

gear014gear015

gear016gear017

gear018

gear019

lkrouglo

ctimmer

schioppo

okounina

colino

goylopez

josa

akorteni

zilizi

massimop

hwalther

bigroots

radoslaw

tpalczew

ruggierg

dagnolo

cciortea

jhegeman

lomidze

clare

patel

olgortch

u_HARPD

aartamon

veenhof

doudarev

l3cos

pchung

kleinkne

marek

bertucci

chiarusi

peyaud

malgeril

safarzad

production

u_NA48CALI

ceccucci

voloshyn

harpshif

zhuanghl

cerminar

na48tech

u_NA61

gpalla

egamberi

dai

jacuh

don

wedlake

winhart

fergus

gianoli

zimanyi

malinin

wjb

hristov

gruenew

capell

robbins

palencia

l3dbsm

saogaki

simoing

pgorb

mvorobie

munday

islepnev

nabgrall

rstock

schach

desalvo

querten

crovelli

na48nut

passelev

newman

sergiant

amsf4

jabitski

bpanzer

takeuti

rvasquez

aboehm

golubeva

duchesne

ondrejch

acgroup

ljones

bagoulia

u_CMST3

alexg

l3prd

dflueras

segoni

stefanpvna48expo

bdulach

pohl

kkousour

gajewski

mclavier

vpoliako

stroebel

u_CMS

na48cali

mcampan

sbologne

yoo

takeut

begona

braccini

tully

mlenti

gear029

gear028

palladuh

na48mbx

formica

gear021

gear020

gear023

gear022

cenci

gear024

gear027

kirkby

cdbase

mbsi

u_DTEAM

sgevorgy

biasini

u_FLUKARP

gear050

mpivanti

lboldizs

rparamat

wreece

lconey

raggi

eddag

mgallas

mannellimmojzis

azzarel

na48u002 na48u003

na48u001

na48u006

na48u007na48u004

na48u005na48u008

na48u009

pjindal

bvallage

zalanna

piotr

bvelghe

kurshets

abizzeti

mzaytsev

fmonraba

crosenbl

shkar

merega49

ferella

abenelli

mdepreng

mcristin

dehmeltk

rplaneta

cecchi

coward

lydia

foka

u_TOTEM

ykohl

imbergam

bifani

asryan

vodib

azemoon

na48mc

u_AMS

mpasqual

piruleta

napolita

spadolsk

u_ITDC

mmozer

dyulenda

kadija

rolandc

rgaglion

zaranek

lenzip

villani

derrej

ascott

nefedov

gfaranta

sgiudici

fsikler

mhansrou

jmartinc

dsaborid

vdezso

u_vl

mellis

choehne

tariq

pmeckiff

riazants

vfriese

mfidecar

winston

vagori

choutko

tspadaro

ciaranfi

drizzi

mfiorini

brybicki

cstrabel

sozzi

cabibbo

aaduszki

wache

ronzhin

nikolicv

gearsgm

molokangmills

poluhina

acervera

gnuessl

mnguyen

larazi

pdupre

gear038

clemenci

murphy49

cblume

jgomezca

pdalpiaz

schwerin

andreasp

amoraux

idymniko

mvelasco

fbucci

eusebio

mmass

anzivino

harpmigr

ponta

kprok

ptgruber

ttrojek

carlino

mmitrov

renebrun

u_OPAL

perretg

holzner

vslepnev

maiera

rbramm

hbeck

benedet

u_vk

wborow

brekhovs

gear034

boiko

hwoehri

l3dbback

vassilim

desantoapacheva

berges

ipharp

u_NTOF

ddifilip

sorel

straessn

msolanog

amakarov

valente

mvandena

pernicka

pdoskaro

plyaskin

anniyev

mateevvp

abajovaq

dilella

vberardi

jsmolikp

ovazquez

na48mon

atlasccc

fmayet

rapuano

fjediny

ndavis49

unalserram

ltrnkova

pavelb

linch

gtof

hitahoch

pnaoumen

bagnaia

kokada

litovl

kscholbe

gonidec

kirillov

abis

u_AMSL3

gowdy

jxu

bonaiuto

na48sg04

na48sg05

moronembwarnaar

na48sg01

na48sg02

na48sg03

michiel

u_PARC

cai

jberdugo

azintche

na48cdr

riemanns

gomezu_HARP

nikonov

emelyano

mpierini

diracww

damianic

cmgtools

sandoval

rgulik

dagos

zucuh

diracoff

u_NA48

u_OPERAu_DELPHI

mbehler

l3cos

samyn

pikna

eiacopin

rubinp

fgreco

lvjv

lsfadmvp

lilee

ipopov

rcarosi

bartalin

sradomsk

pjanot

prusapet

belkin

pentia

gear044

na49soft

guglme

jpanman

yw266

u_L3COS

jmartina

gear039

gugiu

cheze

gear032

gear033gear030

gear031

gear036

jeitler

asemak

gear035

zfodor

trdtest

ddv

janas

na48calimasettil

cmorales

wiebusch

lenab

u_ge

mgouzevi

chimenti

sgaleo

cembra

felcini

u_L3

tomaszc

gostkine

lepew

barna

lazzeroc

goldin

jcatalap romano

booth49

others

nabgra49

u_ALEPH

acaraghe

tagger

costantiblochb

sbazylev

mkopal

calt

l3sgm

koreshevrschroet

apeters

u_ATLAS

mestviri

metzger

pmassaro

levpm

sschifan

u_NA48CDR

majam

jkroon

balev

acassese

mnich

gianluca

smaria

zhons

odyniec

cbern

afanasev

mikulec

veres

u_NA49

campore

rancoita

u_ALICE

phodgson

u_xv

bunyatov

mdelgado

rykalin

andreeva

yallkofe

na48grid

awetzler

mserra

meschi

acottara

adeva

frabetti

steuer

tanticic

swallow

yushchen

ambrosi

khowell

francesc

u_NA38NA50

gdebrecz

mariovor

turban

boothuh

bcabirol

saracino

pue

u_LHCB

makariev

bangeluc

manjitk

tnumao

pepe

kkaadze

schel3

muheim

nesterov

mufast

girrbach

harp_PROD

cavalari

stoynev

laszloa

diracgst

dinh

petrucci

cernyv

rauch

ivanch

popovb49

ferrif

lconey

panosk

u_GEAR

alanaro

u_SLDIV

fmarchet

burin

gennai

fonseca

martinjp

potreb

giordano

amssec

piandani

l3u001

hegner

prodvk

popovb

xbarr

gritsaj

temnikov

sissi

tsenov

kayis

geist

ggrazian

jrfry

bhallgre

blyth

gastaldi

sonia

bobbink

wadhwa

piccini

na48hac

sissi

venditti

ajokhov

melk

cfarnesejengelfr

oussenko

prodna49

rosier

wcreus

fdydak

sconstan

kolesnik

aantonel

psilva

na48u009

rbecker

martoiu

na48nntp

galaktio

vpalladi

gmasetti

rappazzo

perfetto

foc

roberta

gear025

madigo

l3mon

szuba

emanuele

dplasien

evcharp

wieneman

mryb

benitezj

vkruglov

franken

kirsanom

kobayash

aanania

l3u005l3u004

lbetev

u_uh

radu

l3u003

l3u002

jbook

u_C3

u_SFT

na48u010

danpop

bsunanda

jkrasnoh

meurer

mkoval

ikrol

vserdiou

dsoph

choutko

orestano

grappazz

gear013

celeghinsantin

behler

flionett

pastrone

pchrista

ddrijard

phy6phs

petuhov

schopper

slodkow

mutvic

abolshak

u_AMSP

ambrosin

u_NA45

cdelitzs

guaraldo

rozjkov

ylu

mchiba

rovere

sphicas

sstamm

damgov

drijard

roths

seyboth

jorgec yvette

mrizza

sbravar

kowal

goudzovs

lkrcalib

jasinjak

vtrubnik

wanke

aderoeck

nemenov

mdyulend

vetlitsk

cjingzalite

u_COMPASS

wdominik

gear009

gear008gear007

gear006gear005

gear004gear003

gear002gear001

marcello

bolotov

sdutta

portheau

afava

masaaki

buncic

lebedev

petrascu

xmeng

kekel

xyang

dgrandiaelin

wperyt

psoler2

biland

manjit

suenhou

yasar

peppep

gear037

prok

lecoultr

boconnel

plutaj

vite

ddesalva

hayato

mkubesov

ivanshin

rab

berschin

jfield

renfordt

enulescurezinko

vsemenov

brun

fmna49

mulders

aguskov

yfoka

yazkov

rolandi

jemtchou

u_ATLASCAT

u_NOMAD

editgas2 editgas1

shieh

kvi

bourilko

barrin

lucieg

diracdbkatchnov

dmitrifs

u_DIRAC

volnykh

candi

diracxfr

lgatigno

mboguszveszter

srobbins

ufo

choumilo

demina

evaboter

gear026

wahlppetev

ovg

ygouzevi

mnikitin

u_vp

cdbase1

marekg

ciocarln

beaudett

pochon

venelin

wenig

sbg

dinkelak

chmeurer

burgerchumakov

harpprod

cvetan

schacher

gmchen

alopeza ekulish

debeerjwotsch

calvetticogan

biino

tsusa

meunier

anton

grid_NA48

paganouh

okada

fechner

stefansk

komarov

molot

CMS_CERN01_YODA

edoardo

mduda

harpprod

mmoulson

stefanik

pzucchuh

romerov

tschuste

ltortoraburguet

bboimska

mposiada

stefanek

dde

widhalm

mihul

batgerel

nbuncic

fkaster

lamberto

ewk

demelyan


http://www.cern.ch

PES



CERNITDepartment

18 – Outlook

Section 4

Outlook


http://www.cern.ch

PES



CERNITDepartment

19 – Outlook

Outlook

Throughput through simplicity

Relaxed consistency acceptable for monitoring

Horizontal scalability

Flexibility

Many available open-source implementations

New schema design approach

Ñ NoSQL databases seem a good fit for monitoring


http://www.cern.ch

PES



CERNITDepartment

20 – Outlook

Thanks!

Questions?


http://www.cern.ch

nosql databases and monitoring - it monitoring working...

Documents