CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Using AI tools for IT-CS Spectrum-based

monitoringVéronique Lefébure

IT/CS-CE

February 2014

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Content

• SNOW tickets• Monitoring data storage

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Operator’s role today

Checks Spectrum screen:

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Operator

• Sees “Critical (red)” alarms• Follows SNOW KB procedure

– Possibly calls expert– And/or opens SNOW ticket: link to SNOW ticket form

– for Firstline or for Wigner support:» 2 SNOW Record Producer forms

– Copy and paste information

– Types INC ID (and comments) into Spectrum alarm info:

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Operator vs Netcom team

During Working Hours Outside Working Hours

Netcom Team • Critical (red) alarms• Major (orange) alarms

Operator • Critical (red) alarms after 10 minutes

Critical (red) alarms

• Netcom follows alarm procedure link (Sharepoint)

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Creation of SNOW tickets with GNI

• Avoids “copy-and-paste”• Useful both for Operator and Netcom team• Easier follow-up on alarms when there are a

lot of them• Correlation between alarms and SSB

interventions or incidents• GNI dashboard correlation between

Network alarms and other alarms

Need:• INC ID back from GNI (and not EVT ID)

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Data storage

• CS uses Spectrum for storage of – SNMP Events and alarms for last 45-120 days (limited by

Spectrum MySQL DB size)– Service Outages

• CS has home-made storage system for– Alarm long-term history– Part of statistics (in RRD files)– SYSLOG data

• CS provides info to SLS• CS lacks

– Storage for the rest of statistics– Correlation engine between SNMP and SYSLOG data (for

vendors with no syslog trap support)