CentredeCalculdel'InstitutNationaldePhysiqueNucleaireetdePhysiquedesParticules

ELASTICSEARCHUPDATE

1

ESisawesome

LOGSANDMETRICSweneedabackendforstoringandqueryingESisawesomeforlogsESisawesomeformetricsESisawesomeforeverything

ES@CC 2017-02-14 CCIN2P3 2

ESisinsecure

SECURITYFIRSTeverybodywantstoseetheirlogseverybodywantstoseetheirmetricsfreeESisinsecureX-pack(fkaShield)isexpensive

ES@CC 2017-02-14 CCIN2P3 3

Architecture

Elasticsearch

Riemann

syslog-ng

Kibana

Clientsriemann-

dash

aggregationconsolidation

parsingcorrelation

RESTAPI

websocketAPI

Grafana

metrics(collectd)

logs(syslog)

ES@CC 2017-02-14 CCIN2P3 4 . 1

Scale

LOGS

METRICS

Avg:1k/sPeak:20k/sRetention:1year

Avg:15k/sRetention:5years(aggregated)

ES@CC 2017-02-14 CCIN2P3 4 . 2

Securityrequirements(1)

TRANSPORTend-to-endencryptionESdatanodes

(logs)(metrics)

syslog_ngriemann

ES@CC 2017-02-14 CCIN2P3 5

https://syslog-ng.com/http://riemann.io/

Securityrequirements(2)

AUTHENTICATIONKerberos/ (API/CLI)ClientCertificate(API/CLI)WebSSO(Browser)

GSSAPISPNEGO

CAS

ES@CC 2017-02-14 CCIN2P3 6

https://en.wikipedia.org/wiki/SPNEGOhttps://en.wikipedia.org/wiki/Central_Authentication_Service

Securityrequirements(3)

AUTHORIZATION2016:flatfile(managedbypuppet)2017:LDAP(!)User/Team

ES@CC 2017-02-14 CCIN2P3 7

History(1)

PASTWORKWITHHELPFROMKEKSearchGuardv1deployedonESv1.7KibanapatchfordynamicindexNodeJSproxy(ldapfacsimile)apacheCAS&KRB5proxy

ES@CC 2017-02-14 CCIN2P3 8

History(2)

PASTCONCERNSnotusingSearchGuard-SSLfloragunn(SearchGuard)commercial?elastic.coproductsmovingtoofast

minimizeworkforelasticstackversionN+1

ES@CC 2017-02-14 CCIN2P3 9

What'snew?

SEARCHGUARDfloragunnisawesome

developmentcatchesuponESfasterthanlightning

freesearchguardlicensegreatsupport

ES@CC 2017-02-14 CCIN2P3 10 . 1

file:///home/fwernli/projects/coloss/FJPPL/20170215/media/CCIN2P3_INV-000153.pdfhttps://groups.google.com/forum/#!forum/search-guard

What'snew?

SYSLOG-NGsyslog-ngESsupport

destination(Balabit)

client_mode(searchguard)(CCIN2P3 )

client_mode(https)(CCIN2P3 )twootherPRs(bugfixes)tobemerged( , )

elastic-v2

1 2

ES@CC 2017-02-14 CCIN2P3 10 . 2

https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-elasticsearch2.htmlhttps://github.com/balabit/syslog-ng/pull/1223https://github.com/balabit/syslog-ng/pull/1321https://github.com/balabit/syslog-ng/pull/1319https://github.com/balabit/syslog-ng/pull/1342

What'snew?

ES@CC 2017-02-14 CCIN2P3 10 . 3

Challenges

LESSONSLEARNEDSearchGuardconfigistrickybutworthitNoelaboratemappings:they'llbedeprecatedsoonishUseRESTinfavorofTransportMonitoryourcluster(heapusage,indexingrate,)Usebare-metalnodesonlyfordataUsetieredcluster(shardallocationawareness)Usesanerflush_interval(120s)Usestore:falsewheneverpossible(metrics)Throttlesources!

ES@CC 2017-02-14 CCIN2P3 11 . 1

Nominalactivity

ES@CC 2017-02-14 CCIN2P3 11 . 2

Killerquery

ES@CC 2017-02-14 CCIN2P3 11 . 3

Addnewnode

ES@CC 2017-02-14 CCIN2P3 11 . 4

Inprogress

TODOKibanadynamicindexupstream

needslargeraudiencethanHTTPheader($user)+queryparam(override)

migrateexistingdatafromESv1.7tov5.x1. splitcluster2. reindex3. reunitecluster

drawingattention/convincingkibana_own_home

ES@CC 2017-02-14 CCIN2P3 12

https://discuss.elastic.co/t/dynamic-kibana-index-name/73862/6https://github.com/wtakase/kibana-own-home

Reference

REFERENCESearchGuardfloragunnsyslog-ngriemannsamplerrSecuringyourESKstackforfreeusingSearchGuard

ES@CC 2017-02-14 CCIN2P3 13

https://github.com/floragunncom/search-guardhttps://floragunn.com/https://syslog-ng.org/http://riemann.io/https://github.com/ccin2p3/samplerrhttps://www.balabit.com/blog/securing-your-esk-stack-for-free-using-search-guard/

Metrics

WHYUSEESFORMETRICS?wealreadyhaveexperiencewithESit'sveryflexible(custommetadata)onlineaggregationtoElasticsearch:samplerr

ES@CC 2017-02-14 CCIN2P3 14 . 1

https://github.com/ccin2p3/samplerr

Questions?

ES@CC 2017-02-14 CCIN2P3 15