tiad 2016 : continuous integration mesured and controlled
TRANSCRIPT
![Page 1: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/1.jpg)
T I A D
Copyright©2016SplunkInc.
ContinuousIntegrationmeasured& controlledStéphaneLapiePresalesEngineer,Splunk Inc.
listentoyourdata®
![Page 2: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/2.jpg)
T I A D
Presenters
2
StéphaneLapiePresalesEngineer– 2YearsatSplunkBasedinParis,FranceSpecializedinAppDelivery,ITOpsandCloudWentfromDev,toOpstoPresales(formercolleagueofAdegbenga atBNPParibas)
Adegbenga AmusaHPCDeveloper– BNPParibas,RiskSystemsBasedinLondon,UnitedKingdomWorkedonGridComputingforthepast10yearsHadafairshareofOpsinmycareer
![Page 3: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/3.jpg)
T I A D
OnceUponaTimeinaFinancialInstitution…
![Page 4: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/4.jpg)
T I A D
Service
AGroupofPeoplewasWorkingonEnhancingSome…
![Page 5: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/5.jpg)
T I A D
• Requirements• ServiceDefinition
• Implementation• Validation&Testing
• ApplicationMonitoring• IncidentManagement
Business
Development
Operations Service
Customers
AndLikeManyServicesThereWere…
![Page 6: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/6.jpg)
T I A D
Service
ThisGroupofPeopleWasBoundtoTransformtheService
![Page 7: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/7.jpg)
T I A D
RiskCalculation
BusinessAnalysts
Java,C++,Cuda
Developers
ApplicationSupport
RiskSystems
StructurersRiskAnalystsQuantAnalystsDataAnalysts
![Page 8: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/8.jpg)
T I A D
RiskCalculationCounterpartyRisk
A B
• ManySimulations• ManyParameters• ManyRegulations• …maybeslowtocalculate
![Page 9: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/9.jpg)
T I A D
RiskCalculation
• Fresh information
• Accurateinformation
• CompliancewithMarketRegulations
• Fullyparametrizedon-demand simulations
• Longtermsimulations
• Quickresults(understandinstant)
• Easeofuse!!!
• Integratedwiththemanyothersystems
HighCustomerExpectationsStructurersRiskAnalystsQuantAnalystsDataAnalysts
![Page 10: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/10.jpg)
T I A D
ANewProjectWasBorn
10
Goal:RebuildtheCalculationEnginetousehighlyparallelcomputingonGraphicalProcessingUnits
• MuchQuickerCalculations
• MuchMoreSimulationsCapabilities
• EasierModelsImplementation
• ReduceHardwareandMiddlewarecost
ProjectCodename:
HPCEHighPerformanceComputeEngine
![Page 11: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/11.jpg)
T I A D
ANewProjectWasBorn
11
GPU/SIMTCPU/SMT
Complexanddifferenttasks
Notsocomplexscalabletasks
fewvery-smartcoresmanycheapercores
CPUvGPUinanutshell
![Page 12: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/12.jpg)
T I A D12
CPUvGPUinanutshell
MonteCarloMethods
SimulationtocalculateValueatRisk
ParallelCalculatio
ns
ANewProjectWasBorn
![Page 13: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/13.jpg)
T I A D
Project’sTimeline
13
FromZerotoHero…
Q2 Q2
Dev Production
2013 2014+
1. FullyIntegrated
2. FullyFunctional
3. IdenticalResultstoLegacy
![Page 14: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/14.jpg)
T I A D
Project’s Timeline
14
FromZerotoHero…
Q2 Q2
Dev Production
2013 2014+
DevOps…totherescue!
FastForward◉
![Page 15: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/15.jpg)
T I A D
Project’sTimeline
15
FromZerotoHero…
Q2 Q2
Dev Production
2013 2014+
FastForward◉
![Page 16: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/16.jpg)
T I A D
Development AppSupport
16
DEV UAT STG PRDTEST TEST TEST
Automation
End-to-EndVisibility
VersionXYZ
VersionXYZ
Version1.0Version2.0
Version3.0.1
Version3.0.1.1
Version4.2.0.1
Business
TEST TEST TESTTEST TEST TESTTEST TEST TEST
DeliveryPipelineOverview
![Page 17: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/17.jpg)
T I A D
Step#1
17
OpenAccesstoEnvironment’sData
![Page 18: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/18.jpg)
T I A D
WhatisDelivered
18
High Performance Compute Engine
Non-persistent Data
PersistentData
Compute FarmREST
Services
GPUServers
CPUServers
ApacheCassandra
+SybaseIQ
+(FileSystem)
OracleCoherence
+DataServices
IBMPlatformSymphony
+CalculationServices
ApacheTomcat
![Page 19: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/19.jpg)
T I A D
Step#2
19
Createindicatorsforeachsoftwarecomponents,oneafteranother…
![Page 20: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/20.jpg)
T I A D
ConsistentandReusableAnalytics
20
Business AppSupport Development
HPCE
Non-persistent Data
PersistentData
Compute Farm
RESTServices
DEV/UATSTAGINGPRODUCTION
![Page 21: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/21.jpg)
T I A D
Step#3
21
Makeitrepeatabletoimproveprogressively!
![Page 22: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/22.jpg)
T I A D
IntegrationwithinRiskSystems
22
Non-persistent Data
PersistentData
Compute Farm
HPCERESTServices
RiskData Bus
Upstream Systems
Downstream Systems
RiskNavigator
RiskData
Warehouse
DataIn/Out
SimulationsRequests
ResultsBrowsing
![Page 23: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/23.jpg)
T I A D
IntegrationwithinRiskSystems
23
Non-persistent Data
PersistentData
Compute Farm
HPCERESTServices
RiskData Bus
Upstream Systems
Downstream Systems
RiskNavigator
RiskData
Warehouse
DataIn/Out
SimulationsRequests
ResultsBrowsing
End-to-EndVisibility
Business
AppSupport
Development
![Page 24: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/24.jpg)
T I A D
Step#4
24
Increasethescopetogetclosertothe“fullpicture”
![Page 25: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/25.jpg)
T I A D
DataSourcesOverview
VersionControl
QA/TestingTools
CM/DeploymentCI/BuildAutomation
23
Applications&Systemslogs
CUSTOMLOGGING
Business AppSupport Development
![Page 26: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/26.jpg)
T I A D26
Visualize ShareAnalyzeExplore
Schema-Freedatacollection& storage
CodeRepository
QA/TestingTools
CM/DeploymentCI/BuildAutomation
ApplicationsMonitoring
Project&IssueTracking
SystemsMonitoring
DistributedStorage&Processing
ScalestoTBs/day
=DatageneratedbyITSystems
MachineData
DataSourcesOverview
![Page 27: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/27.jpg)
T I A D27
![Page 28: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/28.jpg)
T I A D28
DashboardsWalkthrough
![Page 29: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/29.jpg)
T I A D29
Compute Farm
CPUServers
GPUServers
Non-persistent Data
PersistentData
High Performance Compute Engine
RESTServices
ServiceRequestsDevelopment
Requests/Responses
![Page 30: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/30.jpg)
T I A D30
Development
![Page 31: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/31.jpg)
T I A D31
Development
![Page 32: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/32.jpg)
T I A D32
Compute Farm
CPUServers
GPUServers
High Performance Compute Engine
RESTServices
Non-persistent Data
PersistentData
DataIn andOutBusiness
AppSupport
Development
RiskData Bus
![Page 33: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/33.jpg)
T I A D34
Business
AppSupport
![Page 34: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/34.jpg)
T I A D35
Business
AppSupport
![Page 35: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/35.jpg)
T I A D35
AppSupport
Development
![Page 36: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/36.jpg)
T I A D36
AppSupport
Development
![Page 37: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/37.jpg)
T I A D37
Business
AppSupport
Development
![Page 38: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/38.jpg)
T I A D39
Business
AppSupport
Development
![Page 39: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/39.jpg)
T I A D39
PersistentData
Non-persistent Data
High Performance Compute Engine
Compute Farm
CPUServers
GPUServers
RESTServices
CalculationsAnalysis Business
AppSupport
Development
![Page 40: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/40.jpg)
T I A D40
AppSupport
Development
![Page 41: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/41.jpg)
T I A D41
Business
AppSupport
Development
![Page 42: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/42.jpg)
T I A D42
Development
![Page 43: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/43.jpg)
T I A D43
AppSupport
Development
![Page 44: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/44.jpg)
T I A D44
AppSupport
Development
![Page 45: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/45.jpg)
T I A D45
Business
Development
![Page 46: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/46.jpg)
T I A D46
Business
Development
![Page 47: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/47.jpg)
T I A D47
High Performance Compute Engine
PersistentData
Non-persistent Data
Compute Farm
CPUServers
GPUServers
RESTServices
InfrastructureAppSupport
Development
![Page 48: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/48.jpg)
T I A D48
AppSupport
Development
![Page 49: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/49.jpg)
T I A D49
AppSupport
Development
![Page 50: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/50.jpg)
T I A D50
AppSupport
Development
![Page 51: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/51.jpg)
T I A D51
AppSupport
Development
![Page 52: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/52.jpg)
T I A D52
AppSupport
Development
![Page 53: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/53.jpg)
T I A D53
AppSupport
Development
![Page 54: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/54.jpg)
T I A D54
AppSupport
Development
![Page 55: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/55.jpg)
T I A D55
Wrap-up
![Page 56: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/56.jpg)
T I A D
KeyTakeAway
56
1. Don’twithholdinformation,openaccesstoeveryone(%sensitiveness).Shareyourthoughtsandanalysiswiththeotherteams.
2. TaketimetodefineIndicatorsthatmatters.RepeattheoperationwithDev,OpsandBizpeople.
3. Mostanalysisshouldbeconsistentandrepeatable.EventuallyfromtheDevcomputer’tilProductionplatform(containerscanhelpnowadays).
4. Don’trushandtrydoeverythingatonce.You’llgettherequicklyenoughifyoustreamlineyourefforts.
![Page 57: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/57.jpg)
T I A D
LessonsLearned
57
Ø Todaywemanagedtogetan“Holistic”viewofoursystemsandwecan’timagineworkingwithoutasolutionlikeSplunk.WedidallthisveryquicklywithSplunk andinmuchmoredepththanpreviouslydevelopingcustomtools.
Ø Inthepastmostlogswereconsidered“garbage”butmadeaccessibletheyrevealedmanyUserBehaviorswedidn’texpect.Itisoftremendousvaluetounderstandwhatcustomersdo,whichfeaturestheyuseandhow.Thiskindofanalysiscantriggerchangesinthesolution.WestartedtoputLoggingBestPracticesinplacetomakesurethe“garbage”canbe“recycled”;)
Ø EverybodywithinRiskSystemsisusingSplunk forad-hocsearchesandsharingsearchlinksbutveryfewpeopleshareDashboardsandReportstoday.Goingforwardwe’dliketoenablemorepeopleandmakesuretheycancollaborateonimprovingourinternalSplunk Apps.
![Page 58: TIAD 2016 : Continuous Integration mesured and controlled](https://reader031.vdocuments.net/reader031/viewer/2022022001/5899d95a1a28ab4a0b8b611b/html5/thumbnails/58.jpg)
T I A D
Merci!