amadeus - itdks.su.bcebos.com
Post on 19-Jun-2022
7 Views
Preview:
TRANSCRIPT
RESTRICTED
Confidential
©AmadeusITGroupanditsaffiliatesandsubsidiaries
RESTRICTED
Confidential
AmadeusBuildingtheMongoDBInfrastructurefortheAmadeusInstantSearchApplication
ArkadiuszBorucki
RESTRICTED
Confidential
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_ Wearetheleading technologycompanydedicatedtotheglobal travelindustry
_ Wearepresentin190+countriesandemploy15,000+peopleworldwide
_ Oursolutionsenrichtravelforbillions ofpeople everyyear
_ Weworktogetherwithourcustomers,partnersandotherplayersintheindustrytoimprove
business performanceandshapethefutureoftravel
630+milliontotalbookings processedin2017usingtheAmadeusdistributionplatform
1.6billionpassengersboardedin2017withAmadeusandNavitaire solutions
1oftheworld’stop15softwarecompaniesForbes2017globalrankings
Amadeusinafewwords
2
6th consecutiveyearincludedintheDJSIRecognisedasworldleaderintheSoftware&ServicesindustrysectorintheDowJonesSustainabilityIndexin2017
RESTRICTED
Confidential
Providinglocalexpertise,allovertheworld
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Weworkin
190+markets
3
RESTRICTED
Confidential
Ourcustomers
©AmadeusITGroupanditsaffiliatesandsubsidiaries
touroperatorscarrentalcompanies
cruise andferrylines
railoperators insuranceprovider groups
airportoperators
groundhandlers
43
hotelchains
300+
travelagencies andcorporations, online
andworldwide
90 233 16
110 100
50
airlines
709
4
{„Xi’an2018”:„MongoDBatAmadeus”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
5
BuildingtheMongoDB InfrastructurefortheAmadeusInstantSearch
Application
RESTRICTED
Confidential
RESTRICTED
Confidential
{„Amadeus”: „MongoDB farmoverview”}
4OpsManagersversions:- 3.6.*.
MongoDB Clusterssplit between 4OpsManagers.
„Master”OpsManager.
100 MongoDB clusters:size(100GB– 107TB):
- 574 nodes,3000 mongod.
Micro sharding(even120 shardspercluster).
Automation: JenkinsCI/CD,Ansible, RESTAPI,Python scripts.
Security:SSL/TLS,Kerberos,LDAP, CA,Auditing, Encryption atRest,QRadar.
,6
©AmadeusITGroupanditsaffiliatesandsubsidiaries
7
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”:„MongoDBArchitecture2018”}96MongoDBclusterssplitted betweenOM
{„Amadeus”: „MongoDBusecases”}
• Instantsearching - flightrecommendations data:
3huge MongoDB clusters75 TB,78 TB,127TB,
• MongoDB forreal-timeanalytics – 124 shards(micro-sharding),
agregation framework,
• Apache SparkonMongoDB – fraud detection,
runmachine learningonMongoDB,
• MongoDB Amadeus Payments - payment flows,
• Daschboards: scheduling tools,cheange viewers,applications errorviewers,andmore.
8
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Instance Search: Usecasedescription
Leaving fromNewYork,where canIgowith 1500$ from9th to16th of July?
NCE1186$
SAO1033$
LIS890$LIS
890$
SAO1033$
Usecasedescription
Leaving fromNewYork,where canIgowith 1500$ from9th to16th of July?
NCE1186$
SAO1033$
LIS890$LIS
890$
SAO1033$
JFKNCE 07/0907/16
Travel.com
Search1adulteconomy
Viewdeal
1186$ 5:30pJFK->2:25pNCE14h55m1stop(DUB)4:25pNCE->1:20pJFK20h40m1stop(DUB)
Viewdeal
1501$ 6:25pJFK->11:25pNCE11h00m1stop(ORY)11:30aNCE->2:24pJFK8h54mnonstop
Viewdeal
1892$ 5:55pJFK->12:35pNCE12h40m1stop(MUC)7:00aNCE->1:35pJFK12h35m1stop(FRA)
Deepdiveintothearchitecture
©2016Am
adeusS.A.S.
Cache
Computation
LO N- M AD-12354- I B- AF-123- 1200-1354LO N- MAD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354
LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 123 - 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-
12354- I B- A1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- IAF- 123- 1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354
12354- I
LO N- M AD-12354- I B- AF-123- 1200-1354LO N- MAD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354
LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 123 - 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-
12354- I B- A1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- IAF- 123- 1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354
12354- I
LO N- M AD-12354- I B- AF-123- 1200-1354LO N- MAD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354
LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 12 3- 1200- 1354LO N- M AD-12354- I B- AF- 123 - 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354
LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-
12354- I B- A1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- I B- AF-123- 1200- 1354LO N- M AD-12354- IAF- 123- 1200-1354LO N- M AD-12354- I B- AF-123- 1200- 1354
12354- I
L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4 L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4
L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -
1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -
1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -
1 2 3 5 4 -IB -A F -1 2 3 2 0 0 -1 35 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -
1 2 3 5 4 -IB -A 1 2 0 0 -1 3 54L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 4L O N -M A D -1 2 3 5 4 -IA F -1 2 3 -1 2 0 0-1 3 5 4L O N -M A D -1 2 3 5 4 -IB -A F -1 2 3 -1 2 0 0 -1 3 5 41 2 3 5 4 -I
Indexing
Web Service
{Amadeus :SSPInstantSearch }Searchbillionsofrecommendations inmilliseconds!
Speed! - searchtimeisinaverage2milliseconds!
SSP- 3MongoDB clusters– 200TB ofdata.
always online!
12
©AmadeusITGroupanditsaffiliatesandsubsidiaries
In-houseNoSQLStore
©2016Am
adeusS.A.S.
In-house technology
• Functionaldatapartitioning
• Nonblocking updates,eventual consistency
• Indexing formulti-dimensional rangequeries
• In-memory and/ordisk
Scalability limit• Usecasewithmore than1000
recommendations
• Atomic dataupdate forhigh cachequalityMovetoStandardNoSQL technology
{„Amadeus”: „theBIGGEST clusterinEurope!”}
14
©AmadeusITGroupanditsaffiliatesandsubsidiaries
15
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „Automation”}
• MongoDB asaService.
• Infrastructure asCode- GIT.
• CI/CD withJenkins- orchestratethedeployment processes.
• OpsManager RESTAPI.
• Ansiblemodule forOpsManager /MongoDB.
• Ansibleplaybooks /rolesforMongoDB.
• Frameworkforonline database upgrade.
• Frameworkforonline rollingkernelupgrade.
RESTRICTED
Confidential
RESTRICTED
Confidential
{„Amadeus”: „Whatshouldbeautomated?”}
• MongoDB cluster deployment,
• MongoDB /OpsManager upgrades,
• MongoDB clusterscaling,
• ClusterOSpatching (security,kernel),
• Clusterresilience,
• monitoring, alerting,
• securitysettings,
• backup&restore,
16
RESTRICTED
Confidential
RESTRICTED
Confidential
{„MongoDB :NoDowntime”}
17
RESTRICTED
Confidential
{„OpsManager”: „RESTAPI”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
18
OpsManager followsRESTarchitecturalstyleandprovidesinternalresourceswhichenableprogrammaticaccesstoOpsManagerfeatures.
CallthisHTTPSAPI with instructionsfortheMongoDB cluster!
WEB UI
RESTRICTED
Confidential
19
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „OpsManagerclass”}
.
RESTRICTED
Confidential
20
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „ open-source”}
https://github.com/AmadeusITGroup
• Ansiblerole,• Opsmanager class
RESTRICTED
Confidential
21
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „Jenkins”}
Jenkins isanopen-source continuous integration (CI)tool thathelpsorchestrate thedevelopment processes(build, test,anddeployment) withautomation.
JenkinsisoneoftheleadingtoolsthatwouldhelpaDevOps teamorchestrate its
processes.
RESTRICTED
Confidential
22
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „JenkinsPipeline”}
Acontinuous delivery pipeline isanautomated expressionofyourprocess forgettingsoftwarefrom versioncontrol rightthrough toyourdeployment.
SettingupaPipeline projectmeanswritingascriptthatwillsequentially applysomestepsoftheprocesswewanttoaccomplish.
RESTRICTED
Confidential
23
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „JenkinsPipeline”}
RESTRICTED
Confidential
24
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Xi’an”:„MongoDBatAmadeus”}
Livedemos:
• JenkinspipelineforOpsManagerupgrade,
• JenkinspipelineforMongoDB upgrade,
• Opsmancombo - frameworkforMongoDBdatabasemaintenance
RESTRICTED
Confidential
25
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „projectsweworkingon:”}
• 200TBMongoDBcluster!,
• MongoDBonOpenShift(Microservices,OpsManageronDocker),
• MoreautomationandmoreOpenSource!
staytuned!!
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Youcanfollowuson:
AmadeusITgroupamadeus.comamadeus.com/blog
Thankyou!
top related