helping septa with the pope’s visit to philadelphia | aws public sector summit 2016
TRANSCRIPT
![Page 1: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mike Kuentz, Senior Solutions Architect, AWSMike Zaleski, Director Emerging and Specialty Technology, SEPTA
Tim Raybould, CEO Ticketleap
June 21, 2016
Helping SEPTA with the Pope’s Visit to Philadelphia
![Page 2: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/2.jpg)
Agenda
• Review of events leading up to Papal visit• Not all technical problems are best solved with technical
solutions• Details of AWS Infrastructure Event Management (IEM)• How leveraging partners can help customers• The importance of load testing
![Page 3: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/3.jpg)
Background Pope Francis plans visit to Philadelphia Friday,
September 25, 2015 Estimates put over 1 million people coming to the
city for his visit Strict security perimeter was in place in downtown
Philadelphia To manage the traffic SEPTA had to limit train tickets to only 350,000
![Page 4: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/4.jpg)
Pilgrims
![Page 5: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/5.jpg)
Transit map vs. event map
![Page 6: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/6.jpg)
Ticket sales website requirements
• Sell tickets on first come, first serve basis, no lottery• No “take a number” queuing systems allowed• No third-party e-commerce (eBay, PayPal, etc.) • Complex rules regarding number of tickets allowed for
various stations• Requirements were based on the premise that tickets
will sell over the course of a week—not in under an hour
![Page 7: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/7.jpg)
Ticket sales website 1.0
![Page 8: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/8.jpg)
Reality by the numbers• Considerable media frenzy leading up to site opening at 9 a.m.• 54,000 visits in the first minute• 900,000 visits in the first 10 minutes• ~1,700 transactions per second• By 11 a.m., ticket sales were suspended, because the architecture
could not meet the load
![Page 9: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/9.jpg)
The Internet responds
![Page 10: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/10.jpg)
Review of first architecture
• Areas of concern• Redis servers running on t2.small• Running out of MySQL connections
• As number of connections increases, memory increases as well• No read replicas• Drupal writing log info into the same DB• Tightly coupled design that had to send an email through
Amazon SES before transaction completed• Amazon SES limit had not been raised
• Load testing didn’t match demand
![Page 11: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/11.jpg)
Infr
astr
uctu
re C
ost
Time
Periodic LargeCapital Expenditures
OpportunityCost
PredictedDemandTraditionalData Center ModelActualDemand
CloudComputing
Shortage: Unable to serve
customers
Scale on demand
![Page 12: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/12.jpg)
How to scale?
• Scaled MySQL vertically and horizontally• Scaled Redis vertically and horizontally• Removed bottlenecks in logic• Removed bottlenecks in code
![Page 13: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/13.jpg)
Load testing
900 transactions per second
(We saw 1,700 transactions per second!)
![Page 14: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/14.jpg)
Requirements are forced to change Move to a lottery based system
![Page 15: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/15.jpg)
Lottery based system
Researched third parties (Ticketmaster, Eventbrite, etc.)— they all refused given the original requirements
Engage AWS IEM and Ticketleap Open access for 24 hours De-duplicate entries and fraud checks Lottery prevents/limits scalpers
Get rid of all the servers!
![Page 16: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/16.jpg)
![Page 17: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/17.jpg)
• AWS customer
• Lightweight online ticketing platform used by thousands of event organizers across the US and Canada
• Case study available• https://aws.amazon.com/solutions/case-studies/ticketleap/
![Page 18: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/18.jpg)
Review of second architecture
Amazon SQS Amazon SESAmazon S3
AWS Lambda
Amazon CloudFront
![Page 19: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/19.jpg)
S3 statistics
Amazon S3
![Page 20: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/20.jpg)
Request rate and performance considerations
• If your workload in an Amazon S3 bucket routinely exceeds 100 PUT/LIST/DELETE requests per second
• Talk to us!• Avoid sequential patterns in key names
• examplebucket/2013-26-05-15-00-00/cust1234234/photo1.jpg• examplebucket/2013-26-05-15-00-00/cust3857422/photo2.jpg• ...• examplebucket/2013-26-05-15-00-01/cust1248473/photo4.jpg• examplebucket/2013-26-05-15-00-01/cust1248473/photo5.jpg
![Page 21: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/21.jpg)
Review of second architecture
Amazon SQS Amazon SESAmazon S3
AWS Lambda
Amazon CloudFront
![Page 22: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/22.jpg)
Results
• Load testing• c4.8xlarge instance and The Grinder• POST 200,000 entries in 85 seconds (~2300 TPS)
![Page 23: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/23.jpg)
AWS Infrastructure Event Management
• Initiation• Planning and execution• Review and closure
• https://aws.amazon.com/premiumsupport/iem/
![Page 24: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/24.jpg)
IEM success stories
![Page 25: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/25.jpg)
![Page 26: Helping SEPTA with the Pope’s Visit to Philadelphia | AWS Public Sector Summit 2016](https://reader031.vdocuments.net/reader031/viewer/2022022202/5881dd321a28ab331a8b7c29/html5/thumbnails/26.jpg)
Thank you!