pyconcz'16 - dos youtself a.k.a. load testing
TRANSCRIPT
PyConCZ’16
DOS YOURSELFa.k.a. Load Testing
Dariusz Aniszewski
! DariuszAniszewski
" @aniszewski_eu
PyConCZ’16
➤ z polštiny
➤ 9 to 5
➤ Senior Software Engineer @ Polidea
➤ Python user since 2010
➤ Load testing aware since 2011/2012
➤ After hours
➤ Home-brewer
➤ IoT enthusiast
2
$ WHOAMI
PyConCZ’16
MOTIVATION
PyConCZ’16
➤ Introduction
➤ Example performance problems
➤ Tools
4
AGENDA
PyConCZ’16
1. You must not perform load testing on server you are not authorised to test.
2. You should not perform load testing on live, production server, even yours.
5
DISCLAIMER
PyConCZ’16
DISCLAIMER
6
http://devopsreactions.tumblr.com/post/133458045982/load-testing
PyConCZ’16
INTRO
PyConCZ’16 8
https://commons.wikimedia.org/wiki/File:Umgeni_River_Bridge_Load_Test.jpg
PyConCZ’16
➤ Load testing is the process of putting demand on a software system or computing device and measuring its response.
9
WIKI SAYS:
PyConCZ’16
➤ Load testing is performed to determine a system's behavior under both normal and anticipated peak load conditions.
10
WIKI SAYS:
PyConCZ’16
➤ Make your users happy
➤ Don’t loose money
➤ Ensure system mets non-functional requirements
➤ usable under expected load
➤ usable under N-times larger than expected load
➤ Ensure your hardware is working efficiently
➤ Ensure your architecture is working efficiently
➤ Determine how much traffic you can handle on single node
11
REAL REASONS
PyConCZ’16
WHOA, WAIT…
PyConCZ’16
➤ Unit Tests are Awesome
➤ Integration Tests are Awesome too!
➤ But still not enough…
➤ Unit tests are very isolated
➤ Both of them use minimal data
13
AREN’T UNIT TESTS JUST ENOUGH?
PyConCZ’16
PERFORMANCE ISSUES
“
PyConCZ’16
There is absolutely no way that all of them will hit the same API exactly at the same time.
- Dariusz Aniszewski, 201115
PyConCZ’16
➤ Project for weekly magazine
➤ iPad app & on-premise backend
➤ ~4000 active iPads every week
➤ Everything was running smoothly
16
BACKGROUND
PyConCZ’16
https://www.flickr.com/photos/methodshop/5808144764
17
PyConCZ’16
➤ Virtual bookshelf
➤ Introduced in iOS 5
➤ Workflow:
➤ Silent push that there is new publication available
➤ Application downloads new publication in background
➤ Application displays nice badge on its icon
➤ User opens app and new publication is ready
18
NEWSSTAND
“
PyConCZ’16
There is absolutely no way that all of them will hit the same API exactly at the same time.
- Dariusz Aniszewski, 201119
!! WRONG !!
PyConCZ’16
➤ Every device connected to the Internet hit our API at once
➤ Hundreds of parallel downloads of ~90MB packages
➤ No crash ;-)
➤ Download speed was terrible, usability was poor
➤ On-premise network infrastructure was bottleneck
➤ Moved to S3 week later
20
NEWSSTAND
PyConCZ’16
BAD MODEL DESIGN
PyConCZ’16
➤ Gather lots and lots of stats from mobile app
➤ Store them locally for short period of time
➤ Export them to BigTable
➤ Make it readable via Django
22
OBJECTIVES
PyConCZ’16
SESSION DATA
23
class SessionData(models.Model):
player = models.ForeignKey(PlayerStats)
# 1 - 5 start = models.IntegerField(blank=True,null=True) end = models.IntegerField(blank=True,null=True) length = models.IntegerField(blank=True,null=True) since_last = models.IntegerField(blank=True,null=True) referrer = models.CharField(max_length=2000,blank=True,null=True)
# 6-7 NoAPS = models.TextField(blank=True,null=True) NoGCPAPS = models.IntegerField(blank=True,null=True)
PyConCZ’16
GETTING WORSE
24
# 8 - 23 VoGCPAPS = models.IntegerField(blank=True,null=True) AVoGCPAPS = models.IntegerField(blank=True,null=True) NoDPAPS = models.IntegerField(blank=True,null=True) VoDPAPS = models.IntegerField(blank=True,null=True) AvoDPAPS = models.IntegerField(blank=True,null=True) USDSPS = models.FloatField(blank=True,null=True) VoDPPS_USD = models.FloatField(blank=True,null=True) VoGCPPS_USD = models.FloatField(blank=True,null=True) VoGCPPS_DINARS = models.FloatField(blank=True,null=True) EUPS = models.IntegerField(blank=True,null=True) EDPS = models.IntegerField(blank=True,null=True) EPPS = models.FloatField(blank=True,null=True) NoTAAPS = models.IntegerField(blank=True,null=True) VoTAAPS = models.FloatField(blank=True,null=True) DoTAAPS = models.TextField(blank=True,null=True) GLaSS = models.IntegerField(blank=True,null=True)
PyConCZ’16
AND WORSE
25
# 24-40 GLaSE = models.IntegerField(blank=True,null=True) PaSS = models.IntegerField(blank=True,null=True) PaSE = models.IntegerField(blank=True,null=True) SCBaSS = models.IntegerField(blank=True,null=True) SCBaSE = models.IntegerField(blank=True,null=True) GCBaSS = models.IntegerField(blank=True,null=True) GCBaSE = models.IntegerField(blank=True,null=True) GCUPS = models.IntegerField(blank=True,null=True) GCCPS = models.IntegerField(blank=True,null=True) GCBPS = models.TextField(blank=True,null=True) DBaSS = models.IntegerField(blank=True,null=True) DBaSE = models.IntegerField(blank=True,null=True) DUPS = models.IntegerField(blank=True,null=True) DCPS = models.IntegerField(blank=True,null=True) DBPS = models.TextField(blank=True,null=True) EBaSS = models.IntegerField(blank=True,null=True) EBaSE = models.IntegerField(blank=True,null=True) EUPS_all = models.IntegerField(blank=True,null=True)
PyConCZ’16
AND WORS… OH COME ON...
26
# 41 - 56 OECOS = models.IntegerField(blank=True,null=True) PECPPS = models.FloatField(blank=True,null=True) EBPS = models.TextField(blank=True,null=True) APPS = models.TextField(blank=True,null=True) VoAAPPS = models.FloatField(blank=True,null=True) APwDPS = models.TextField(blank=True,null=True) VoAAPwDPS = models.FloatField(blank=True,null=True) APwCPS = models.TextField(blank=True,null=True) VoAAPwCPS = models.FloatField(blank=True,null=True) ExPPS = models.IntegerField(blank=True,null=True) VoAEPPS = models.FloatField(blank=True,null=True) EPwCPS = models.IntegerField(blank=True,null=True) VoEPwCPS = models.FloatField(blank=True,null=True) EPwDPS = models.IntegerField(blank=True,null=True) VoEPwDPS = models.FloatField(blank=True,null=True) poAMOBEP = models.TextField(blank=True,null=True)
PyConCZ’16
UH. END.
27
# 57 - 63 APADtIF = models.TextField(blank=True,null=True) LABSE = models.CharField(max_length=200,blank=True,null=True) LPCaSS = models.IntegerField(blank=True,null=True) LPCaSE = models.IntegerField(blank=True,null=True) PADaPS = models.IntegerField(blank=True,null=True) LTaSP = models.IntegerField(blank=True,null=True) SPDO = models.IntegerField(blank=True,null=True)
PyConCZ’16
Overloaded database with big inserts
Drastic decrease of response time
Service was unusable
Drastic decrease of active users28
PROBLEM
PyConCZ’16 29
HOTFIX
def receive_session_stats(request):
[...]
session = SessionData() session.player = stats
# 1 - 5 session.start = data.get("1",0) session.end = data.get("2",0)
[...]
session.save() return HttpResponse()
def receive_session_stats(request): return HttpResponse()
PyConCZ’16
class NewSessionData(models.Model): player = models.ForeignKey(PlayerStats) data = models.TextField()
➤ Gather lots and lots of stats from mobile app
➤ Store them locally for short period of time
➤ Export them to BigTable
➤ Make it readable via Django
30
LONG TERM SOLUTION
PyConCZ’16
INEFFICIENT FRAMEWORK USAGE
PyConCZ’16
➤ Is awesome!
➤ Is easy to use, powerful and generally great
➤ Is dangerous!
➤ It makes it very easy to forget about performance
➤ Is magical
➤ You don’t see queries that are generated
➤ Those queries might be not optimal
32
DJANGO ORM
PyConCZ’16 33
DJANGO ORM
class Author(models.Model): first_name = models.CharField(max_length=64) last_name = models.CharField(max_length=128)
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField() def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16 34
DJANGO ORM
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField()
def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16 35
DJANGO ORM
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField()
def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16 36
DJANGO ORM
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField(db_index=True)
def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16 37
DJANGO ORM
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField(db_index=True)
def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16 38
DJANGO ORM
class Book(models.Model): title = models.CharField(max_length=128) author = models.ForeignKey(Author) pages = models.IntegerField(db_index=True)
def get_books_by_size(request, pages_min, pages_max): books = Book.objects\ .select_related('author')\ .filter(pages__gte=pages_min, pages__lte=pages_max)
return JsonResponse({ "books": [{ "id": book.id, "title": book.title, "pages": book.pages, "author": { "id": book.author.pk, "last_name": book.author.last_name, } } for book in books] })
PyConCZ’16
AND SO ON
PyConCZ’16
LET’S TEST!
PyConCZ’16
➤ Inside job
➤ You must
➤ Have a deep understanding of system
➤ Set up internal monitoring software on your system
➤ Without it, you just monitor average response time
41
BEFORE YOU BEGIN
PyConCZ’16
➤ Application monitoring
➤ Availability monitoring
➤ Error reporting
➤ SLA calculator
➤ Integrates with your app via agent
➤ Works very well with Python
42
NEW RELIC
PyConCZ’16
AVERAGE RESPONSE TIME
43
PyConCZ’16
SLOWEST ENDPOINTS
44
PyConCZ’16 45
DETAILED BREAKDOWN
PyConCZ’16
➤ Custom breakdown segments available.
46
EVEN MORE DETAILED BREAKDOWN
PyConCZ’16
➤ Good to know
➤ Helps with maintenance planning
47
REQUESTS PER MINUTE
PyConCZ’16 48
SLOWEST SQL QUERIES
PyConCZ’16 49
DATABASE INSIGHTS
PyConCZ’16
TOOLS
PyConCZ’16
APACHE BENCH
PyConCZ’16
➤ part of Apache Server
➤ ab -n 100 -c 10 http://hit--me.herokuapp.com/100ms
➤ -n <= total number of requests to be made
➤ -c <= maximum concurrency level
52
APACHE BENCH
PyConCZ’16 53
APACHE BENCH
PyConCZ’16 54
http://www.myloadtest.com/performance-testing-memes/
PyConCZ’16
JMETER
PyConCZ’16
JMETER
56
➤ Pros:
➤ very powerful load testing tool
➤ tests can be imported to some cloud services
➤ load tests almost everything
➤ Cons:
➤ Complicated
➤ Big entry threshold
PyConCZ’16
JMETER
57
PyConCZ’16
JMETER
58
PyConCZ’16
JMETER
59
PyConCZ’16
JMETER
60
PyConCZ’16
JMETER
61
PyConCZ’16
JMETER
62
PyConCZ’16
CUSTOM SCRIPT
PyConCZ’16
➤ Pros:
➤ Can test exactly what you need
➤ Cons:
➤ Possible re-inventing a wheel
➤ Who said your testing script is optimal… :)
64
CUSTOM SCRIPT
PyConCZ’16
LOCUST
PyConCZ’16
➤ locust.io
➤ Python based, using gevent
➤ Well documented
➤ Load testing as Python code
➤ Distributed tests support
➤ Small problem: Python 2 only
66
LOCUST
PyConCZ’16 67
LOCUST
from locust import HttpLocust, TaskSet, task
class HitMeTasks(TaskSet): @task(3) def index(self): self.client.get("/")
@task(10) def test100ms(self): self.client.get("/100ms")
class HitMeLocust(HttpLocust): host = "http://hit--me.herokuapp.com" task_set = HitMeTasks min_wait = 100 max_wait = 100
PyConCZ’16
LOCUST
68
PyConCZ’16
LOCUST
69
PyConCZ’16 70
http://www.myloadtest.com/performance-testing-memes/
PyConCZ’16
CLOUD
PyConCZ’16
➤ Load Testing as a Service:
➤ loader.io
➤ blazemeter.io
➤ loadimpact.com
➤ and-so-on.com
➤ Server ownership verification
➤ Generally easy to use
72
CLOUD
PyConCZ’16
➤ Distributed
➤ API available
➤ Email results
➤ Free plan
➤ 1 host
➤ 10,000 users
➤ 2 endpoints
➤ 1 minute test
73
LOADER.IO
➤ Pro plan
➤ ∞ hosts
➤ 100,000 users
➤ 10 endpoints
➤ 10 minutes tests
PyConCZ’16
➤ Include Load Testing into your workflow
➤ No magic formula, adopt solution to problem
➤ Be rational
➤ Don’t be afraid to break your system!
➤ Don’t Load Test your production!
➤ Don’t “Load Test” other servers!!
79
FINAL THOUGHTS
PyConCZ’16
THANK YOUQuestions?
! DariuszAniszewski
" @aniszewski_eu