20160503 amazed by aws | tips about performance on aws
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
김일호, Solutions Architect03-May-2016
Tips about performance on AWSAmazed by AWS
EC2 performance dive deep
InstancesAPIs
Networking
EC2EC2
Purchase Options
Amazon Elastic Compute Cloud is Big
Host ServerHypervisor
Guest 1 Guest 2 Guest n
Amazon EC2 Instances
2006 2008 2010 2012 2014 2016
m1.small
m1.largem1.xlarge
c1.mediumc1.xlarge
m2.xlarge
m2.4xlargem2.2xlarge
cc1.4xlarge
t1.micro
cg1.4xlarge
cc2.8xlarge
m1.medium
hi1.4xlarge
m3.xlargem3.2xlarge
hs1.8xlarge
cr1.8xlarge
c3.largec3.xlarge
c3.2xlargec3.4xlargec3.8xlargeg2.2xlarge
i2.xlargei2.2xlargei2.4xlargei2.4xlarge
m3.mediumm3.large
r3.larger3.xlarger3.2xlarger3.4xlarger3.8xlarge
t2.microt2.smallt2.med
c4.largec4.xlargec4.2xlargec4.4xlargec4.8xlarge
d2.xlarged2.2xlarged2.4xlarged2.8xlargeg2.8xlarge
t2.largem4.large
m4.xlargem4.2xlargem4.4xlarge
m4.10xlarge
Amazon EC2 Instances History
What to Expect from the Session
• Defining system performance and how it is characterized for different workloads
• How Amazon EC2 instances deliver performance while providing flexibility and agility
• How to make the most of your EC2 instance experience through the lens of several instance types
Defining Performance
• Servers are hired to do jobs• Performance is measured differently depending on the job
Hiring a Server
?
• What performance means depend on your perspective:– Response time– Throughput– Consistency
Defining Performance: Perspective Matters
Application
System libraries
System calls
Kernel
Devices
Workload
Simple Performance Model for Single Thread
• Using CPU: executing (in user mode)
• Not using CPU: waiting for turn on CPU, waiting for disk or network I/O, thread locks, memory paging, or for more work.
Performance Factors
Resource Performance factors Key indicatorsCPU Sockets, number of cores, clock
frequency, bursting capabilityCPU utilization, run queue length
Memory Memory capacity Free memory, anonymous paging, thread swapping
Network interface
Max bandwidth, packet rate Receive throughput, transmit throughput over max bandwidth
Disks Input / output operations per second, throughput
Wait queue length, device utilization, device errors
Resource Utilization
• For given performance, how efficiently are resources being used• Something at 100% utilization can’t accept any more work• Low utilization can indicate more resource is being purchased
than needed
Example: Web Application• MediaWiki installed on Apache with 140 pages of content• Load increased in intervals over time
Example: Web Application• Memory stats
Example: Web Application• Disk stats
Example: Web Application• Network stats
Example: Web Application• CPU stats
• Picking an instance is tantamount to resource performance tuning• Give back instances as easily as you can acquire new ones• Find an ideal instance type and workload combination
Instance Selection = Performance Tuning
Delivering Compute Performance with Amazon EC2 Instances
CPU Instructions and Protection Levels
• CPU has at least two protection levels: ring0 and ring1• Privileged instructions can’t be executed in user mode to protect
system. Applications leverage system calls to the kernel.
Kernel
Application
Example: Web application system calls
X86 CPU Virtualization: Prior to Intel VT-x
• Binary translation for privileged instructions• Para-virtualization (PV)
• PV requires going through the VMM, adding latency• Applications that are system call bound are most affected
VMM
Application
Kernel
PV
X86 CPU Virtualization: After Intel VT-x• Hardware assisted virtualization (HVM)• PV-HVM uses PV drivers opportunistically for operations that are
slow emulated: • e.g. network and block I/O
KernelApplication
VMM
PV-HVM
Tip: Use PV-HVM AMIs with EBS
Time Keeping Explained
• Time keeping in an instance is deceptively hard• gettimeofday(), clock_gettime(), QueryPerformanceCounter()• The TSC
• CPU counter, accessible from userspace• Requires calibration, vDSO• Invariant on Sandy Bridge+ processors
• Xen pvclock; does not support vDSO• On current generation instances, use TSC as clocksource
Tip: Use TSC as clocksource
tsc
source=tsc
CPU Performance and Scheduling
• Hypervisor ensures every guest receives CPU time• Fixed allocation
• Uncapped vs. capped• Variable allocation
• Different schedulers can be used depending on the goal• Fairness• Response time / deadline• Shares
Review: C4 Instances
Custom Intel E5-2666 v3 at 2.9 GHzP-state and C-state controls
Model vCPU Memory (GiB) EBS (Mbps)c4.large 2 3.75 500c4.xlarge 4 7.5 750c4.2xlarge 8 15 1,000c4.4xlarge 16 30 2,000c4.8xlarge 36 60 4,000
What’s new in C4: P-state and C-state control
• By entering deeper idle states, non-idle cores can achieve up to 300MHz higher clock frequencies
• But… deeper idle states require more time to exit, may not be appropriate for latency sensitive workloads
Tip: P-state control for AVX2
• If an application makes heavy use of AVX2 on all cores, the processor may attempt to draw more power than it should
• Processor will transparently reduce frequency• Frequent changes of CPU frequency can slow an application
Review: T2 Instances
• Lowest cost EC2 Instance at $0.013 per hour• Burstable performance• Fixed allocation enforced with CPU Credits
Model vCPU CPU Credits / Hour
Memory (GiB)
Storage
t2.micro 1 6 1 EBS Onlyt2.small 1 12 2 EBS Onlyt2.medium 2 24 4 EBS Onlyt2.large 2 36 8 EBS Only
How Credits Work
• A CPU Credit provides the performance of a full CPU core for one minute
• An instance earns CPU credits at a steady rate
• An instance consumes credits when active
• Credits expire (leak) after 24 hours
Baseline Rate
Credit Balance
BurstRate
Tip: Monitor CPU credit balance
Monitoring CPU Performance in Guest
• Indicators that work is being done• User time• System time (kernel mode)• Wait I/O, threads blocked on disk I/O
• Else, Idle• What happens if OS is scheduled off the CPU?
Tip: How to interpret Steal Time
• Fixed CPU allocations of CPU can be offered through CPU caps
• Steal time happens when CPU cap is enforced• Leverage CloudWatch metrics
Delivering I/O Performance with Amazon EC2 Instances
I/O and Devices Virtualization
• Scheduling I/O requests between virtual devices and shared physical hardware
• Split driver model • Intel VT-d
• Direct pass through and IOMMU for dedicated devices• Enhanced Networking
Hardware
Split Driver ModelDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Split Driver Model
• Each virtual device has two main components• Communication ring buffer • An event channel signaling activity in the ring buffer
• Data is transferred through shared pages• Shared pages requires inter domain permissions, or granting
Review: I2 Instances
16 vCPU: 3.2 TB SSD; 32 vCPU: 6.4 TB SSD365K random read IOPS for 32 vCPU instance
Model vCPU Memory (GiB)
Storage Read IOPS Write IOPS
i2.xlarge 4 30.5 1 x 800 SSD 35,000 35,000i2.2xlarge 8 61 2 x 800 SSD 75,000 75,000i2.4xlarge 16 122 4 x 800 SSD 175,000 155,000i2.8xlarge 32 244 8 x 800 SSD 365,000 315,000
Granting in pre-3.8.0 Kernels
• Requires “grant mapping” prior to 3.8.0• Grant mappings are expensive operations due to TLB flushes
read(fd, buffer,…)
Granting in 3.8.0+ Kernels, Persistent and Indirect
• Grant mappings are setup in a pool once• Data is copied in and out of the grant pool
read(fd, buffer…)Copy to and from grant pool
Tip: Use 3.8+ kernel
• Amazon Linux 13.09 or later• Ubuntu 14.04 or later• RHEL7 or later• Etc.
Event Handling
• Guest vCPUs are interrupted to process events. • Pre-2.6.36 kernels: notifications went to a single virtual
hardware interrupt • Post-2.6.36 kernels: allow instance to tell hypervisor to deliver
notification to a specific vCPU for balancing• Check "dmesg" for the following text: "Xen HVM callback vector for
event delivery is enabled“• Also, check version of irqbalance is 1.0.7 or higher
Hardware
Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
Frontend driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Device Pass Through: Enhanced Networking
• SR-IOV eliminates need for driver domain• Physical network device exposes virtual function to
instance• Requires a specialized driver, which means:
• Your instance OS needs to know about it • EC2 needs to be told your instance can use it
Hardware
After Enhanced NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
NIC Driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
SR-IOV Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
After Enhanced NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
NIC Driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
SR-IOV Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Hardware
After Enhanced NetworkingDriver Domain Guest Domain Guest Domain
VMM
Frontend driver
NIC Driver
Backend driver
DeviceDriver
Physical CPU
Physical Memory
SR-IOV Network Device
Virtual CPU Virtual Memory
CPU Scheduling
Sockets
Application
Tip: Use Enhanced Networking
• Highest packets-per-second• Lowest variance in latency• Instance OS must support it• Look for SR-IOV property of instance or image
Inter-instance latency
How to build enhanced network driver on Linux
Let’s start to create AMI for Enhanced networking enabled
CentOS 6.5 with c4.8xlarge
Let’s start with the AMI officially provided CentOS, should be clean. CentOS is provided in awsmarketplace.
Only missing c4.8xlargeL
Anyway, let’s go to AMI search
Find and Select AMI
CentOS 6 x86_64 (2014_09_29) EBS HVM-74e73035-3435-48d6-88e0-89cc02ad83ee-ami-a8a117c0.2
ami-c2a818aa (IAD)
This is the CentOS AMI at awsmarketplace CentOS. Nice ?!?!
Check the requirements ofEnabling Enhanced Networking on Linux• C3, C4, D2, I2, M4 and R3• HVM AMI with Linux kernel above V.2.6.32 • Launch the instance in VPC
• A network driver to support enhanced networking on Linux.
Check kernel version and network driver
[root@ip-192-168-1-171 ~]# cat /etc/redhat-releaseCentOS release 6.5 (Final)
[root@ip-192-168-1-171 ~]# uname -na
Linux ip-192-168-1-171 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-192-168-1-171 ~]# modinfo ixgbevfmodinfo ixgbevf
filename: /lib/modules/2.6.32-431.29.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.koversion: 2.7.12-k
license: GPL
description: Intel(R) 82599 Virtual Function Driver
author: Intel Corporation, <[email protected]>
srcversion: E75203124BB105EC871944Falias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*depends:
vermagic: 2.6.32-431.29.2.el6.x86_64 SMP mod_unload modversionsparm: debug:Debug level (0=none,...,16=all) (int)
Launch c4.large instance and login
Let’s update all including kernel
root@ip-192-168-1-171 ~]# yum update -yLoaded plugins: fastestmirror, prestoLoading mirror speeds from cached hostfile* base: mirrors.mit.edu* extras: linux.cc.lehigh.edu* updates: mirrors.lga7.us.voxel.net
Setting up Update ProcessResolving Dependencies--> Running transaction check---> Package audit.x86_64 0:2.2-4.el6_5 will be updated---> Package audit.x86_64 0:2.3.7-5.el6 will be an update---> Package audit-libs.x86_64 0:2.2-4.el6_5 will be updated---> Package audit-libs.x86_64 0:2.3.7-5.el6 will be an update---> Package authconfig.x86_64 0:6.1.12-13.el6 will be updated---> Package authconfig.x86_64 0:6.1.12-19.el6 will be an update---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be updated---> Package bash.x86_64 0:4.1.2-29.el6 will be an update---> Package binutils.x86_64 0:2.20.51.0.2-5.36.el6 will be updated---> Package binutils.x86_64 0:2.20.51.0.2-5.42.el6 will be an update---> Package ca-certificates.noarch 0:2014.1.98-65.0.el6_5 will be updated---> Package ca-certificates.noarch 0:2014.1.98-65.1.el6 will be an update---> Package centos-release.x86_64 0:6-5.el6.centos.11.2 will be updated---> Package centos-release.x86_64 0:6-6.el6.centos.12.2 will be an update---> Package coreutils.x86_64 0:8.4-31.el6_5.2 will be updated---> Package coreutils.x86_64 0:8.4-37.el6 will be an update---> Package coreutils-libs.x86_64 0:8.4-31.el6_5.2 will be updated………………………
Reboot and check the updated
[root@ip-192-168-1-171 ~]# cat /etc/redhat-release
CentOS release 6.6 (Final)
[root@ip-192-168-1-171 ~]# uname -na
Linux ip-192-168-1-171 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-192-168-1-171 ~]# modinfo ixgbevf
filename: /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.12.1-k
license: GPL
description: Intel(R) 82599 Virtual Function Driver
author: Intel Corporation, <[email protected]>
srcversion: 8797AC845BB302315230490
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 2.6.32-504.12.2.el6.x86_64 SMP mod_unload modversions
parm: debug:Debug level (0=none,...,16=all) (int)
Upgraded network driver installedversion: 2.7.12-k -> 2.12.1-k
Enable SR-IOV
Install AWS CLI or EC2 CLI toolsNot supported at AWS Console yet L
1. Stop the instance2. Enable SR-IOV of the instance with CLI3. Check the status4. Start the instance
a82066443ffe:~ ilho$ aws ec2 modify-instance-attribute --instance-id i-681280bf \--sriov-net-support simple
a82066443ffe:~ ilho$ aws ec2 describe-instance-attribute --instance-id i-681280bf \--attribute sriovNetSupport
{
"InstanceId": "i-681280bf",
"SriovNetSupport": {
"Value": "simple"
}
}
A problem with more than 32 vCPUs on Linux
CentOS 6.x does not support more than 32 vCPUs in kernel.It can not boot when you launch c4.8xlarge(36 vCPUs) L
• d2.8xlarge and m4.10xlarge? -> L
A solution is to add an option to kernel boot parameter
Add maxcpus option to kernel boot parameter
$ vi /boot/grub/menu.lstAdd maxcpus=32
[root@ip-10-10-10-242 ~]# cat /boot/grub/menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,0)
# kernel /boot/vmlinuz-version ro root=/dev/vda1
# initrd /boot/initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=1
serial --unit=0 --speed=115200
terminal --timeout=1 serial console
title CentOS (2.6.32-431.29.2.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 maxcpus=32 ro root=UUID=dcb1645e-05a6-4311-8bce-a9c12bec5801 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 crashkernel=auto SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img
Now let’s launch c4.8xlarge instance
I think it’s ready to go.RebootLaunch a c4.8xlarge instance
It can not launch.
Why?
AWS Marketplace AMI does not support to launch an instance type not in the list.
Change the base AMI
Find CentOS 6.5 community version AMI : CentOS-6.5-base-20150305 (ami-0e80db66)
Repeat three steps1. $ sudo yum update –y2. Enable SR-IOV3. Add maxcpus=32
Network driver version should be checked
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.htmlTo enable enhanced networking on your instance, you must ensure that its kernel has the ixgbevf module installed and that you set the sriovNetSupport attribute for the instance. For the best performance, we recommend that the ixgbevf module is version 2.14.2 or higher.
Build and Install the network driver #2
[ec2-user@ip-192-168-1-50 src]$ make;sudo make installmake -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/srcmodulesmake[1]: Entering directory ̀ /usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_main.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_param.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_ethtool.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/kcompat.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_vf.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_mbx.oLD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.oBuilding modules, stage 2.MODPOST 1 modulesCC /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.mod.oLD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko.unsignedNO SIGN [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko
make[1]: Leaving directory /̀usr/src/kernels/2.6.32-504.12.2.el6.x86_64'make -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/srcmodulesmake[1]: Entering directory ̀ /usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
Building modules, stage 2.MODPOST 1 modules
make[1]: Leaving directory /̀usr/src/kernels/2.6.32-504.12.2.el6.x86_64'gzip -c ../ixgbevf.7 > ixgbevf.7.gz# remove all old versions of the driverfind /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko -exec rm -f {} \; || truefind /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko.gz -exec rm -f {} \; || trueinstall -D -m 644 ixgbevf.ko /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko/sbin/depmod -a 2.6.32-504.12.2.el6.x86_64 || trueinstall -D -m 644 ixgbevf.7.gz /usr/share/man/man7/ixgbevf.7.gzman -c -P'cat > /dev/null' ixgbevf || true
http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/
Check the new driver installed
[ec2-user@ip-192-168-1-50 src]$ modinfo ixgbevf
filename: /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.16.1
license: GPL
description: Intel(R) 10 Gigabit Virtual Function Network Driver
author: Intel Corporation, <[email protected]>
srcversion: 3B690FE23A02C25EF74012F
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 2.6.32-504.12.2.el6.x86_64 SMP mod_unload modversions
parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int)
reboot
[ec2-user@ip-192-168-1-50 ~]$ ethtool -i eth0
driver: ixgbevf
version: 2.16.1
firmware-version: N/A
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
Share AMI to the users
Sharing the customized AMI with your team ~
Let’s get some sleep… J
Reply back with 1/2 (OS unreachable)
Root cause found
the 70-persistent-net.rules file /etc/udev/rules.d/ having an entry with the MAC address of the original instance the AMI was taken from. When the image is taken, the MAC from the original instance persists, which tells the OS after the new instance is deployed that the now-nonexistent MAC of the original instance should be eth0. Since the OS cannot find the device with the original MAC address, eth0 fails to be identified and isn't brought up.
A solution to avoid this caveat
Before creating AMI, You must remove/etc/udev/rules.d/70-persistent-net.rules file
Create AMIShare AMI Now launch c4 instance successfully with the latest driver.
A small contribution
The note was added in public document “enhanced networking” section.
Lesson learned
Just use CentOS 7 or Amazon Linux. J
EBS designing for performance
Amazon EBS overview
EBS =
What is EBS?
• Network block storage as a service
• EBS volumes attach to any Amazon EC2 instance in the
same Availability Zone
• Designed for five nines of availability
• 2 million volumes created every day
EBS volume types
Magnetic General purpose (SSD) Provisioned IOPS (SSD)
EBS volume types
IOPS: Typically 100, best effortThroughput: 40-90 MB/sLatency: Read 10-40ms, Write 2-10msBest for infrequently accessed data
Magnetic
EBS volume types
IOPS Baseline: 100-10,000 (3 / GiB)IOPS Burst: 30 minutes @ 3,000Throughput: Up to 160 MB/sLatency: Single-digit msPerformance consistency: 99%Most workloadsGeneral purpose (SSD)
EBS volume types
IOPS: 100-20,000 (customer provisioned)Throughput: Up to 320 MB/sLatency: Single-digit msPerformance consistency: 99.9%Mission Critical workloadsProvisioned IOPS (SSD)
Performance
Queuing theory
Little’s law is the foundation for performance tuning theory• Mathematically proven by John Little in 1961
𝑾 =𝑳𝑨
W = Wait time = average wait time per requestL = Queue length = average number of requests waitingA = Arrival rate = the rate of requests arriving
EBS performance is related to this law
Performance optimization is measured by:
IOPS: Read/write I/O rate (IOPS)
Latency: Time between I/O submission and completion (ms)
Throughput: Read/write transfer rate (MB/s); throughput = IOPS X I/O size
Key components to performance
EC2 instance
I/O
EBS
Network link
A day in the life of an I/O
A day in the life: I/O
All I/O must pass through I/O domainRequires “grant mapping” prior to 3.8.0Grant mappings are expensive operations due to TLB flushes
EBS
Grant mappingread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance
A day in the life: I/O (continued)
Responses
Requests
Instance I/O domain
READ8KB @ 1234
Request queue is a single memory pageEach I/O request has 11 grant references (4KiB/reference)Maximum data in queue = 1408 KiB
3.8.0+ Kernels – Persistent grants
Grant mappings are setup in a pool onceData is copied in and out of the grant poolCopying is significantly faster than remapping
EBS
Grant poolread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance
3.8.0+ Kernels – Indirect grants
Responses
Requests
Instance
READ8KB @ 1234
I/O domain
Each I/O request has grant references that contain grant referencesMaximum data in queue = 4096 KiB (default)
Instance I/O: Before 3.8.0
0 1 2 3 4 5 30 31
128KiB
44KiB 44KiB 40KiB
Instance I/O: Linux 3.8.0+
0 1 2 3 4 5 30 31
128KiB
Tip: Use 3.8+ kernel
Amazon Linux 2013.09 or laterUbuntu 14.04 or laterRHEL7 or laterEtc.
Queue depth
An I/O operation
EBS
After it’s gone, it’s gone
EC2
Queue depth is the pending I/O for a volume
Workload/ software
Typical block size
Random/Seq?
Max EBS @ 500 MB/s instances
Max EBS @ 1 GB/s instances
Max EBS @ 10 GB/s instances
Oracle DB Configurable:2 KB–16 KBDefault 8 KB
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
Microsoft SQL Server
8 KB w/ 64 KB extents
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS
PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MongoDB 4 KB sequential ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
Apache Cassandra
4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS
Cheat sheet: Sample storage workloads on AWS
Example workload
Transaction (OLTP)Examples: eCommerce website, metadata storageBenchmark: MySQL + sysbench
Tip: Workload
Where possible, use real production workloads for performance testing
Baseline configuration
Availability Zone: US West (Oregon)Instance type: m2.4xlarge
vCPU: 8Memory: 68.4GiBEBS-optimized
Data volume: 500GiB EBS magneticOS: Amazon Linux 2015.03.1
Optimization: Increase parallelism
MySQL threads
Tran
sact
ions
(n)
Baseline
2 n
Tip: Parallelism
Increase parallelism of your system
Key components to performance
EC2 instance
I/O
EBS
Network link
m2.4xlargeCPU: Intel Xeon vCPU: 8Memory: 68.4 GiBPrice: $0.98/hour*
Instance selection
r3.2xlargeCPU: Intel Xeon E5-2670 v2vCPU: 8Memory: 61 GiBEnhanced networkingPrice: $0.70/hour*
* All pricing from US West (Oregon)
EBS optimized instances
• Most instance families support the EBS-optimized flag
• EBS-optimized instances now support up to 4 Gb/s
• Drive 32,000 16K IOPS or 500 MB/s
• Available by default on newer instance types
• EC2 *.8xlarge instances support 10 Gb/s network
• Max IOPS per node supported is ~48,000 IOPS @ 16K I/O
Tip: Use EBS-optimized instances
Use EBS-optimized instances for consistent EBS performance
Updated configuration: Instance type
Availability Zone: US West (Oregon)Instance type: r3.2xlarge
vCPU: 8Memory: 61 GiBEBS-optimized
EBS volume: 500GiB magneticOS: Amazon Linux 2015.03.1
25%
Optimization: Current generation instances
MySQL threads
Tran
sact
ions
(n)
Baseline
r3.2xlarge
2 n
Tip: Instance selection
Use the right instance family for your workload
Use current generation instances
Key components to performance
EC2 instance
I/O
EBS
Network link
Volume selection
EBS magneticLatency:
Read: 10-40msWrite: 2-10ms
SSD backedLatency:
Read/Write: Single-digit ms
File systems
Use a modern, journaled filesystemext4, xfs, etc.
Ensure partitions are aligned on 4KiB boundaries
Pre-warming
Volume initialization
Newly created volumes• Just attach, mount, and go!• Pre-warming is no longer recommended
Volumes restored from snapshots• You can use your volume right away• Accelerate data loading by reading
Updated configuration: EBS volumes
Availability Zone: US West (Oregon)Instance type: r3.2xlarge
vCPU: 8Memory: 61 GiBEBS-optimized
Boot volume: 8 GiB – EBS general purposeData volume: 500 GiB – EBS general purposeOS: Amazon Linux 2015.03.1
Optimization: Volume selectionTr
ansa
ctio
ns (n
) 19% 50%
MySQL threads
Baseline
r3.2xlarge
r3.2xlarge gp2
2 n
Tip: Volume selection
Use SSD backed volumes when performance matters
EBS IOPS vs. Throughput
20,000 IOPS PIOPS volume
20,000 IOPS
320 MB/s throughput
You can achieve 20,000 IOPS when driving smaller I/O operations
You can achieve up to 320 MB/s when driving larger I/O operations
EBS IOPS vs. Throughput
8,000 IOPS PIOPS volume
8,000 IOPS
320 MB/s throughput
8,000 x 8 KB = 64 MB/s8,000 x 16 KB = 128 MB/s8,000 x 32 KB = 256 MB/s
16,000 x 8 KB = 128 MB/s
8,000 x 64 KB=512 MB/s
5,000 x 64 KB = 320 MB/s
Striping
Increases performance, or capacity, or both
Don’t mix volume types
Typically RAID 0 or LVM stripe
Avoid RAID for redundancyEBS
EC2
Striping: Snapshots
Quiesce I/O1. Database: FLUSH and LOCK tables2. Filesystem: sync and fsfreeze3. EBS: snapshot all volumes
When snapshot API returns, it is safe to resume
EBS-optimized instance
Four key components: Balanced
EC2
A “boatload” of I/O
Right-sized EBS
Monitoring
Amazon CloudWatch
Important Amazon CloudWatch metrics: • IOPS and bandwidth
• Latency
• Queue depth
All EBS metrics are prefixed with “Volume”
CloudWatch: Instance bandwidth
m4.2xlargeInstance: 128MB/s
m4.4xlargeInstance: 256MB/s
m4.10xlargeVolume: 320MB/s
S3 Performance tips
Distributing Key Names
Don’t do this
<my_bucket>/2013_11_13-164533125.jpg<my_bucket>/2013_11_13-051033564.jpg
<my_bucket>/2013_11_13-061133789.jpg<my_bucket>/2013_11_13-051033458.jpg<my_bucket>/2013_11_12-063433125.jpg<my_bucket>/2013_11_12-021033564.jpg<my_bucket>/2013_11_12-065533789.jpg
<my_bucket>/2013_11_12-011033458.jpg<my_bucket>/2013_11_11-022333125.jpg<my_bucket>/2013_11_11-153433564.jpg<my_bucket>/2013_11_11-065233789.jpg<my_bucket>/2013_11_11-065633458.jpg
Distributing Key Names
Add randomness to the beginning of the key name
<my_bucket>/521335461-2013_11_13.jpg<my_bucket>/465330151-2013_11_13.jpg
<my_bucket>/987331160-2013_11_13.jpg<my_bucket>/465765461-2013_11_13.jpg<my_bucket>/125631151-2013_11_13.jpg<my_bucket>/934563160-2013_11_13.jpg<my_bucket>/532132341-2013_11_13.jpg
<my_bucket>/565437681-2013_11_13.jpg<my_bucket>/234567460-2013_11_13.jpg<my_bucket>/456767561-2013_11_13.jpg<my_bucket>/345565651-2013_11_13.jpg<my_bucket>/431345660-2013_11_13.jpg
Other Techniques for Distributing Key Names
Store objects as a hash of their name• add the original name as metadata
• “deadmau5_mix.mp3” à 0aa316fb000eae52921aab1b4697424958a53ad9– watch for duplicate names!
• prepend keyname with short hash• 0aa3-deadmau5_mix.mp3
Epoch time (reverse)• 5321354831-deadmau5_mix.mp3
Randomness in a Key Name Can Be an Anti-Pattern
Lifecycle policies LISTs with prefix filtersMaintaining thumbnails of images
• craig.jpg -> stored as orig-09329jed0fc • thumb-09329jed0fc
When you need to recover a file with its original name
Solving for the Anti-Pattern
Add additional prefixes to help sorting
Amazon S3 maintains keys lexicographically in its internal indices
<my_bucket>/images/521335461-2013_11_13.jpg
<my_bucket>/images/465330151-2013_11_13.jpg<my_bucket>/movies/293924440-2013_11_13.jpg<my_bucket>/movies/987331160-2013_11_13.jpg<my_bucket>/thumbs-small/838434842-2013_11_13.jpg<my_bucket>/thumbs-small/342532454-2013_11_13.jpg
<my_bucket>/thumbs-small/345233453-2013_11_13.jpg<my_bucket>/thumbs-small/345453454-2013_11_13.jpg
Amazon CloudFront
CloudFront dynamic content accleratoion
Region
Edge Location
12 Regions32 Availability Zones54 Edge Locations
Need to update
We’re here J
Configure multiple origins
Elastic Load Balancing
Dynamic content
Amazon EC2
Static content
Amazon S3
* (default)
/error/*/assets/*
Amazon CloudFrontexample.com
CloudFront Behaviors
CloudFront
CustomerLocationwww.mysite.com
Path Pattern Matching/*.jpg; /*.php etc.
GET http://mysite.com/images/1.jpg to ORIGIN AGET http://mysite.com/index.php to ORIGIN B
GET http://mysite.com/web/home.css to ORIGIN CGET http://mysite.com/* (DEFAULT) to ORIGIN D
Origin A: S3 bucket
Origin B: www.mysite.com
Origin C: S3 Bucket
Origin D: www.mysite.com
Path Pattern Matching
/*.php
/images/*.jpg
/web/*.css
/*.* (DEFAULT)
Region
Edge Location
12 Regions32 Availability Zones54 Edge Locations
Need to update AWS optimized network
Internet
Demo J
S3 Transfer AccelerationNew~
Amazon S3 Transfer AccelerationEmbedded WAN acceleration
S3 BucketAWS EdgeLocation
Uploader
OptimizedThroughput!
Move over long geographic distances
Up to 300% (6x) faster
No firewall mods, no client software
54 global edge locations
Change your endpoint, not your code
Accelerate Speed Comparison
• Test URL• http://s3-accelerate-speedtest.s3-
accelerate.amazonaws.com/en/accelerate-speed-comparsion.html
• bit.ly/news3ta• Test Result (May-02-2016)
• Tested on May-02-2016, LGU+ Wifi at GSTower in Seoul• http://bit.ly/newss3taresult
Testing S3 Transfer Accelerator by AWSCLI
$ sudo pip install –upgrade awscli
$ aws configure set default.s3.use_accelerate_endpoint true
Testing S3 Transfer Accelerator by AWSCLI
$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/ --endpoint-url http://ilho-saopaulo-01.s3-accelerate.amazonaws.com
S3 Transfer Acceleration Pricing
Starting at $0.04/GB transferred (+ usual bandwidth charges). Up to $0.08/GB in some regions
Pay only for what you use
Accelerated performance or no charge
Compare to hardware, per-GB or licenses