How To Performance Test Your Apps to

Scale

  Vydianath Iyer Jaswinder Rattanpal

Vydianath Iyer Principal Technical Evangelist Salesforce.com

viyer@salesforce.com /in/vydianath

Jaswdiner Rattanpal Associate Technical Evangelist Salesforce.com jrattanpal@salesforce.com /in/jrattanpal

2

Safe harbor statement under the Private Securities Litigation Reform Act of 1995:

This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.

The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.

Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

Safe Harbor

3

Agenda

1

2

3

4

5

Scale Testing - What, Why & How

How to get a Large Data Volume (LDV) Sandbox Org

Tools and Techniques

Demo

Partner Resources & Related Offerings

6 Q & A

4

What is Scale Testing ?

5

What is Scale Testing (Salesforce) ?

Testing for Large Data Volumes (LDV) Testing for Large Concurrent User Volumes

6

Test for realistic customer sizes   Large Data Loads

  Large User Loads (Concurrent)

API

Platform Stress Testing

Platform Capacity Testing

Network Latency Testing

Testing 3rd Party Integrations

Server Availability/Failover Test

Web

IN SCOPE OUT OF SCOPE

What’s the scope?

7

Why Test ?

8

Increasing application complexity

Increased customer expectations

Minimize downtimes and disruption

Competitive Differentiator

User productivity & Satisfaction

Better ROI

Happy Customers = More Business.

Why Test for Performance ?

 

9

Where do we start ?

10

https://goo.gl/2Rq8Yf

Fill up the Data & API Consumption spreadsheet

Rules of thumb Do significant objects have more than a few million records ?

Do you anticipate a thousand concurrent users or more ? Seek your TE’s advise: Help determine the scope of required testing

Help with tools, best practices and processes

To begin with...

11

High Level Steps - Test in your Developer Org

1 Single User Test in a Developer Org Pre-Requisite

Why is step 1 a pre-requisite ?

●  Is your application designed well ?

●  Does it follow Salesforce coding best practices ?

●  Triggers Bulkified, Selective Queries, View State minimized ?

Max 10 concurrent users in a Developer Org

Prepare

2

12

3

4

5

Single User Test

Performance/Load Testing - 100s of concurrent users

Test with 10 Concurrent Users

Critical Steps

Know the rules

High Level Steps - Test in a Large Data Volume (LDV) Org

13

•  Open up a case 2 weeks before start of testing

•  Mimic normal realistic “thinking time”

•  Concurrent user test loads should be realistic

•  Salesforce does not support stress testing

•  Test only against sandbox instances

TESTING LOAD EXAMPLE:

5K users * 5 processes/ hour =  25 K processes /hour

1 business process = 3 transactions 25K processes/hour = 75K transactions/hour

Equates to 20 transactions per second (TPS)

Average load< 40 TPS Moderately heavy load 40-100 TPS Extremely Heavy load> 100 TPS

Salesforce will not provide back-end logs or results

Step 4 - Rules & Requirements

14

 Number of Atomic database actions* performed by a certain entity per second

Page DB Save DB Save Trigger WF update DB Save

JS Query

JS

Page

DML

Single Transaction

Transaction 1

Transaction 2

What is a Transaction (Force.com context) ?

 * Atomic action = guaranteed isolation from concurrent processes

15

How to get a LDV Org ?

16

LDV Org Options

17

  Less than 100 GB { <50 million records }

  Max 2 Orgs per calendar year

  Max 60 days

  Open up a case

  Provisioned in 14 biz days

  FREE !!!

  Greater than 100 GB { >50 million records }

  Unlimited number

  Renewed Yearly (Can be cancelled/reduced)

  Open up a case, Contact PAM (if you have)

  Provisioned in 14-21 biz days

  $96/GB/YEAR

FREE LDV ORG PAID LDV ORG

Tools & Techniques

18

Grinder Gatling Tsung JMeter **

OS Any Any Linux/Unix Any

GUI Console Only Recorder Only No Full

Test Recorder TCP (including HTTP) HTTP HTTP, Postgres HTTP

Test Language Python, Clojure Scala XML XML

Extension Language Python, Clojure

Scala Erlang Java, Beanshell, Javascript, Jexl

Load Reports Console HTML HTML CSV, XML, Embedded Tables, Graphs, Plugins

Protocols

HTTP SOAP  JDBC POP3 SMTP  LDAP JMS

HTTP JDBC JMS

HTTP  WebDAV Postgres MySQL XMPP

 WebSocket AMQP MQTT LDAP

HTTP FTP JDBC SOAP LDAP TCP JMS SMTP POP3 IMAP

Host Monitoring No No No Yes with PerfMon plugin

Miscellaneous Python knowledge required, reports are basic

Limited support for protocols, Scala knowledge required Supported only on Linux Reports are basic

Performance Test Tools Matrix (free open source) Tool

Feature

** All sample scripts in the partner community chatter group are based on JMeter 19

20

A Few Performance Analysis tools Lightning Inspector Plugin (Chrome) •  Google Chrome Dev Tools extension •  Navigate component tree, inspect components and the DOM elements •  Monitor and Modify server responses which help in debugging server interactions •  Inspect component attributes, and profile component performance. •  Understand the sequence of event firing and handling Apex Timeline •  SOQL queries used in the business transaction along with timings •  DMLs and objects and the cumulative time taken to update/insert/upsert the object •  A visual representation of the overall flow of events •  A JSON representation of the debug log for easier readability. •  Use the tooling API and automatically display the logs in your org Web Page Test •  Open source tool developed and supported by Google •  Used to test a website performance •  Perform a visual comparison of urls and also against industry pages

 

Some Other Tools ..

Gatling – Load Testing

Test Monitoring

Test Reporting

Event Log Analysis

21

22

A Few Free Bulk Data Load Tools Bulk Data Load Tool

•  Reference architecture for creating mock data for Salesforce orgs using Ruby on Rails.

•  Generates mock records and associations for any S object 

•  Uses native Rails functionality to generate mock data

•  Demo: https://partners.salesforce.com/0693A000005q6XR

Salesforce Foundation data creation

•  Test data generation and deployment ETL jobs

Demo Data/Metadata creator  •  Heroku tool that uses Bulk API and MDAPI to create Data and Metadata

JMeter Caveats - LEX

23

Demo

24

Analyzing Results

25

26

27

 

BlazeMeter

29

30

Want to know more …

ISV Partner Resources Partner Community Page - http://p.force.com/scaletesting JMeter Scripts (Lightning and Classic) Free Bulk Data Load Tools Demo Orgs and Videos LDV Org request process Best Practices for Performance Optimization - https://goo.gl/fUk7SE Partner Community Chatter Group - https://goo.gl/HPUlxX

Talk to your TE/PAM if you are an existing Partner

Navigate to the Partner Community Chatter Group

http://p.force.com/scaletesting 31

Performance Hammer

32

https://goo.gl/8NLNPr

Performance Hammer in a Nutshell ..

•  Point in Time Performance Testing between Salesforce Releases

•  Prevent Performance Regression

Partner Release

Salesforce Release

•  Same version of package deployed in 2 different release sandboxes

•  Partner provides JMeter Scripts to run tests

•  Salesforce provides results (and stores summary of results)

33

FREE for App

Innovation Partners

Parting thoughts…

34

Key to Success

◆  Performance Starts with Design

◆  Implement Best Practices Early

◆  Optimize Single User Transactions

◆  Loop back Customer Behavior patterns

◆  Collaborate with your Salesforce Team

Incorporate Performance Testing in your App Life Cycle

35

Q & A

36