Load Testing Templates:Artifacts of NYU’s Load Testing

ImplementationMax Whitney & Erik Froese

NYU/ITS/eServices

Repeatable Load TestingWow, it really is that time consuming.

Max Whitney

NYU/ITS/eServices

max@nyu.edu

Context

Current Environment

• Blackboard Learning System– Fall 2006

• 5,219 sections• 127,899 enrollments• 3,964 instructors & 38,624 students

– Spring 2007• 4,952 sections• 146,889 enrollments• 4,329 instructors & 38,559 students

Under-served Communities

• School of Medicine• Math and Sciences• School of Continuing and Professional

Studies: The Virtual College, 100% online program

• School of Law: LL.M. in Tax, hybrid online and in person program

Long Time ListenerFirst Time Caller

• NYU joined as a Sakai Educational Partner in Summer 2004

• Local pilots– Stern School of Business – Computer Science Department of

Courant Institute of Mathematical Sciences

• Enterprise Pilot: September 2006

Phased PilotLacking the roll out support of commercial software, NYU is rolling out in phases, building the service with each phase:

• Documentation• User Support • Pedagogy Support• Critical System Support:

– Security– Integration– High Availability– Performance Assurance– Disaster Recovery

Snowballing to Production

January 2007 Phase 1Dashboard (My Workspace) Web Content

Announcements Resources

Syllabus Assignments

Roster Default Discussion Board

Course Site Statistics Gradebook

Help Email Archive

23 Course Sites20 Faculty 891 Students

Backup & Disaster Recovery Implemented

Penetration Testing Passed & Security Report Filed

Smaller Summer Sets - 2 Phases

Critical System Support for Phase 3 - September 2007

• Integration with Student Information System• Performance Assurance

Repeatable Load Testing

• Select Software• Select Hardware• Identify Network Requirements• Identify Ancillary System

Requirements• Commit to Long-term Retention

Method for Tests, Results & Configurations

• Describe Test Cases• Identify User and Site

Characteristics • Identify Executive Stakeholders

• Identify Load Characteristics• Implement Test Cases• Create Test Users• Create Test Sites• Create Test Site Content• Test• Store Configuration and Results• Reset • Analyze Results• Report to Executive

Stakeholders

Load Testing Software

• Key questions– Is there a budget?

– Are there reasons to go with a proprietary or an open source tool?

– Does it matter if you can share your test case code?

– Where should it be in the adoption cycle?

– What internal expertise exists?

NYU’s Choice

• The Grinder 3– No budget– Charged with using open source where mature– Others can use the resulting test cases– Neither bleeding edge nor nearing obsolescence– Jython/Python scripting in a Java framework

Artifacts

• Installation and Configuration Cheat Sheet – In progress– Shared

Load Testing Hardware

• Key questions– Buy or borrow?

– Where should machines live?– How many?

• Bleeds into Network Requirements

NYU’s Choices• Console– Borrow– Least secure network consistent with data stored– Long term (near-permanent) cname

• Agents– Borrow– On same subnet as application, on NYU-NET, off NYU-NET,

across VPN, across NYU-DIAL, on NYU-ROAM– Number desired will be derived from test case characteristics– Number used will be negotiated with ‘volunteer’ organizations

• Network Requirements– Remember to handshake with the NOC early

Artifacts

• Derivation of agent machine counts – In progress– Shared

• Machinery– Not shared

Ancillary Systems• Key questions

– What are you really testing?– Are the any non-Sakai systems in the mix?

NYU’s Answers• LDAP– Not testing the speed of our LDAP response– Don’t slow other production systems– Don’t expose real user dataSet up an independent LDAP system with well specified test

accounts and test passwords

• Automated data processes– Future issue– Disable non-core processes for duration of load testing– Schedule specific load tests to learn the impact of batch

processes on user response times

Artifacts

• None– Share test account specifications– Missing something shareable?

Long Term Retention

• Key questions– Retain results?– Retain test scripts?– Retain network and machine specs?– Retain configuration data?– Who is the data steward?– Where to keep it all?

NYU’s Choices

• Retain Network and Machine Specs• Retain Test Scripts• Retain Results• Retain a Record of Versions and Configuration Settings• Without a formal QA department, Dev is the Steward• Subversion repository

Artifacts

• Subversion repository recommendations– Will share own best practices for

tagging/branching svn repository– Missing something shareable?

Itemize Test Cases

• Derived from existing Learning Management Systems in production

• Derived from actual Sakai usage in production

• Theorized for new tools

Blackboard Usage• Statistics information is exported to offline database

schema (bb_bb60_stats)• Query for counts

– Time blocks: pre-semester, first week, midterms, reading week, finals, one-time events

• Analyze events• Translate to Sakai equivalents• Derive relative counts of each activity for each time block

“Administrators have open access to the statistics database to use for analysis and creating reports.” at http://library.blackboard.com/docs/r7/70/en_US/admin/bbas_r7_0_admin/advanced_system_reporting.htm and http://library.blackboard.com/docs/cp/learning_system/release6/administrator/advanced_system_reporting.htm

Ah, Tedium

Artifacts

• Queries• Transliteration from leaf case to Sakai test• Multiplication factor from leaf case to event

– In progress– Shared

Sakai Usage

• In pilot, little valid data• Biggest usage results turned out to reflect

our pen testing

Theoretical Usage

• New tool means no historical data• Key is to identify and document all

assumptions• Update assumptions as data accumulates

Artifacts

• Sakai site statistics queries– In progress– Shared

• Assumptions worksheet– Not quite on the radar yet

Elements of a Test Case

• Name• Number - for correspondence to Grinder test• Human readable description• User type executing test case• Data assumed to exist before the test case starts• Success criteria - only successful tests are

counted in timing statistics• Categorization - login, logout, content read,

content create, discussion read, discussion write

Artifacts

• Test Case Documentation– Not started– Shared

Implement Test Cases

• The Grinder proxy captures clicks of a test case• Programmer effort

– Clean up proxy output– Parameterize user data– Parameterize course site data– Check for success criteria

• Not to be sneezed at• Test the Test Cases

Artifacts

• The Grinder Test Case Scripts– Not started– Shared

Create Test Users and Sites

• Now know user types required and relative counts of each type.• Now know data requirements for each test case, and the course sites

and starting contents required.

• Script the generation of users and sites, and of the data files used by The Grinder agents– LDAP account allocation

– Permissioning within course sites

– Content creation within course sites: seed content files, lorem ipsum

Artifacts

• User creation scripts• Course site creation scripts• Public domain content files

– Not started– Shared

Test

• Set up grinder configuration: counts, ramp up speed, duration

• Install The Grinder on agent machines to speak to common console

• Check in to subversion all documentation, test cases and configuration data

• Take a full back up of the target system

Run the test.

Test

bit anticlimactic really

Artifacts

• Configuration files• Target system characteristics• Raw results

– Not started– Shared

Reset the System

• Restore from the full backup

Results

• Commit the raw results to subversion alongside the tests and configurations used to generate the results

Report

• Report to stakeholders

Repeat Until Done

Test

• Set up grinder configuration: counts, ramp up speed, duration

• Install The Grinder on agent machines to speak to common console

• Check in to subversion all documentation, test cases and configuration data

• Take a full back up of the target system

Run the test.

Reset the System

• Restore from the full backup

Results

• Commit the raw results to subversion alongside the tests and configurations used to generate the results

Report

• Report to stakeholders

Repeat Until Done

Test

• Set up grinder configuration: counts, ramp up speed, duration

• Install The Grinder on agent machines to speak to common console

• Check in to subversion all documentation, test cases and configuration data

• Take a full back up of the target system

Run the test.

Reset the System

• Restore from the full backup

Results

• Commit the raw results to subversion alongside the tests and configurations used to generate the results

Report

• Report to stakeholders

Repeat Until Done

Thank You

and good day