oscon2008 qa leak testing latest slides
Post on 16-Apr-2017
2.109 Views
Preview:
TRANSCRIPT
Firefox 1.5 Beta 1
OSCON 2008 - Fixing Hard Problems Through Iterative QA and Development
Clint Talbert - Carsten BookMozilla Corporation
24.July 2008
Introduction
Process Overview
Leak Testing Overview
Tools
Memory Leak Debug Test Builds
Leak Gauge
Manual Tests (Litmus)
Add-on Testing
Milestones so far
The Approach
Development and Test Development collaborate on tests and tools
Create widely applicable, reproducible tests
Test Tools are treated as first rate products
Implement a reporting structure for results
Collaboration
Developers and Test developers bring different perspectives to the table
Developers concentrate on the deep tools to measure behavior of code (i.e. Ref-Counting Analysis etc)
Test Developers work on harnesses and reporting infrastructure, lowering barrier of entry
Having reliable test tools that are run and developed on a regular basis help bring a sense of testing into the development culture.
Reproducible
Use VM's where you can
Keep tests and tools simple
By taking reliable test tools into impure systems, you can get results that you can trust.
Test Tools Are Products
Test Tools must grow with the product
Test Tools must be simple and maintainable
Test Tools must be developed and QA'd just like the product they are testing
Test Tools should be very easy to run
Test Tools should require only the most basic level of infrastructure possible to run. (But should also fit into a high infrastructure scenario for reporting)
Reporting Results
Use results to further a culture of testing
Use results to understand if the test is reliable, and if so broaden your testing scenarios
Easy to understand results empower community members
Reporting Results
What our results reporting looks like
Leak Testing Overview
Lots of Memory Improvements for Firefox 3:
The XPCOM cycle collector continuously cleans up unused memory. Plus, hundreds of memory leaks are now remedied
Memory Leak Tests after every Code change to make sure we introduce no new Memory Leaks
QA also started with a lot of Tests for Memory Leaks
Mozilla QA runs manual tests using the Litmus Testcase Suite (using Tools such as Debug Builds and Leak Gauge)
Regression Memory Leak Testing of Firefox
Memory Leak Tests of new and popular Addon's
Future: More Automated Tests run by Mozilla QA
Tools
Various Tools for Performance and Leak Testing
QA uses Debug Builds and Leak Gauge
Debug Builds: Debug Builds can be build with Trace-Malloc Support to search for Memory Leak Builds every time up to date build with latest checkins and overview over Leaking Components (but Debug Builds need to be built individually and you need a Development Environment (Xcode, Visual Studio Express, etc) for most platforms)
Leak Gauge: Developed by David Baron - It is designed to assist in detecting what leaks of large object graphs occur during normal browsing activity. The logging can be run during normal browsing without significant overhead. Log taken by setting environment variables in a release build.
Debug Builds
Using Firefox Debug Builds with Trace Malloc enabled for all Plattforms (Windows XP/Vista, Linux and Mac 10.4/10.5)
Trace Malloc allows searching for Memory Leaks
Can provide a Log File Output
Identify Leaking Components, this information helps Developers to debug and fix the Memory Leak !
Leak Gauge
Easy to use Memory Leak Test Tool developed by David Baron
Only Requirement is a environment variable and works on all platforms c:\NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,NodeInfoManagerLeak:5
And set NSPR_LOG_FILE=c:\leak1.log to define a Leak Log
Works on Firefox 2 and Firefox 3+ Release Builds
Leak Log Upload Form on http://mxr.mozilla.org/mozilla/source/tools/footprint/leak-gauge.html for analysis
Upload Form indicates if a Memory Leak was found
Leak Gauge Log Output
When a Memory Leak was found the output of the Log analysis look like this:
Memory Leak
Litmus based Tests
To make sure we have no hidden Memory Leaks in Firefox Mozilla QA was running a FFT (Full Functional Tests) Testrun in Litmus (http://litmus.mozilla.org) with Debug Builds on Linux, Mac and Windows. Memory Leaks were Reported to the Bugzilla DB.
Litmus is a Open Source Test Case Management Tool. Litmus contains manual testcases and a Full Functional Test contains around 750 manual testcases from all Firefox areas.
Add-ons Testing
Addon Memory Leak Testing is a big focus of the QA Memory Leak Testing Work
We test new Extensions uploaded to Addons.mozilla.org and also the Top-Downloaded and Recommend Addon's
Leak Testing is done with Debug Builds and also with Leak Gauge
We provide also a Leak Gauge How-To for Addon Developers and AMO-Editors
In the Future A successful Leak Test is a requirement for new Addons on Addons.mozilla.org
A best practices Document is in progress to give Addon Developers a hand in avoiding Memory Leak
Addon Testing Practice
We use new Profiles to avoid any false-positive Results from other Extensions
We test all aspects of the Extension (Install, Uninstall, Features of the Extension)
Provide Detailed Steps to reproduce so that this Memory Leak is reproducible by Developers and also to be able to verify the fix of this Leak.
In case a Memory Leak was found we file a Bug in the Mozilla Bug Database and inform the Extension Developer
Future Plan: We will use more Automation to do this testing !
Success so far
Successfully identified various types of Leaks in Extensions and Firefox Components also in specific Scenarios beyond automated Tests
Memory Leak Logs from Debug Builds and Leak Gauge Help Developers to identify the cause of the Leak
Memory Leak Testing of Extensions help to maintain the great Firefox 3 Performance
Questions ?
References:
http://wiki.mozilla.org/Performance:Leak_Tools Overview over Performance/Leak Tools
http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ Blog Post from Stuart Parmenter
http://blog.mozilla.com/tomcat/2008/03/21/extension-memory-leak-testing/ Blog Post about the QA Extension Testing
Contact:
Clint Talbert (ctalbert on irc.mozilla.org) ctalbert@mozilla.com
Carsten Book (tomcat on irc.mozilla.org) cbook@mozilla.com
top related