may 29 th, 2003 curtis anderson sivaprasad padisetty
TRANSCRIPT
May 29th, 2003
Curtis Anderson
Sivaprasad Padisetty
Microsoft Confidential
Overview of Presentation
Part I – Software Development and Automation• The Project Cycle• Benefits of Automation• Effective Automation
Part II – Test Distribution and Development• Test Distribution Systems• Test Development • Summary
Software Development and Automation
Curtis Anderson
Microsoft Confidential
The Project CycleTest Investment in Automation
• The investment in test automation starts early• Focus on adding testability to spec/design phase• Investment usually peaks as Development slows feature coding
Planning (M0)
Major MilestonePhase M1,2…
Release Phase
Time
TestingInvestment
TestingPhase
Microsoft Confidential
The Project Cycle Automation Payback
Planning (M0)
Major MilestonePhase M1,2…
Release Phase
Time
AutomationPayback
TestingPhase
• Return on Investment is near zero early• Largest gains at project end game and sustaining engineering
Microsoft Confidential
Benefits of AutomationWhat does Automation Bring to the Project?
•Consistent Test Execution•Stability Guarantees•Reliability Metrics•Reproduce-able Quality Assessment
Microsoft Confidential
Advantages of AutomationWhy Automate ?
• Machines instead of People• Run existing tests automatically on a new build
or version• Exact same test happens every time
• Do More Testing• Simplistic, repetitive (BORING) tests are run for
free. People spend time looking only at results• People are free to ad hoc and search for bugs
Microsoft Confidential
Advantages (continued)• Automation can perform costly, hard, or impossible to
execute tests• Perf, Stress, Scalability, Config, OM/API, Matrix
• Consistency and Reproducibility• Performance, Regression
• Reusability• How many File Open drivers are there?• Logging, etc.
• Ensure a level of stability before a human spends time• Keeps the ‘patient healthy’• Easy to hand off (external teams & sustaining engineering)
Microsoft Confidential
Risks and Problems• Over-reliance on automation results• Its easy to generate lots of results that creates too
much noise• Maintenance• Exact same test happens every time• False failures and false positives• Skill set to produce• Re-inventing the wheel
Time Management is Key!
Microsoft Confidential
Test Categorization
• Unit Test• Interoperability• Localizability• Interoperability• Performance• Reliability• Scalability• Security• Globalization
• Checkin Test• Build Verification• Acceptance• Test Pass/Comprehensive• Regression• DogFood/Deployment• System Integration• Code Coverage
What does it do vs. When does it run
Categorizing tests helps target testing and keep things focused.
Microsoft Confidential
Make Your Automation Effective
• Time Management is key• Cost of authoring• Re-use everything • Write solid automation• Document well• Base all automation on test cases • Be willing to punt if it’s not working• Run the right automation at the right time• Roll up the right metrics to your project
Microsoft Confidential
Test Distribution System OverviewWhy do we need a system?• Standard way to execute tests• Automate the execution• Standard reporting and history
What does it do?• Repository for managing automation• Test distribution• Report Results• Track & manage failures
Test Distribution and Development
Sivaprasad Padisetty
Microsoft Confidential
Test Distribution System Components
• User Interface• Middle Tier• Database• Machine Management/Controller• Client Agent• Test Harness• Machine Imaging and other common tasks• Results Viewer• Email
Microsoft Confidential
Test Distribution System Features/Requirements of a good system
• User Friendly, Modular, Flexible• Integrated Automated/Manual test support• Distributes execution – lab management• Multi stage, multi machine• Offline execution and Private network support• Test Version Control• Object Model• Queuing and Resource management• Hardware and Software tracking• Configuration management & Matrix support• Deployment/Image management• Results reporting
Microsoft Confidential
Test Development
• Abstract from code as much as possible – the less code that changes to add a test case the better
• Separate infrastructure code from test code• Re-use as much code as possible• Modeling and Data Driven tests help scale
Microsoft Confidential
Test DevelopmentGood Authoring is:
• Focused• Consistent• Accurate & Reproducible• Portable, Easily Leveraged• Impervious to Feature Changes
• Feature churn• Language, even application, user, etc.
Cost / Benefit is the ONLY measurement (Don’t forget sustain engineering)
Microsoft Confidential
Test Development Writing Automation is Software Development
• Log-Log-Log-Log-Log everything• Plan & Design • Develop coding standards• Write maintainable code• Document you code and your test cases• Peer-reviews before check in• Use a version control system (Source Depot or VSS)• Test your test.• Log-Log-Log-Log-Log everything
This code will last as long as the product code!
Microsoft Confidential
Test DevelopmentDevelop Configuration Agnostic tests
No hard coded anything!• links like files in the definition• No dependency on a specific machine name• No dependency on a specific lab• Should be package-able.• Configuration changes & dependencies should be
clearly defined.
Microsoft Confidential
Reporting and Metrics
• Good logging saves time• Individual Log files from Test Execution• Roll up summary reports for all tests• Available through system UI and email• Project roll up• Metrics help with project status
Microsoft Confidential
Summary
• Test Automation is a competitive advantage for MS• Lack of Automation hurts innovation in the product• Test Development is Software Development• Many resources and experts exist across the
company
Microsoft Confidential
Test Automation ResourcesAutomated Testing Handbook – Linda G. HayesSoftware Test Automation, Efffective Use of Test Execution Tools, Mark Fewster and Dorothy
Graham
Internal• Windows (WTT & WCTI)
• http://wttweb, http://team/sites/wtt, http://winwebtest/wcti and http://wttforum• Test Categoization doc link: http://teams/sites/wcti/stc/default.aspx• Join Partner DL wttpart, to participate in Shared Development community.
• Office • http://office and http://oasysnet• Automation Strategy Doc -
http://officenet/Teams/DnR/DnR%20Test/Specs/Office%20Automation%20Strategy.htm
• Office automation contacts – mailto:offauto• DevDiv
• http://devdiv and http://maddog/ • MadDog forums - http://maddogforum/forums/• MadDog General Docs -
http://maddog/MadDog%20General%20Docs/Forms/AllItems.htm
Questions?