1 a day in the life of a tester irinel crivat irinel crivat sdet technical lead asp.net
Post on 30-Dec-2015
218 Views
Preview:
TRANSCRIPT
1
A Day in the Life of a TesterA Day in the Life of a Tester
Irinel CrivatIrinel Crivat
SDET Technical LeadSDET Technical Lead
ASP.NETASP.NET
http://www.asp.nethttp://www.asp.net
2
OverviewOverview
The purpose of testing a program is to The purpose of testing a program is to find problems in itfind problems in it
What a tester does?What a tester does? Design test case (automated & manual)Design test case (automated & manual) Find problems is the core of a tester workFind problems is the core of a tester work Write problem reportsWrite problem reports
3
What are we testing?What are we testing?Writing dynamic, high-performance Web applications has never been easier
ASP.NET helps you deliver real world Web
applications in record time.
ASP.NET combines developer productivity with performance, reliability, and deployment.
4
What are we testing?What are we testing?
Developer Productivity Easy Programming Model Flexible Language Options Flexible Language Options Great Tool SupportGreat Tool Support
Rich Class FrameworkRich Class Framework
Enhanced Reliability Memory Leak, Memory Leak,
DeadLock and Crash DeadLock and Crash Protection Protection
Easy Deployment
"No touch" application "No touch" application deploymentdeployment
Dynamic update of running Dynamic update of running applicationapplication
Easy Migration PathEasy Migration Path
New Application Models XML Web ServicesXML Web Services Mobile Web Device SupportMobile Web Device Support
Improved Performance and Scalability
Compiled execution Rich output cachingRich output caching Web-Farm Session StateWeb-Farm Session State
5
Feature CrewsFeature Crews
Feature CrewFeature Crew
““A small A small self-organizingself-organizing, , cross-disciplinecross-discipline group that group that works together to works together to completelycompletely deliver a discrete deliver a discrete Feature”.Feature”.
Crew has flexibility to decide how to work, how to meet Crew has flexibility to decide how to work, how to meet
quality barquality bar Feature: Independently testable functionality taking 1-6 Feature: Independently testable functionality taking 1-6
weeks of elapsed time to completeweeks of elapsed time to complete Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as
appropriate appropriate No remaining work is left to do on a feature when it gets No remaining work is left to do on a feature when it gets
checked inchecked in interim builds of the product are higher qualityinterim builds of the product are higher quality Tasks are done earlier in the development cycleTasks are done earlier in the development cycle
6
Feature Crew TenetsFeature Crew Tenets
Small, agileSmall, agile work units – the smaller the work units – the smaller the betterbetter
Dedicated Dedicated resources to reduce context resources to reduce context switchingswitching
AutonomyAutonomy and decision making pushed to and decision making pushed to lowest levels – you decide how to work, who lowest levels – you decide how to work, who does whatdoes what
All members “All members “in it togetherin it together” – begin and end ” – begin and end togethertogether
Feature Crew continues until all work is Feature Crew continues until all work is “done” - feature complete, no bugs, tests in “done” - feature complete, no bugs, tests in place, etc.place, etc.
7
Key ConceptsKey Concepts
Quality GatesA set of quality metrics and deliverables that must accompany A set of quality metrics and deliverables that must accompany a feature when it is first submitted to the producta feature when it is first submitted to the product. .
FeatureFeatureA discrete piece of independently testable functionality that A discrete piece of independently testable functionality that integrates at the same time into the PU Branchintegrates at the same time into the PU Branch..
Feature BranchFeature BranchA light-weight source branch that is dedicated to the A light-weight source branch that is dedicated to the development of a single feature. Goal is to allow Feature Crew development of a single feature. Goal is to allow Feature Crew to fully stabilize in isolation before integratingto fully stabilize in isolation before integrating..
SCRUMA project management methodology that involves regular A project management methodology that involves regular check-in meetings to map progresscheck-in meetings to map progress..
8
What do Feature Crews What do Feature Crews offer?offer? Increased EfficiencyIncreased Efficiency
More parallel involvement of all disciplines (better Dev and Test More parallel involvement of all disciplines (better Dev and Test integration)integration)
Less context switching in the core phase of building a productLess context switching in the core phase of building a product Design issues and feature bugs are found earlier when they are Design issues and feature bugs are found earlier when they are
cheaper to fixcheaper to fix More effective and regular communication with cross discipline More effective and regular communication with cross discipline
counterpartscounterparts Fewer intra-version breaking changesFewer intra-version breaking changes
Enhanced Agility and PredictabilityEnhanced Agility and Predictability More honest schedulesMore honest schedules More consistent definition of “done” with quality gatesMore consistent definition of “done” with quality gates More confidence in release datesMore confidence in release dates Spend more time building and less time stabilizingSpend more time building and less time stabilizing
Improved QualityImproved Quality Higher quality new featuresHigher quality new features More stable builds to allow for better customer feedbackMore stable builds to allow for better customer feedback
9
Feature Crew Life CycleFeature Crew Life Cycle
10
De
sig
n P
ha
se
Te
sting
Co
din
g
Checkpoint 2Status Review
Feature complete,
ready to integrate
Dev Test PM UE
Discipline ActivitiesDiscipline ActivitiesIm
ple
me
nta
tion
Ph
as
eIn
teg
ratio
n P
ha
se
Feature design, dev doc,
prototyping
Coding,bug fixing
Unit tests, bug fixing
Finalizing quality gates,
testing, bug fixing
Feature design, test plans,
Test libraries,automation
Testing
Finalizing quality gates,
testing
Feature design, functional spec
Threat model, bug triage,
Feature DCRs,Bug triage,
writing samples, quickstarts
Finalizing quality gates,
testing
Feature design, content plans
Writing content
Finalizing quality gates
RI
Checkpoint 1Design Review
11
Responsibility AreasResponsibility Areas
PMPM Lead the feature crew Status tracking/reporting FC schedule
Functional specs Bug Triage
Write samples, quickstarts App BuildingReview UE docs Test features (scenario testing)
DevDev Design docs Code features, F1 Code unit tests
Fix bugs Test features Review UE docs
TestTest Test plan Automation tests
Test features Review UE docs
UEUE Content plan Write documentation Write samples, quickstarts
Review FC plans, designs Review UI strings Test features (scenario testing)
Architects Architects Review FC plans Review dev designs
Review code
Loc, UX, Loc, UX, Fundamentals, Fundamentals, PartnersPartners
Review FC plans, specsReview FC plans, specs Test features as appropriateTest features as appropriate
Leads, MgmtLeads, Mgmt Review FC plans, specs, statusReview FC plans, specs, status Manage change, peopleManage change, people
Fea
ture
Cre
w M
emb
ers
12
SCRUMSCRUM
Agile programming methodologyAgile programming methodology Small, dedicated groupsSmall, dedicated groups
Daily sync up meetings among crew membersDaily sync up meetings among crew members Anyone can attend, but only crew members can talkAnyone can attend, but only crew members can talk Led by a Scrum Master who tracks progress and drives Led by a Scrum Master who tracks progress and drives
problem solvingproblem solving
Ideally – short ~10 minute meeting where crew Ideally – short ~10 minute meeting where crew members answer only three questions:members answer only three questions: What did you do since we last met?What did you do since we last met? What are you doing next?What are you doing next? Are you blocked? Yes/NoAre you blocked? Yes/No
Problem solving and discussion happens after the Problem solving and discussion happens after the SCRUM meetingSCRUM meeting
13
Phase-by-Phase BreakdownPhase-by-Phase Breakdown
FC KickoffFC Kickoff Design PhaseDesign Phase Checkpoint #1 – Design ReviewCheckpoint #1 – Design Review Implementation PhaseImplementation Phase Checkpoint #2 – Progress ReviewCheckpoint #2 – Progress Review Integration PhaseIntegration Phase Feature Complete/Integration VerificationFeature Complete/Integration Verification
14
FC Design PhaseFC Design Phase
ActivitiesActivities – agreeing what to do, how to do it – agreeing what to do, how to do it Create functional specs, dev design plans, test plans, UE plansCreate functional specs, dev design plans, test plans, UE plans
All disciplines document plans from their perspective in parallelAll disciplines document plans from their perspective in parallel Enumerate Quality Gates, Dependencies, FundamentalsEnumerate Quality Gates, Dependencies, Fundamentals
Build into FC plans, decide when/who/howBuild into FC plans, decide when/who/how Identify risks, issues with anyIdentify risks, issues with any Schedule Quality Gate requirementsSchedule Quality Gate requirements
Updated costing of features, updates to schedule if neededUpdated costing of features, updates to schedule if needed Define Checkpoint process for FC (depends on feature complexity)Define Checkpoint process for FC (depends on feature complexity)
DeliverablesDeliverables functional spec, dev design, test plans, UE planfunctional spec, dev design, test plans, UE plan FC Schedule solidified, Work items identified, Feature list updated with appropriate FC Schedule solidified, Work items identified, Feature list updated with appropriate
infoinfo Feature Branch created, builds setupFeature Branch created, builds setup Tools in placeTools in place
How it worksHow it works FC works together to define designs, all participate in all aspects of design, FC works together to define designs, all participate in all aspects of design,
iterative brainstorming meetingsiterative brainstorming meetings Interactive, iterative discussions between PM, Dev, Test, UE Interactive, iterative discussions between PM, Dev, Test, UE Design phase completion triggers Checkpoint #1Design phase completion triggers Checkpoint #1
FC decides when/how to schedule checkpoint with mgmt teamFC decides when/how to schedule checkpoint with mgmt team
Design specs
DesignPhase
Functional specs
15
Testing - Design PhaseTesting - Design Phase
Characteristics of a good testCharacteristics of a good test It has a reasonable probability of catching It has a reasonable probability of catching
an erroran error It is not redundantIt is not redundant It’s the best of its breedIt’s the best of its breed It’s neither too simple nor too complexIt’s neither too simple nor too complex It makes program failures obviousIt makes program failures obvious
16
Testing - Design PhaseTesting - Design Phase
Techniques to come up with powerful Techniques to come up with powerful test casestest cases Equivalence class analysisEquivalence class analysis Boundary analysisBoundary analysis Testing state transitionsTesting state transitions Testing race conditions and other time Testing race conditions and other time
dependenciesdependencies Doing error guessingDoing error guessing
17
Test case design Test case design Equivalence class analysisEquivalence class analysis
If you expect the same result from two If you expect the same result from two tests, you consider them equivalenttests, you consider them equivalent
A group of tests forms an equivalence A group of tests forms an equivalence class whenclass when They involve the same input variablesThey involve the same input variables They result in similar operations in the They result in similar operations in the
programprogram They affect the same output variablesThey affect the same output variables None force the program to do error None force the program to do error
handling or all of them dohandling or all of them do
18
Test case design Test case design Boundary analysisBoundary analysis
The boundary values are the biggest, The boundary values are the biggest, smallest, soonest, shortest, loudest, smallest, soonest, shortest, loudest, fastest, ugliest members of the class, fastest, ugliest members of the class, i.e. the most extreme valuesi.e. the most extreme values
Many testers include a mid-range value Many testers include a mid-range value in their boundary tests. This is a good in their boundary tests. This is a good practice.practice.
19
Test case design Test case design Testing state transitions Testing state transitions
Every interactive program moves from one visible Every interactive program moves from one visible state to another. If you do something that changes state to another. If you do something that changes the range of available choices or makes the the range of available choices or makes the program display something different on the screen, program display something different on the screen, you’ve changed the program stateyou’ve changed the program state
Advice:Advice: Test all paths that you think people are particularly likely Test all paths that you think people are particularly likely
to followto follow If you suspect that choices at one menu level or data entry If you suspect that choices at one menu level or data entry
screen can affect the presentation of choices elsewhere, screen can affect the presentation of choices elsewhere, tests the effects of those choices or entriestests the effects of those choices or entries
Try a few random paths through the program. Try a few random paths through the program.
20
FC Implementation FC Implementation PhasePhase ActivitiesActivities – getting it done – getting it done
Coding, creating unit tests,Coding, creating unit tests, Test automationTest automation Testing and bug fixingTesting and bug fixing Quality work: API Reviews, Threat models, etc.Quality work: API Reviews, Threat models, etc. Fundamentals (perf, stress, etc.)Fundamentals (perf, stress, etc.)
DeliverablesDeliverables Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC)Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC) Feature code and unit tests complete Feature code and unit tests complete Automation tests in placeAutomation tests in place Documentation, including tech reviews of content, samples, app buildingDocumentation, including tech reviews of content, samples, app building All FC bugs addressedAll FC bugs addressed
How it worksHow it works Dev and Test work in tandem - iterative, parallel progress made throughoutDev and Test work in tandem - iterative, parallel progress made throughout PM facilitates, tracks/communicates status, drives bug triagePM facilitates, tracks/communicates status, drives bug triage All FC members help out as appropriate for load balancing – no discipline All FC members help out as appropriate for load balancing – no discipline
boundariesboundaries Checkpoint #2 scheduled by FC during Implementation phase when main Checkpoint #2 scheduled by FC during Implementation phase when main
coding is completecoding is complete
Coding
Testing
Bug fixing
ImplementationPhase
21
Testing – Testing – Implementation PhaseImplementation Phase
ActivitiesActivities – getting it done – getting it done Find bugsFind bugs File problem reportFile problem report,, Test automationTest automation Run automated tests and perform analysis on the Run automated tests and perform analysis on the
failuresfailures Quality work: API Reviews, Threat models, etc.Quality work: API Reviews, Threat models, etc. Fundamentals (perf, stress, etc.)Fundamentals (perf, stress, etc.)
DeliverablesDeliverables 70% automation tests in place70% automation tests in place 100% passing rate on the automated suite100% passing rate on the automated suite All problem report (bugs) are verified/closedAll problem report (bugs) are verified/closed
Coding
Testing
Bug fixing
ImplementationPhase
22
Fundamentals DetailsFundamentals Details
Perf and ScalabilityPerf and Scalability Stress and CapacityStress and Capacity Globalization/LocalizationGlobalization/Localization Security Security CompatibilityCompatibility Side-by-SideSide-by-Side AccessibilityAccessibility User ExperienceUser Experience HostingHosting 64-bit64-bit ServicingServicing
23
Problem reportProblem report
A good report isA good report is
You need to track the problem so you You need to track the problem so you must describe it in writing. Otherwise, must describe it in writing. Otherwise, some details (or the whole problem) will some details (or the whole problem) will be forgotten. be forgotten.
You also need a report for testing the fix You also need a report for testing the fix laterlater
24
Problem reportProblem report
A good report is A good report is
Track Problem Reports numerically.Track Problem Reports numerically.Assign a unique number to each report.Assign a unique number to each report. If you use a computerized database, the If you use a computerized database, the
report number will serve as a report number will serve as a key fieldkey field. . This is the one piece of information that This is the one piece of information that
always distinguishes one report from all always distinguishes one report from all the rest.the rest.
25
Problem reportProblem report
A good report isA good report is
simple = not compoundsimple = not compound
Describe only one problem on one reportDescribe only one problem on one report
Why multiple bugs on a single report are a Why multiple bugs on a single report are a problem?problem?
The programer can fix only some of them and The programer can fix only some of them and pass the report back as fixed, even though pass the report back as fixed, even though some bugs have not been fixed. This wastes some bugs have not been fixed. This wastes time and leads to bad feelings. Remaining time and leads to bad feelings. Remaining problems often stay unfixed long time…problems often stay unfixed long time…
26
Problem reportProblem report
A good report is A good report is
You must describe the program’s You must describe the program’s problematic behavior clearly. problematic behavior clearly.
Keep all unnecessary steps out of the Keep all unnecessary steps out of the steps required to reproduce the steps required to reproduce the problem.problem.
27
Problem reportProblem report
A good report is A good report is
Think of the person reading it. Unless you Think of the person reading it. Unless you are reporting a disaster, the programmer are reporting a disaster, the programmer will toss an illegible report onto his/her will toss an illegible report onto his/her pile of things to look at next yearpile of things to look at next year
28
Problem reportProblem report
A good report isA good report is
Nobody likes being told that what they did Nobody likes being told that what they did was wrong. As a tester, that’s what you was wrong. As a tester, that’s what you tell people every day. tell people every day.
Do not express personal judgments in Do not express personal judgments in your reportsyour reports
The programmer and the tester are a team The programmer and the tester are a team that makes the product betterthat makes the product better
29
Problem reportProblem report
A good report isA good report is
How can I make the bug reproducible?How can I make the bug reproducible? You can describe how to get the program into a You can describe how to get the program into a
known state, and from there describe an exact series known state, and from there describe an exact series of stepsof steps
Find the simplest, shortest and most general Find the simplest, shortest and most general conditions that will trigger the bugconditions that will trigger the bug
Find related problemsFind related problems Find alternate paths to the same problemFind alternate paths to the same problem Find the most serious consequencesFind the most serious consequences Look for the critical stepLook for the critical step
30
Test AutomationTest Automation
Automation framework (in house)Automation framework (in house) Building Maintainable Testsuites Building Maintainable Testsuites Automation Results ReportingAutomation Results Reporting Automation auto-analysis toolsAutomation auto-analysis tools Code coverage Code coverage
Keeping test suites reliable Keeping test suites reliable Gauntlet and Code ReviewsGauntlet and Code Reviews
31
FC Integration PhaseFC Integration Phase
Activities Activities – making sure it works– making sure it works integrate from PU branchintegrate from PU branch Run all tests, verify feature works in integrated worldRun all tests, verify feature works in integrated world
Several functional runsSeveral functional runs StressStress PerformancePerformance Code Coverage RunCode Coverage Run
Fix and verify bugs (includes Product bugs)Fix and verify bugs (includes Product bugs) Quality Gates final verificationQuality Gates final verification integrate into PU branch, final verification in PU branchintegrate into PU branch, final verification in PU branch
DeliverablesDeliverables Feature List data updatedFeature List data updated Quality Gates complete, verifiedQuality Gates complete, verified Final integration testing completeFinal integration testing complete
How it worksHow it works FC tests feature with latest integrated code from PU BranchFC tests feature with latest integrated code from PU Branch All FC members verify feature meets requirements and works as expectedAll FC members verify feature meets requirements and works as expected Team members move onto next Feature CrewTeam members move onto next Feature Crew
32
Resources:Resources:
BooksBooks How to Break Software: A Practical Guide to How to Break Software: A Practical Guide to
Testing, Testing, by Whittaker, James A. (Author)by Whittaker, James A. (Author) Lessons Learned in Software Testing, Lessons Learned in Software Testing, by Cem by Cem
Kaner (Author) Kaner (Author) Effective Software Testing: 50 Specific Ways to Effective Software Testing: 50 Specific Ways to
Improve Your TestingImprove Your Testingby Elfriede Dustin (Author)(Author)
33
Comments, Q & AComments, Q & A
top related