MF AM Tutorial

9/30/2013 8:30:00 AM

"Implementing Crowdsourced

Testing"

Presented by:

Rajini Padmanaban and Mukesh Sharma

QA InfoTech

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Rajini Padmanaban

QA InfoTech

As Sr. Director of Engagement at QA InfoTech, Rajini Padmanaban leads the engagement and

relationship management for some of QA InfoTech's largest and most strategic accounts. Rajini

has more than twelve years of professional experience, primarily in the software quality

assurance area.

Mukesh Sharma

QA InfoTech

As founder and CEO of QA InfoTech Worldwide, Mukesh Sharma is responsible for the

company's vision and leads the organization's worldwide operations, marketing, sales, and

development efforts. Founding QA InfoTech with a vision to provide unbiased testing solutions,

Mukesh has grown the organization to five Centers of Excellence with more than 650

employees. He began his technology career with DCM Data Systems, and then worked at IBM

Corporation, Quark Inc., Gale Group, and Adobe Systems in software engineering and testing

disciplines.

Corp04 1

Your Software Testing Partner We help you build better software

Implementing Crowd Sourced Testing

Mukesh Sharma, Rajini Padmanaban

Agenda (1 of 2)

Slide 2

Topic Time (in minutes)

A Peek into Software Quality 10

Crowd Sourced Testing - Defined 10

Understanding Varied forms for Crowd Sourcing 15

Let’s be the Crowd – Exercise Time 20

Crowd Sourced Testing Relevance in Current Scenario

10

Limitations of Crowd Sourced Testing 15

Corp04 2

Agenda (2 of 2)

Slide 3

Topic Time (in minutes)

Practices for a Successful Crowd Sourced Test Effort

30

Know What Not to Crowd Source 15

Getting Stake Holder Buy-in 15

Case Study & Examples 30

Myths and Facts 15

Conclusion and Q&A 15

Requirements

A device to connect to the internet for one of

our exercises (laptop, smart phones, tablets…)

Slide 4

Corp04 3

Test is responsible for

Quality Focused on creating a quality deliverable

Ensure we don’t sacrifice quality for the sake of schedule

Empower the rest of the team to partake in improving quality

A Peek into Software Quality

Slide 5

Defining Quality

• ISO 9000: Degree to which a set of inherent characteristics fulfill

requirements

• Six Sigma: Number of defects per million opportunities

• Philip B. Crosby: Conformance to requirements

• Joseph M. Juran: Fitness for use by the customer

• Gerald M. Weinberg: Value to some person

• Robert Pirsig: The result of care

• American Society for Quality: A subjective term for which each person

has his or her own definition. In technical usage, quality can have two

meanings:

a. The characteristics of a product or service that bear on its ability to

satisfy stated or implied needs

b. A product or service free of deficiencies

Slide 6

Corp04 4

Software Development

Slide 7

Attacking the Inverse Exponential

Testability

Fault Injection

Stress Testing

End-to-End

Exploratory

Code Inspection

Leverage the Beta Test Crowd

Slide 8

Corp04 5

Portfolio Selection Theory

Diversification in

investing tells us

that risk lessens as

the number of

investments in the

portfolio increases

10%10% 10%

10%

10%

10%

10%

10%10%

10%

Slide 9

Audience Interaction Time…

What Does Quality Mean to you ?

Slide 10

Corp04 6

What is Crowd Sourced Testing?

Dailycrowdsource.com

Crowd sourcing is the process of getting work, usually online, from a crowd of people. The word is a

combination of the words 'crowd' and 'outsourcing'. The idea is to take work and outsource it to a crowd

of workers

CrowdSourcing.org (similar definition on Wikipedia)

Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that

exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards

software quality assurance and control. With this new form of software testing, the product is put to test

under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all,

bug-free

Crowd Source Testing.com

Crowd sourcing your software testing consists of delegating onto a number of internet users the task of

testing your web or software project while in development to ensure that it contains no defects, referred

to as bugs

Slide 11

I prefer the word engaging

rather than delegating

What is Crowd Sourced Testing?

…….so what really is Crowd Sourced Testing?

• Is it all about a pool of testers leveraged and paid

per valid bug? No, not just that…

• Think: • Sourcing relevant people from within your company

• Across disciplines and levels

• Sourcing end users from various disciplines – e.g.

teachers, students, nurses, bankers

• Partnerships with universities, organizations leveraging

domain knowledge

• In essence, think of the community at large to test your

product

Slide 12

Corp04 7

What is Crowd Sourced Testing?

Slide 13

Services Companies

Product Companies

External

Crowd Sourcing

Internal

Crowd Sourcing

Technologies Domains Company Scale

Understand Factors that Motivate the Crowd

Portfolio Selection Theory

Crowd Sourced Testing:

Portfolio Selection Theory

applied in QA

10%10% 10%

10%

10%

10%

10%

10%10%

10%

Slide 14

Corp04 8

A Practical Solution to Leverage

Slide 15

Set # 1

• Crowd Sourcing:

• Is not restricted to any single company, technology,

domain

• Has a much larger yet simpler meaning than what it is

often portrayed to be

Slide 16

Corp04 9

Types of Crowd Sourcing - Explained

Slide 17

Crowd Creation

Crowd Voting

Crowd Wisdom

Crowd Funding

Types of Crowd Sourcing - Explained

• Crowd Creation

• Invite crowd to create subject content

• Common usage areas: software

development, translations, photos repository

• Content usage by crowd or by organizations

• Typical crowd motivators – Money, fun,

community involvement, brand loyalty

• Well known examples – Linux, iStockPhoto,

99 designs

Slide 18

Corp04 10

Types of Crowd Sourcing - Explained

Slide 19

Types of Crowd Sourcing - Explained

• Crowd Voting

• Leverage crowd’s judgement to organize,

filter, stack rank content

• Common usage areas: retail, media, simple

yet powerful decisions across domains

• Results used by an organization

• Typical crowd motivators – Fun, community

involvement, brand loyalty

• Well known examples – American Idol,

Threadless.com

Most popular of crowd sourced versions

(1:10:89 rule) 1% create

10% vote and rate

89% consume -Jeff Howe, Author of CrowdSourcing

Slide 20

Corp04 11

Types of Crowd Sourcing - Explained

Slide 21

Types of Crowd Sourcing - Explained

• Crowd Wisdom

• Harnesses crowd’s knowledge to solve

problems, predict future outcomes

• Common usage areas: reality shows, quality

assurance, exchanges markets

• Results used by an organization, crowd

• Typical crowd motivators – Money, fun,

product transparency, brand loyalty

• Well known examples – Who wants to be a

Millionaire

Slide 22

Corp04 12

Types of Crowd Sourcing - Explained

Did You Know?

The crowd’s answer in “Who Wants to be a

Millionaire” was right 91% of the time compared to

“Ask an Expert”, which was right 65% of the time

Source: The Wisdom of

Crowds, James Surowiecki

Slide 23

Types of Crowd Sourcing - Explained

• Crowd Funding

• Leverages crowd to finance individuals or

groups that might otherwise be denied

credit or opportunity

• Common usage areas: ideas in developing

nations, educational domain

• Results have far reaching impact

• Typical crowd motivators – Money, social

causes / community involvement

• Several interesting examples at: http://www.alumnifutures.com/2012/07/crowdsourced.html

Slide 24

Corp04 13

Set # 2

• Crowd Sourcing Options are:

• plentiful

• available for a diverse set of community needs – both

commercial and not-for-profit

• to be customized based on results needed and the

crowd motivating factors

Slide 25

Exercise Time – Let’s Crowd Source

Slide 26

Who has won the most oscars? 1.Walt Disney

2.Elizabeth Taylor 3.Meryl Streep

4.Jack Nicholson

Corp04 14

Exercise Time – Let’s Crowd Source

Slide 27

On which national flag is there an eagle and

a snake? 1.China

2.Mexico 3.Greece 4.Spain

Exercise Time – Let’s Crowd Source

Slide 28

Which American state produces most

potatoes? 1. Oregon

2. California 3.Idaho

4.Washington

Corp04 15

Did You Know?

The crowd’s answer in “Who Wants to be a

Millionaire” was right 91% of the time compared to

“Ask an Expert”, which was right 65% of the time

Source: The Wisdom of

Crowds, James Surowiecki

Set # 3

Collective wisdom

of the crowd often

surpasses that of

an expert

• Please take about 10 minutes to test

www.amazon.com from usability and

accessibility standpoints

• Need tools - screen readers, magnifiers? • http://en.wikipedia.org/wiki/List_of_screen_readers

• http://mediaaccess.org.au/digital-technology/assistive-

tech/screen-magnifiers/ - e.g. Magnifier on Windows, Zoom

for Mac

Slide 30

Exercise Time – Let’s Crowd Source

Corp04 16

• Discuss test results / feedback –

• Goal of our study: demonstrating richness

and diversity of crowd’s feedback as end

users

• Feedback from our visually challenged crowd

Slide 31

Exercise Time – Let’s Crowd Source

• Shrinking release cycles

• Close scrutiny on overall spend

• Collective ownership of quality

• Need to: • Focus on product domain knowledge; not just

disciplinary knowledge

• Understand competing products

• Creatively emulate end user scenarios

• Mimic user environments via lab and simulations

• Consider global product distribution effects

Slide 32

Product Quality Scenario – as it stands today

Corp04 17

A Practical Solution to Leverage

• Distribute quality effort

• Enhance productivity through global solutions

• Pool in end users into quality implementation

• Use live environments to test

• Analyze potential partnerships for SME

• Flexible, selective and cost effective testing

.........Bring in the Crowd

Slide 33

When does Crowd Sourced Testing

Succeed?

• Diversity of knowledge, background,

experience

• Independence in testing process

• Wide spread domain background for product

requiring SMEs

• End user scenarios and environments difficult

to simulate in-house

• Work aligns with factors that motivate crowd

Slide 34

Corp04 18

Inherent Challenges

Slide 35

• Random test efforts do not fit into quality strategy

• Choosing, sustaining a crowd sourced team

• Ongoing motivation, floating crowd

• Keeping everyone in sync on product dynamics

• Communication challenges

• Management overhead including logistics

• Stakeholder buy in

• Securing product IP before release

Any Challenges from your experience that

you would like to share?

Perceived Limitation

Slide 36

Quality of product adversely impacted by an

amateur crowd

- Solidify your implementation plans

- Use instrumentation wherever possible

- Refer case studies – understand successes

and failures

Corp04 19

Set # 4

• Current product scenario demands creative solutions to

test within existing constraints

• Crowd Sourcing is not a no-brainer solution to all problems

• Not even a stand-alone solution in most cases

• Understand its strengths and challenges in customizing

it to your needs

• Understand “What, When and How to Crowd Source in

Testing”

Slide 37

Practices for Successful Crowd Testing

Customize your

practices mindful of your constraints

Slide 38

Corp04 20

Slide 39

• When to Crowd Test:

• Product works reasonably well E2E

• Ready to incorporate crowd’s feedback

• No time or resources for formal testing….try to

avoid this situation

• Source content files are ready

• Ongoing feedback from a chosen SME team

at specific stages

“What, When and How” of Crowd Testing

Slide 40

Examples

Started off with informal testing due to lack of time

Ongoing MVP programs at Microsoft

Product ready to be tested E2E and feedback incorporated

Corp04 21

Slide 41

• What to Crowd Test:

• User facing features

• Areas where external team feedback is

important – e.g. design, feature set, performance

• Specialized areas of test: • localization (context based verification)

• performance

• compatibility, devices testing

• Content testing – valuable SME knowledge

• Align crowd’s focus areas into test strategy

• Know what not to crowd source

“What, When and How” of Crowd Testing

Slide 42

Examples

Exam Grading

Content Testing using SME / Localization

Corp04 22

Slide 43

• How to Crowd Test:

• Pick the right areas, team and time • Minimize duplication, overhead of sifting through

known issues

• Clear internal ownership: • Communication, technical query resolution

• Prompt follow up and responses

• Team up-to-date on product changes

• Use of collaborative tools

• Think about interactions amongst testers

“What, When and How” of Crowd Testing

Slide 44

• How to Crowd Test:

• Leverage cloud, VPC for ease, secured access

• Identify crowd motivators

• Use management theories – Maslow’s

hierarchy of needs?

• Identify tasks that align with crowd motivators

• Work on stakeholders buy-in

“What, When and How” of Crowd Testing

Self Actualization – Pursue Inner Talent, Creativity, Fulfillment

Self Esteem – Achievement, Mastery,

Recognition, Respect

Belonging – Friends, Family, Spouse, Lover

Safety – Security, Stability, Freedom from Fear

Psychological – Food, Water, Shelter, Warmth

Corp04 23

Set # 5

• Understand when a crowd test effort succeeds:

• Diversity of knowledge, background, experience

• Independence in the testing process

• Wide spread domain background for product

requiring SMEs

• End user scenarios and environments difficult to

simulate in-house

• Work aligns with factors that motivate the crowd

Any Examples, Best Practices from your experience that you would like to share?

Slide 45

Slide 46

• Features with moving pieces, demanding close

collaboration

• Sensitive IP

• Environment specific complex testing

• Tasks requiring immediate and regular turnaround

- BVTs

• Core testing activities • Test automation, TDD scripts

• Regression testing

• First round of performance, security, integration testing

• Mundane testing tasks that don’t need diversity

Know What Not to Crowd Test

Corp04 24

Slide 47

• Simple performance tests such as PLTs – use tools

• Understand pros and cons specific to your scenario – study by University of Texas, Austin* on Crowd Sourcing for Usability Testing vs. Lab Usability Testing on a college website

Some Examples

Lab Usability Test Crowd Sourced Usability Test

Participants 5 55 (14 spammers)

Participant Demographics Students Crowdworkers

Age 24 to 33 19 to 51

Education level Bachelor’s degree and Master’s degree All levels

Experience with similar

websites

Yes: 100% Yes: 77%

No: 23%

Speed Approximately 30 min. per session. Less than 4 hours total.

Participant

Costs

None $2.92 for pilot test

$23.41 for final test

(Avg: $0.48/tester)

* - http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf

Slide 48

Some Examples

Major Problems Identified

Lab Usability Test Crowd Sourcing Usability Test

Font size too small

Out-of-date information

Menu overlap

Irrelevant picture

Invisible tools

Information not cross-linked

Lack of sort function

Navigation unclear

Search box difficult to locate

Corp04 25

Slide 49

Some Examples

Advantages Disadvantages

More participants Lower Quality Feedback

High Speed Less Interaction

Low Cost Spammers

Various Backgrounds Less Focused User Groups

In Conclusion: 1. In this scenario, usability testing would be better off done by students of the college website;

choosing target users is very important 2. If crowd sourced testing is relevant based on your user profile, design the effort with care; using

same tests and questions as that of lab testers for crowd testers may not yield great results (see section 4.1.2 on Test Redesign)

Slide 50

Stake holder Buy In

• Stakeholder resistance to Crowd Sourcing

largely inline with model’s challenges:

• Product IP, privacy issues

• Internal team motivation

• Quality of test effort and product

• Additional overhead in effort management

• Randomized test effort – tactical in nature

Corp04 26

Slide 51

Stake holder Buy In

Identify stakeholders; prioritize the team;

engage early

Walk through the test strategy including crowd

sourcing plans; understand their

concerns

Explain Solutions Undertake Pilot if

needed

Ongoing Communication

1. Supplemental test technique 2. Acknowledge problems 3. Explain the technique as applicable to your product; don’t assume their know-how

1. Educated decision 2. Explain implementation plans – what & what not, when, how, 3. Define checks and balances for internal team and CS team roles 4. Map solutions to address each concern identified earlier

1. Practical demonstration of the model; adds to your confidence too!

2. Additional overhead at start but pays off in longer run

3. Run a pilot like a regular CS test program but of smaller scale

1. Show steady progress 2. Stick to pre-defined

communication protocols 3. Communicate the good and the

bad 4. Be on top of new stakeholders

or new concerns

Set # 6

• Acknowledging what not to crowd source will:

• Fasten stakeholder approval process

• Help not impact internal team motivation

Slide 52

Corp04 27

Core

Work Skills

Unique

Work Skills

Future

Work Skills

In-Job

Behaviors

Organizational

Citizenship

Behaviors

Successful Work Patterns

Slide 53

Core

Work

Skills

Unique

Work

Skills

Future

Work

Skills

In-Job

Behaviors

Organizational

Citizenship

Behaviors

Successful Work Patterns

Slide 54

Corp04 28

Localization Testing

• Hard, Large-Scale

Problem

• Windows 7 ships in 100

languages

• Thousands of strings

and screens per release

• Traditional model of

localization testers

expensive and difficult to

find

Slide 55

Let’s Play…

Slide 56

Corp04 29

The Language Quality Game

Slide 57

Results

Significant Quality Improvements for Windows 7

Positive Impact on Ship Schedule

Team Morale and Subsidiary Engagement

Internal sourcing alleviates security and access issues

Total Screens Reviewed: Over 500,000

Total Number of

Reviewers: Over 4,500

Screens per Reviewer: Average 119

Slide 58

Corp04 30

Amplify Skill with Volume

Individual dialects,

nuances, hard to

detect with a single

vendor – crowd does

a better job

Slide 59

Reduce Cost with Discovery,

Instrumentation

No need to install

Telemetry to direct

effort

Slide 60

Corp04 31

Reduce Risk with Diversity

Portfolio theory in

Quality Assurance

Slide 61

Trust and Transparency Increase

Effectiveness

Inclusion = Trust, Trust =

Enthusiasm

Subsidiary

engagement with

Language Quality

Game

Momentum for Win 7

Slide 62

Corp04 32

Players earn points

for disaster relief

agencies

Microsoft donates $

based on leader

board

Individual players

can sponsor tasks

or scenarios

Slide 63

Results

Significant Quality Improvements for Communicator “14”

Positive Impact on Ship Schedule

Team Morale and Dog-food User Engagement

Players Over 1,000

Feedback increase: > 16x

Feedback received: 10,000

Slide 64 Source : Ross Smith, Director of Test, Microsoft Corporation

Corp04 33

Crowd Testing – An Internal Example

Slide 65

• QA InfoTech’s application for Crowd Testing

• Encourages users, testers to test, make

additional income

• Detailed features around registration, events,

defects, metrics, bug money

• Application tested by internal crowd

• Registration money used to distribute prize

money

Crowd Testing – An Internal Example

Slide 66

Corp04 34

Crowd Testing – An Internal Example

Slide 67

Crowd Testing – An Internal Example

Slide 68

Corp04 35

Crowd Testing – An Internal Example

Slide 69

Crowd Testing – An Internal Example

Slide 70

Corp04 36

Crowd Testing – An Internal Example

Slide 71

• Amazing turnaround in 2.5 hours

• 200+ testers

• 3000+ bugs

• Bugs across all categories including performance

via simultaneous application use

• Good team motivation

• Participation by managers, leads and ICs alike

• Diverse bugs reported by people of varied

experiences

Crowd Testing – An Internal Example

Slide 72

• Motivators – Money, Group fun, Pride and

Recognition amongst all in company including

CxO level visibility

Corp04 37

Other Examples

Slide 73

• Working with Blind Relief Association to bring in

real users into accessibility testing

• Working with universities for content grading

• Mobile application testing across devices within

the company

Other Examples

Slide 74

Listen to Ross Smith, Director of Test at Microsoft, on his thoughts on Crowd Sourced Testing

Corp04 38

Crowd Sourced Testing – Myths and Facts

Slide 75

Crowd Sourced Testing impacts core testing team adversely; threatens their positioning in the product team

Really? 1. Crowd Sourced Testing - supplemental technique, not stand alone 2. Does not work in all situations; areas of niche to be reserved for internal testing 3. Internal team to build a sense of empowerment that the crowd adds to product quality

# 1

Crowd Sourced Testing – Myths and Facts

Slide 76

Crowd Sourcing is only for software testing. Development is a very technical and specialized area to leverage the crowd for

Really? 1. Think of Open Source Software….Linux - popular example of open software collaboration 2. Crowd Sourcing easier for testing than development as crowd often represents user base

3. Security, IP easier to manage in testing – no access to source

code 4. Practices discussed extendible to development as well 5. Top Coder.com another good example

# 2

Corp04 39

Crowd Sourced Testing – Myths and Facts

Slide 77

Management overhead is significantly higher in Crowd Sourced Testing. Given the short project deadlines, we do not have time or resources to manage a crowd sourced test

team

Partly True… 1. Management overhead slightly more...but this is inevitable in current day global development models 2. Crowd is a smart and self – sufficient group; YOU DON’T WANT TO MICRO-MANAGE

3. Best practices of what, when, how to crowd source will make

the effort streamlined and not chaotic

# 3

Crowd Sourced Testing – Myths and Facts

Slide 78

Crowd Testers can be ramped up or down at very short notice, giving great head count

flexibility

Partly True…

1. Scale potential of resources is huge

2. However, getting right resources at right time is challenging

3. Maintain a pool / common database and engage with the crowd

on ongoing basis even in lean periods

# 4

Corp04 40

In Conclusion

Slide 79

Let’s:

• Pictorially walkthrough

Crowd Sourcing

• Revisit Take-Aways

• Look at Call to Action

Crowd Sourcing – Pictorial Walkthrough

Slide 80

Corp04 41

Crowd Sourcing – Pictorial Walkthrough

Slide 81

Crowd Sourcing – Pictorial Walkthrough

Slide 82

Corp04 42

Take-Aways revisited

• CS used across companies, domains,

technologies

• Understand varied manifestations, crowd

motivators, to customize your implementation

• Educated decision of what, when, how to

crowd source

• Acknowledge model’s challenges; be

transparent in seeking stakeholder approval

Slide 83

Call to Action

• Evaluate programs in your

group; gradually build on them

• Start small

• Try a Pilot / Proof of Concept

• Register to be a crowd

sourced tester – internally,

externally - http://www.qainfotech.com/CS_Reg_Step1.php

Slide 84

Corp04 43

About QA InfoTech

• An independent software quality assurance and testing company,

founded in 2003, currently employing 650 people

• Five testing “Centers of Excellence” across the USA and India

• World-class testing labs

• Experience working with clients across various domains

• Bagged the “Top 100 places to work for in India*” award, three years

in a row

• Focus on the right balance of people, processes, technology • CMMi III, ISO 9001:2008, 20000-1:2005 certified

QA InfoTech facilities in India Slide 85

* Study conducted by Great Places to

Work Institute, India

Q&A – Let’s find answers together!

Slide 86

Corp04 44

References

• http://outsideinmarketing.wordpress.com/2010/05/08/four-types-of-crowdsourcing/

• http://explore2win.blogspot.in/2012/10/what-is-crowdsourcing.html

• http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf

Slide 87

Thank You

For more information, please:

• Contact us at info@qainfotech.com

• mukesh@qainfotech.net, rajini.padmanaban@qainfotech.net

• Visit us at www.qainfotech.com

• Read our blog at www.qainfotech.com/blog

• Follow us on Twitter at www.twitter.com/qainfotech

USA

Office

International

Headquarters

Noida

Uttar Pradesh, India

Phone: +91-120-4292222 (Three additional testing facilities in India)

Farmington Hills

Michigan, U.S.A.

Phone: +1-248-719-3409

Slide 88