three-step framework for overcoming challenges of site optimization

© 2012 Webtrends, All Rights Reserved. #optimizeplan @joestanhope @bob_garcia

Three-Step Framework for Overcoming Challenges of Site Optimization

June 19th, 2012

© 2012 Webtrends, All Rights Reserved. |2#optimizeplan @joestanhope @bob_garcia

“

It’s all about relevance

Relevance means understanding

a consumer’s digital behavior

across channels and then being

able to use that intelligence to

optimize digital marketing.

“


Webtrends Focus

Enabling digital marketers to deliver real-time relevance to their customers


what?who?

how?


Where Does Optimization Fit?

Analytics

Identify and quantify opportunities thru insight

Optimization

Test, segment and target to improve performance


Site Optimization Includes …


Optimization Team

Optimization Consultant

• Strategy and roadmap • Analytics evaluation• Optimization project lead• Campaign performance / KPIs

Web Developer

• Build out/development• Implementation• QA

Designer

• Creative strategy• Asset generation• Usability

QA

• Verify against plan• Multiple Browsers• Conversion Checks


Joe Stanhope (Guest Speaker)+1 617.613.8929

[email protected]

@joestanhope

www.forrester.com

blogs.forrester.com/joseph_stanhope

PresentersForrester + Webtrends

Bob Garcia+ 1 877.932.8736

[email protected]

@bob_garcia

www.webtrends.com

blogs.webtrends.com

mailto:[email protected]

https://twitter.com/%23!/joestanhope

http://www.forrester.com/

http://blogs.forrester.com/joseph_stanhope



https://twitter.com/%23!/bob_garcia

http://www.webtrends.com/

http://blogs.webtrends.com/

© 2012 Forrester Research, Inc. Reproduction Prohibited9 © 2009 Forrester Research, Inc. Reproduction Prohibited

A Complete Approach To Site Optimization: Anticipating Challenges and Harnessing Opportunities To Build An Online Testing Program

Joe Stanhope, Senior Analyst, Forrester Research

June 19, 2012

© 2012 Forrester Research, Inc. Reproduction Prohibited10

Website traffic is a valuable commodity. To make the most of this

asset, ensure that your marketing and customer experience efforts will

resonate with visitors and drive measurable benefits.

#optimizeplan

@bob_garcia

@joestanhope


What is Online Testing & Why Do You Need It?

Online Testing Maturity: The Scale Conundrum

Success Factor 1: Process

Success Factor 2: Organization

Agenda

#optimizeplan

@bob_garcia

@joestanhope


What is online testing?

Online testing defined: A statistical methodology that allows site owners to

test the impact of specific Web site components on target metrics

The primary experimental techniques of online testing are:

– A/B testing - Test multiple variations of a site

element against one another

– Multivariate testing - Test multiple variations

of multiple site elements against one

another simultaneously

#optimizeplan

@bob_garcia

@joestanhope


How is online testing useful?

Meet marketing or campaign goals (e.g. conversion, sales,

click-throughs, etc.)

#optimizeplan

@bob_garcia

@joestanhope





Improve customer experience

#optimizeplan

@bob_garcia

@joestanhope






Validate segmentation strategies

#optimizeplan

@bob_garcia

@joestanhope






Validate segmentation strategies

Statistically based insight into what works and what doesn’t

– Understanding of:

– Factor importance

– Factor interaction

#optimizeplan

@bob_garcia

@joestanhope


Testing addresses a variety of challenges

But users test primarily to enhance revenue and usability

3%

23%

39%

42%

42%

58%

70%

Other, please specify

Marketing integration with other online channels (suchas search, email, and offsite display advertising)

Content relevance (such ascontent presentedonsite, searchresults, or on landing pages)

Visitor segmentation and targeting

Overall customer engagement and satisfaction

Siterevenue generation (such as via offers,promotions and cross sell/up sell)

Site usability (such as site design, navigationand multistep processes)

“Whichtopicsrepresent the three greatest challenges thatyou seektoresolve withyour onlinetesting program?”

Base: 71respondents who use an online testing platform(multiple responses accepted)

Source: Q4 2011 Global online Testing User Online Survey

#optimizeplan

@bob_garcia

@joestanhope


Testing works for sites of all sizes

10%

14%

20%

11%

18%

13%

6%

7%

Upto 50,000 unique visitors

Upto 250,000 unique visitors

Upto 1 million unique visitors

Upto 2.5 million unique visitors




Over 50 million unique visitors

Base: 71respondents who use an online testing platform(percentages may not total 100 because of rounding)

“Onaverage, what is the monthlyvolume of unique visitors on theweb propertieson whichtesting is performed?”


#optimizeplan

@bob_garcia

@joestanhope


Ultimately, online testing is effective

3%

6%

9%

10%

20%

20%

21%

24%

28%

39%

73%

No benefits


Increased customer satisfaction (surveycontrolledgroup of customers)

Increased formcompletion (form submittals/visitsthat start form)

Increased site based searchconversion (ordersattributedto searchers/visitsto searchresults page)

Increasedregistration (registrations/visits)

Improved customer segmentation and targeting

Increasedaverage ordervalue (online revenue/orders)

Increased landing page conversion(orders/visitsto landing page)

Increasedconversion (orders/visits)


“What are thetop three benefitsyourcompany hasrealized through A/B and/or multivariatetesting?”

Increased off-site display ad clickthough(clicks/ad impressions)


#optimizeplan

@bob_garcia

@joestanhope

© 2012 Forrester Research, Inc. Reproduction Prohibited2424Entire contents © 2010 Forrester Research, Inc. All rights reserved.

Source: Insurance Institute for Highway Safety

© 2012 Forrester Research, Inc. Reproduction Prohibited2525Entire contents © 2010 Forrester Research, Inc. All rights reserved.

Source: Insurance Institute for Highway Safety

Online Testing Maturity: The Scale

Conundrum


Online Testing Program Longevity Is Diverse But Skews YoungFebruary 2012 “The State Of Online Testing 2012”

#optimizeplan

@bob_garcia

@joestanhope


Few Online Testing Users Run Large Numbers Of ExperimentsFebruary 2012 “The State Of Online Testing 2012”

#optimizeplan

@bob_garcia

@joestanhope


“Onaverage, approximately how many A/B and/or multivariatetestsdoyou runconcurrently onyourweb properties?

4%

39%

32%

7%

3%

3%

1%

0%

10%

1-2

3-5

6-8

9-11

12-14

15-17

18-19

20 or more

0


Concurrency is low


#optimizeplan

@bob_garcia

@joestanhope


1%

20%

20%

21%

27%

41%

42%

47%

48%

51%

73%

Other

Breadth of available online testing techniques

Strength of vendor’sreference customers

Availability of advanced or proprietaryfeatures andfunctionality

Vendor’sreputation (e.g.known as a leader in this field)

The online testing application is a module within a largersuite of analytics and/or marketing tools

Test deployment method (i.e.type of site tags, etc.)

Vendor’s abilityto servicemy business needs

Total cost of ownership

Ease of use and automation

Vendor’s thought leadership

User interface

Professional services, managed services,and training capabilities

Standard features and functionality (such asreporting,segmentation and targeting)

Ease of implementation and ongoing management


“Which of thefollowing are the five most important factors whenselecting an onlinetesting solution?”

Experience in our industry

17%

17%

14%

13%

11%

10%

Product support capabilities

Selection criteria reflects maturity and support needs

Steady


#optimizeplan

@bob_garcia

@joestanhope


1%

20%

20%

21%

27%

41%

42%

47%

48%

51%

73%

Other

Breadth of available online testing techniques

Strength of vendor’sreference customers

Availability of advanced or proprietaryfeatures andfunctionality

Vendor’sreputation (e.g.known as a leader in this field)

The online testing application is a module within a largersuite of analytics and/or marketing tools

Test deployment method (i.e.type of site tags, etc.)

Vendor’s abilityto servicemy business needs

Total cost of ownership

Ease of use and automation

Vendor’s thought leadership

User interface

Professional services, managed services,and training capabilities

Standard features and functionality (such asreporting,segmentation and targeting)

Ease of implementation and ongoing management


“Which of thefollowing are the five most important factors whenselecting an onlinetesting solution?”

Experience in our industry

17%

17%

14%

13%

11%

10%

Product support capabilities

Selection criteria reflects maturity and support needs

Steady

On the Rise


#optimizeplan

@bob_garcia

@joestanhope


Components of an online testing program

Each element is crucial

#optimizeplan

@bob_garcia

@joestanhope



Culture of measurement and innovation

– Online testing is an iterative program, not a discrete test

– Online testing is an experiment, not a pass/fail exercise


#optimizeplan

@bob_garcia

@joestanhope






Process

– Plan, design, execute, analyze, validate, repeat

– Marketing input

– Program goals


#optimizeplan

@bob_garcia

@joestanhope






Process


– Marketing input

– Program goals

Organization

– Defined ownership for the program is critical

– Skills to support the process


#optimizeplan

@bob_garcia

@joestanhope






Process


– Marketing input

– Program goals

Organization

– Defined ownership for the program is critical

– Skills to support the process

Technology

– Site management

– Analytics


#optimizeplan

@bob_garcia

@joestanhope


Success Factor 1: Process


14%

25%

28%

32%

Conducting tests is predominantly an ad hoc process

Testing followsformalized processesfor some tasksbut is mostly ad hoc

Testing follows a highly formalized process throughoutthe development, execution, and analysis of experiments

Testing followsformalized processesfor most tasks,but some tasks are ad hoc


“To what degree doesyour firm adhereto a structured processtoconduct onlinetesting?”

The majority of firms struggle with processes


#optimizeplan

@bob_garcia

@joestanhope


The elements of a testing plan

#optimizeplan

@bob_garcia

@joestanhope


The process is critical

#optimizeplan

@bob_garcia

@joestanhope


Online Testing Processes

Who are the stakeholders?

How do we ideate experiments?

What is the criteria for prioritizing experiments?

Can we get support from existing process management expertise such as

our PMO?

How does the team stay informed?

What information populates our online testing knowledgebase?

It’s important to consider:

#optimizeplan

@bob_garcia

@joestanhope


Success Factor 2: Organization

Image source: Hostile Work Environment (http://hostilework.blogspot.com/)


3%

14%

14%

14%

16%

17%

18%

18%

18%

30%

34%

Usability of thetesting interface

Other

Abilityto segment/target test participants

Lack of corporate buy-in

Modifyingweb pagesfortesting

Establishing testing scenarios (i.e.concept, areasto test)

Acting upon testing results

Securing ITresourcesto deploy and manage testcodeon thewebsite

Demonstrating ROI

Interpreting test results

Developing testcontent (such as creative elementsand/or promotionsto serve astestvariations)

Ability of testing application to accommodate desiredtesting techniques and designs

Attaining enough site trafficfor statistically significant tests

Lack of staffto manage the testing program and utilizetesting applications

Prioritizing testing initiatives


“What challenges doyouexperience in operatingyourcompany'sA/B and/or multivariatetesting application?

No challenges

13%

10%

7%

6%

6%

4%

Securing funding for testing initiatives


#optimizeplan

@bob_garcia

@joestanhope


1%

6%

7%

10%

20%

24%

32%

IT

Dedicated onlinetestingor site optimizationgroup

Other

Business intelligence orcorporate analyticsgroup

Web analyticsgroup

eCommerce

Marketing

Base: 71respondents who use an online testing platform

“What department inyour firm isresponsiblefor onlinetesting?”

Online testing is part of the business


#optimizeplan

@bob_garcia

@joestanhope


The Ownership Model SpectrumAugust 2010 “Organizing For Site Optimization”

#optimizeplan

@bob_garcia

@joestanhope


3%

11%

14%

17%

20%

35%

Other, please describe

A single full time resource

Acombination of part and full time resources

A single part timeresource

Multiple full time resources

Multiple part timeresources


“How is the onlinetesting program staffedatyour firm?”

Online testing is a team effort


#optimizeplan

@bob_garcia

@joestanhope


1%

1%

6%

20%

35%

37%

Allow external agencyto manage the entire process


Allowvendorto manage the entire process

Work with an external agency orconsultancy

Manage entirely in house

Work with our vendor’s professional servicesgroup


“What isyour preferred method of managing A/B and/or multivariatetesting onyourwebsite(s)?”

Most firms leverage third party resources


#optimizeplan

@bob_garcia

@joestanhope


How can a third party service provider help?

Services to support all stages of online testing

#optimizeplan

@bob_garcia

@joestanhope



Technical services

– Implementation

– Training

– Integrations


#optimizeplan

@bob_garcia

@joestanhope



Technical services

– Implementation

– Training

– Integrations

Program support

– Test ideation and prioritization

– Test design

– Creative services

– Experiment analysis

– Strategic advisory


#optimizeplan

@bob_garcia

@joestanhope



Technical services

– Implementation

– Training

– Integrations

Program support

– Test ideation and prioritization

– Test design

– Creative services

– Experiment analysis

– Strategic advisory

Managed services


#optimizeplan

@bob_garcia

@joestanhope


Third Party Services

Which components of the testing program would benefit from external

support?

How much help do you need to support a robust online testing program?

How much control do you want?

Do we want to become self-sufficient (eventually)?

What is the opportunity cost of delayed testing versus the expense of

services?

It’s important to consider:

#optimizeplan

@bob_garcia

@joestanhope


Make the business case for testingProve the concept with a modest, well chosen pilotJustify investment to grow and overcome challenges

Start with the basicsWork the maturity curve from A/B to Multivariate testingBuild up cadence, complexity, and concurrency over time

Online testing programs require ownershipA named resource must own the online testing program, even if in a

part-time capacity

Online testing is a people businessRealistically assess your current organization and skills relative to

managing an effective online testing program





part-time capacity






Online testing programs require ownershipA named resource must own the online testing program, even if in a part-time capacity






Online testing programs require ownershipA named resource must own the online testing program, even if in a part-time capacity

Online testing is a people businessRealistically assess your current organization and skills relative to managing an effective online testing program


Summary + Close


Summary - Why It Matters?

1. Make More Money – You are likely leaving money on the table

We often deliver 20-50% lift translating to thousands and millions of new found revenue.

2. Customer Experience – Customers own your future

A business that listens and relates will win. What are your competitors doing?

3. Eliminate Risk – You’re taking big and unnecessary risks

Not only does testing make money, but it helps you avoid making costly mistakes.


British fashion drives

e-commerce relevance

Business Challenge

Reduce cart abandonment and encourage repeat

website visitors for new merchandise launches,

both on the UK and USA sites.

Solution

Webtrends Optimize for testing and targeting

during both sales events and non-sales time

periods, and for testing website components for

best responses.


Gearing Emails for

One-to-One Merchandising

Business Challenge

Find more effective ways to act on rich customer

data and provide dynamic, one-to-one offers based

on individual views, searches and cart activity

Solution

Webtrends Analytics identifies promotional

opportunities for products that generate a high

number of views, searches or purchases.


Great Forrester Quotes


Realities …

“Testing Programs Require Investment to Achieve Scale.”

“… experience, process, and resource constraints throttle their efforts to scale up online testing program.”

“… firms must commit to investing in online testing programs to achieve scale and long-term success.”

“… testing delivers measureable benefits across a range of success metrics”

Forrester Research, Inc., The State of Online Testing 2011, February 2012


Q&A

#OptimizePlan | @joestanhope | @bob_garcia


Joe Stanhope (Guest Speaker)+1 617.613.8929

[email protected]

@joestanhope

www.forrester.com

blogs.forrester.com/joseph_stanhope

Thank YouForrester + Webtrends

Bob Garcia+ 1 877.932.8736

[email protected]

@bob_garcia

www.webtrends.com

blogs.webtrends.com


https://twitter.com/%23!/joestanhope

http://www.forrester.com/

http://blogs.forrester.com/joseph_stanhope



https://twitter.com/%23!/bob_garcia

http://www.webtrends.com/

http://blogs.webtrends.com/

three-step framework for overcoming challenges of site optimization

Business

webtrends focus

online testing maturity

site owners

optimization fit

forrester researchjune

reproduction prohibited2012

relevance relevance

digital marketing