national cancer institute cancer.gov main site

Satisfaction Insight ReviewDiscovery

NATIONAL CANCER INSTITUTE CANCER.GOV MAIN SITE

Michelle HertzSenior Satisfaction Research AnalystCovering: October 01, 2010 – September 30, 2011Presented: January 2012

2NCI Main > Satisfaction Insight Review 2

Agenda˃ Methodology and Background˃ SIR Objectives˃ Capturing the Voice of Visitor˃ Aggregate Results˃ Social Media and Mobile Overview˃ Comparison to cancer.gov en Español˃ Findings, Conclusions, Recommendations, and Next Steps˃ Appendix

3NCI Main > Satisfaction Insight Review

The ForeSee methodology uses the American Customer Satisfaction Index (ACSI) The ACSI, developed by the National Quality Research Center at the University of Michigan, has been the leading national indicator of Customer Satisfaction with goods and services in the U.S. economy since 1994. The ACSI is the only cross industry, national indicator that links Customer Satisfaction to financial returns.

ACSI model applied to web satisfaction utilizes a series of equations to define the relationships between:

Drivers (or elements) of Satisfaction and overall Satisfaction

Overall Satisfaction and Future Behaviors.

The results of this analysis will enable your organization to…Establish a reliable metric for measuring the satisfaction of site visitors on a continuous basis

Diagnose and prioritize areas for improvement by identifying the impact of change on satisfaction and customers’ future behavior

Proactively identify areas in need of improvement

Methodology (ACSI)


Past Reporting Foci

Past Satisfaction Insight Reviews (SIRs) have focused on the key visitor groups by role. The most recent SIR examined scientist researchers.

Current SIR Focus

The current report provides a snapshot of fiscal year 2011 ACSI model results for the site in aggregate and comparisons to benchmarks.

This report also reviews top visitor role groups’ responses to questions regarding social media and mobile use.

This report provides an aggregate-level comparison to cancer.gov en Español survey results for social media as an acquisition source.

SIR Objectives


Vital Statistics

Start of Data CollectionDecember 30, 2008

Survey PlacementEnglish-language pages

Reporting PeriodOctober 1, 2010 – September 30, 2011

Surveys Completed12,806

Sampling ConditionsThe survey invitation presents randomly to 4% of visitors who view four or more pages of cancer.gov. When visitors accept the invitation, the survey presents when they leave the site. A persistent cookie prevents visitors from seeing the invitation again for at least 60 days.

Capturing the Voice of Visitor

6

Aggregate Results


Satisfaction drives loyalty, retention and word-of-mouth, all of which drive website success.

Customer Satisfaction

Impact on Future

Behavior Future Behaviors Score3.9 Likelihood to Return 894.7 Primary Resource 784.2 Recommend 88

12806

82

Number of respondents:

C.I. +/- 0.3 at 90% level of confidence

How is your site doing? How does your site’s score compare with others?

Where should you focus your efforts? Why does improving Satisfaction matter?

MaximumAverageMinimum

Aggregate ResultsOctober 01, 2010 – September 30, 2011

September 2011 Benchmarks


8.6

8.4

8.3

8.1

7.8 8.3 8.8

Please rate how well the NCI Web site is organized.

Please rate the options that are available for you to navigate on the NCI Web site.

Please rate how well the NCI Web site layout helps you f ind what you are looking

for.

Please rate the degree to which the number of steps to get where you want on

the NCI Web site is acceptable.

Priority Map Top Priority Element Model Questions

Top Priority ElementsWithin navigation, the top priority aspect of the visit experience and best opportunity to continue improvements, visitors report lowest ratings for the number of steps to get where they want to on the site.

Aggregate ResultsPriority Areas of Focus

October 01, 2010 – September 30, 2011

9

Social Media and Mobile Overview


Social Media and Mobile OverviewPurpose of Social Media Value Calculation

Social Media Value CalculationThe purpose of the Social Media Value Calculation is to help companies better gauge the effectiveness of social media as an acquisition source for their website.


Social Media and Mobile OverviewApproach to Social Media Value Calculation

Referring URLWhere visitors came from

Custom QuestionRank the top 3 factors that brought you to the website

The Social Media Value Calculation categorizes visitors based on the following:


Social Media and Mobile OverviewSocial Media Value Calculation Custom Question

To capture the acquisition source of respondents one primary custom question is added to the survey.

Custom question as it appears on the survey

Answer choices as they appear in all three “Rank” drop down menus

Note: Answer choices are randomized.

s s


Social Media answer choices are consistent and allow us to Benchmark Social Media Value on all participating surveys.

v

Non-Social Media answer choices can be personalized or changed and are not a part of the Benchmark.

v

Because the first ranking is required, there must be answer choices for those who don’t see another appropriate response.

v

Social Media and Mobile OverviewSocial Media Value Calculation Answer Choices


Social Media and Mobile OverviewSocial Media Value

November 02, 2010 – September 30, 2011

Referrals Segmentation

Social Media Referral Composition

Amount of Social Influence % Total Survey Respondents Satisfaction Likelihood to

ReturnPrimary

Resource Recommend

Social Media Influence 12% 82 88 80 87

No Social Media Influence 88% 82 89 77 88

Total 100% 82 89 77 88

Amount of Social Influence Percentage of Respondents Satisfaction Likelihood to

ReturnPrimary

Resource Recommend

Direct Social Media Influence (Referring URL) 0% 74 81 65 84

Primary Social Media Influence 5% 82 87 78 87

Secondary Social Media Influence 5% 81 88 80 87

Tertiary Social Media Influence 4% 82 89 81 88


N: 11,230


Social Media and Mobile OverviewRelationship Between Satisfaction and Social Media Value



49%

39%

31% 30%

12%

5%3% 3% 3% 1% 1% 1%

43%40%

44%

25%

11%

6%

2%4% 5%

1% 1% 0%0%

10%

20%

30%

40%

50%

60%

Facebook YouTube Wikipedia None/Not applicable

LinkedIn Twitter MySpace Flickr Other Stumble Upon

Delicious Ning

US Patient Non-US Patient

Social Media and Mobile OverviewPatient – Social Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)Facebook, Wikipedia, and YouTube are the top-visited sites for both US and non-US patients. Wikipedia has superior usage by non-US patients than by those in the US. A greater proportion of US patients than non-US report no use of social media websites at all.


Bebo, Digg, Reddit, and Technorati each have zero percent of response for both US Patient and Non-US Patient visitor groups.

N: 5,648 N: 995


83%

37% 35%30% 29%

23% 22% 21% 20% 19% 17% 15% 14%10%

78%

33%28% 25%

21%18% 20% 19%

14% 16%12% 13% 11% 11%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%


Social Media and Mobile OverviewPatient – Internet Activities

In the last three months, which of the following have you done: (Check all that apply)Most patients have conducted Internet searches. US patients have done more of the listed activities than have non-US patients. The most common activities are similar between US and non-US patients.


See Appendix for full wording of answer options.

N: 12,350 N: 1,819

Only answer options with 10% or more of response for either segment are displayed.


43%

16% 15%8% 8%

2% 2% 1% 1% 1% 1% 1% 0%

53%

11%14%

4% 4%1% 2% 1% 1% 3% 2% 2% 1%

0%

10%

20%

30%

40%

50%

60%



Social Media and Mobile OverviewPatient – Acquisition Source

Which of the following sources drove you to visit the Cancer.gov site today?Search engines bring over half of non-US patients, and they are also the largest acquisition source for US patients, as well. Familiarity with NCI or with cancer.gov is higher among US patients than among non-US patients.


Answer choices with zero percent of response for both groups are not displayed.

N: 2,744 N: 493


56%

42%

35%

24%

16%

6%4% 5%

2% 1% 1% 0% 1%

52%

40% 41%

22%

14%

9%

2%5% 5%

1% 1% 1% 0%0%

10%

20%

30%

40%

50%

60%



Digg Delicious Ning

US Family member, Friend or Acquaintance Non-US Family member, Friend or Acquaintance

Social Media and Mobile OverviewFamily, etc. – Social Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)Facebook, Wikipedia, and YouTube are the top-visited sites for both US and non-US family and friends, as well, with Wikipedia again getting greater usage by non-US visitors. Nearly a quarter of family and friends report no use of social media websites at all.


Bebo, Reddit, and Technorati each have zero percent of response for both US Family and Non-US Family visitor groups.

N: 7,919 N: 1,762


85%

43% 42% 40%

31%25% 26% 28% 25% 25%

19% 18% 17%

8%

79%

32% 32% 30%

21% 18% 19% 18% 19% 17%12% 14% 12% 12%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%


Social Media and Mobile OverviewFamily, etc. – Internet Activities

In the last three months, which of the following have you done: (Check all that apply)Similarly, most family and friends of patients have done Internet searches. US family and friends have engaged in the listed Internet activities more so than have non-US family and friends.


See Appendix for full wording of answer options. Only answer options with 10% or more of response for either segment are displayed.

N: 18,423 N: 3,183


46%

18%13%

8% 7%1% 1% 1% 1% 1% 1% 1% 1%

57%

10% 11%6% 4% 2% 2% 1% 1% 2% 1% 1% 1%

0%

10%

20%

30%

40%

50%

60%


N: 801


Social Media and Mobile OverviewFamily, etc. – Acquisition Source

Which of the following sources drove you to visit the Cancer.gov site today?Similarly, search engines are also the largest source for non-US family and friends of patients, as well as for their US counterparts. US family and friends report greater familiarity with NCI or with cancer.gov than do non-US.



N: 3,554


51%

40%

36%

27%

15%

5%3% 4% 2% 1%

40%

47% 47%

22%

9%6%

2% 2%4%

0%0%

10%

20%

30%

40%

50%

60%



US Health Care Provider Non-US Health Care Provider

Social Media and Mobile OverviewHealth Care Provider – Social Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)While US health care providers make greatest use of Facebook, non-US health care providers most commonly visit Wikipedia and YouTube. Among health care providers, a greater proportion of US than non-US report using no social media websites at all.


Bebo, Delicious, Digg, Ning, Reddit, and Technorati each have zero percent of response for both US HCP and Non-US HCP visitor groups.

N: 2,299 N: 974


85%

50%46%

42% 41%35% 34%

28% 26% 26%22% 21% 18%

78%

30%

21%27%

20% 20% 20%13%

17%13% 13%

7% 9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

US Health Care Provider Non-US Health Care ProviderN: 1,669

Social Media and Mobile OverviewHealth Care Provider – Internet Activities

In the last three months, which of the following have you done: (Check all that apply)US health care providers engage in the listed Internet activities at a higher rate than do non-US health care providers by a larger margin than seen among patients and their family and friends. Health care providers use online learning tools at a high rate, compared to patients and their family and friends.



N: 6,097


44%

21%

14%10%

5%1% 1% 1% 1% 1% 1% 0% 0%

32%28%

14%10%

5%2% 1% 1% 0% 1% 2% 2% 1%

0%5%

10%15%20%25%30%35%40%45%50%

US Health Care Provider Non-US Health Care ProviderN: 486


Social Media and Mobile OverviewHealth Care Provider – Acquisition Source

Which of the following sources drove you to visit the Cancer.gov site today?US health care providers most commonly come to the site as a result of their familiarity with NCI or with cancer.gov, whereas non-US health care providers as a results of a search engine nearly as often as a result of their familiarity with the site itself.



N: 1,070


62% 59% 58%

43%

13% 13% 11%

5%2% 3% 3% 2% 2% 1% 1% 1%

34%

57%

39%

22% 20%

7% 6%3%

6%0% 0% 0% 0% 0% 0% 0%

0%

10%

20%

30%

40%

50%

60%

70%

US Scientist Researcher Non-US Scientist Researcher

Social Media and Mobile OverviewScientist Researcher – Social Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)US scientist researchers report the highest of social media use. Over half visit Facebook, Wikipedia, and YouTube, and nearly half visit LinkedIn. Non-US scientist researchers report lower rates of YouTube than do other top non-US roles.


Bebo, Delicious, Digg, Ning, Reddit, and Technorati each have zero percent of response for both US SR and Non-US SR visitor groups.

N: 660 N: 402


92%

61% 58% 55% 52%48% 44% 42% 41% 38% 38%

32%27%

12% 14% 12%5%

77%

30%25% 26%

18%25% 24% 20%

25%16% 15% 15%

9%4% 5% 3%

11%

0%10%20%30%40%50%60%70%80%90%

100%

US Scientist Researcher Non-US Scientist ResearcherN: 732

Social Media and Mobile OverviewScientist Researcher – Internet Activities

In the last three months, which of the following have you done: (Check all that apply)US scientist researchers report the highest rate of engaging in the listed Internet activities of any of the top visitor groups; while non-US scientist researchers engage in Internet activities at about the same rate as other non-US roles.



N: 1,626


38%

29%

14%10%

4%1% 1% 0% 0% 0% 0% 0% 0% 0%

24%26%

12% 12%

6%3% 2%

4% 4% 2% 2% 1% 1% 1%0%5%

10%15%20%25%30%35%40%

US Scientist Researcher Non-US Scientist ResearcherN: 190


Social Media and Mobile OverviewScientist Researcher – Acquisition Source



N: 216

Which of the following sources drove you to visit the Cancer.gov site today?Familiarity with NCI or cancer.gov is the greatest driver of US scientist researcher traffic, while non-US scientist researchers are just as likely to arrive at the site via search engine as because of their familiarity with NCI or the site. Search engines and site familiarity bring over half of all scientist researcher traffic.


63%

53% 51%

42%

21%16%

11%8% 6% 5% 3% 3% 1% 1%

0%

10%

20%

30%

40%

50%

60%

70%

US Advocate, Media/Journalist, Public Health Official, Policy Maker

Social Media and Mobile OverviewUS Advocate, etc. – Social Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)Facebook, Wikipedia, YouTube, and LinkedIn all receive high levels of use by US advocates, journalists, public health officials, and policy makers.


Bebo and Technorati each have zero percent of response. Note: Insufficient sample for Non-US; combined results were similar to US alone.

N: 441


90%

59% 58%51% 48% 46% 44% 43% 39% 38% 37% 34%

30%

18% 15% 12% 10%

0%10%20%30%40%50%60%70%80%90%

100%


Social Media and Mobile OverviewUS Advocate, etc. – Internet Activities

In the last three months, which of the following have you done: (Check all that apply)US advocates, journalists, public health officials, and policy makers are also heavily engaged in the listed Internet activities.



N: 1,056

Note: Insufficient sample for Non-US; combined results were similar to US alone.


34%

28%

14%11%

6%

2% 2% 2% 2% 1%0%

5%

10%

15%

20%

25%

30%

35%

40%



Social Media and Mobile OverviewUS Advocate, etc. – Acquisition Source

Which of the following sources drove you to visit the Cancer.gov site today?US advocates, journalists, public health officials, and policy makers also arrive at the site most often due to their familiarity with NCI or cancer.gov. They next most often arrive via a search engine.


Answer choices with zero percent of response are not displayed.

N: 133


50%

65%

49%53%

70%

83%

59%

78%

35%

50%

35%

51%47%

30%

18%

41%

22%

65%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%


US Family … Non-US Family …

US Health Care Provider

Non-US Health Care Provider

US Scientist Researcher

Non-US Scientist

Researcher

US Advocate …

Yes No

Note: Insufficient sample for Non-US Advocate

Social Media and Mobile OverviewBrowsed Using Mobile, Looked For Health Information

When you have browsed websites using your mobile phone/device, have you looked for health information?For each of the top roles, non-US visitors report a higher rate of looking for health information on a mobile device than do US visitors. Health care providers report the highest rates overall.

April 01, 2011 – September 30, 2011

N: 242 N: 31 N: 411 N: 75 N: 190 N: 40 N: 44 N: 18 N: 26

32

Comparison to cancer.gov en Español



ReturnPrimary

Resource Recommend



Total 100% 83 90 84 89

EspañolEspañol

Comparison to cancer.gov en EspañolSocial Media Value

Main: November 02, 2010 – September 30, 2011 / Español: November 04, 2010 – September 30, 2011

Referrals Segmentation


ReturnPrimary

Resource Recommend



Total 100% 82 89 77 88

Main

Main N: 11,230 / Español N: 9,826



ReturnPrimary

Resource Recommend






Español


ReturnPrimary

Resource Recommend






Comparison to cancer.gov en EspañolSocial Media Value

Social Media Referral CompositionMain

Main N: 11,230 / Español N: 9,826



53%

43%

38%

24%

15%

7%5% 4% 3% 1% 1% 1%

57%

46%

38%

18%

4%

9%

2%5% 4%

0% 0% 0%0%

10%

20%

30%

40%

50%

60%

70%



Digg Delicious

Main Español

Comparison to cancer.gov en EspañolSocial Media Sites Visited

In the past 30 days, which of the following social media sites have you visited? (Multi-Select)Like visitors to Main, visitors to Español most often visit Facebook, Wikipedia, and YouTube. In fact, they are greater users of Facebook and YouTube than are visitors to Main. Fewer visitors to Español than to Main report no use of social media websites at all.


Bebo, Ning, Reddit, and Technorati each have zero percent of response for both Main and Español.

N: 25,087 N: 21,659


83%

41% 39% 36%29% 27% 25% 25% 24% 24%

19% 18% 16%

79%

24%28%

23%

13% 11% 12%17%

27%

11%5%

10%7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Main EspañolN: 55,722 N: 33,368

In the last three months, which of the following have you done: (Check all that apply)Visitors to Main engage on more activities on the Internet than do visitors to Español, with both groups engaging most commonly in Internet searches.

Comparison to cancer.gov en EspañolInternet Activities




41%

20%

14%9%

6%2% 2% 1% 1% 1% 1% 1% 1% 0% 0% 0%

39%

12%

18%

6% 5%2% 3%

6%1% 1% 2% 2% 1% 1% 1% 1%

0%5%

10%15%20%25%30%35%40%45%

Main Español


Comparison to cancer.gov en EspañolAcquisition Source



Which of the following sources drove you to visit the Cancer.gov site today?Search engines are the top driver of traffic to cancer.gov for both Main and Español.

83%

41% 39% 36%29% 27% 25% 25% 24% 24%

19% 18% 16%

79%

24%28%

23%

13% 11% 12%17%

27%

11%5%

10%7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Main EspañolN: 11,212 N: 9,800


55%

67%

45%

33%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Main Español

Yes No

Main N: 1,288 / Español N: 819

Comparison to cancer.gov en EspañolBrowsed Using Mobile, Looked For Health Information

When you have browsed websites using your mobile phone/device, have you looked for health information?Visitors to cancer.gov en Español report a higher rate of looking for health information on a mobile device than do visitors to the Main site.

April 01, 2011 – September 30, 2011

39

Findings, Conclusions, Recommendations, and Next Steps


Findings

For the Main site, satisfaction is the same, regardless of whether the visitor came to the site as the result of a social media influence or not. Visitors to the Español portion of cancer.gov report accessing the site due to a social media influence at twice the rate of visitors to Main. For Español visitors, a social media influence contributes to higher satisfaction with the visit and higher likelihoods to engage in the future behaviors, a pattern not seen with visitors to Main.

Visitors to the Español site are greater users of Facebook and Wikipedia than are Main visitors. The top three social media sites accessed by both Main and Español visitors are Facebook, YouTube, and Wikipedia. For visitors to Main, Facebook is the most commonly used by US patients, and Wikipedia is the most commonly used by non-US patients. Among family and friends of patients, Facebook is the most commonly used social media site for both US and non-US.

Visitors to the Main site engage in more Internet activities than do visitors to Español. Similarly, US visitors to Main have done more of the activities than have non-US visitors. Conducting a search is the most common activity for all groups. Among US visitors, professional visitors report a much higher rate of using an online learning tool than do patients or their family and friends.

For both Main and Español, the top driver of traffic to cancer.gov is search engines. For public visitors like patients and their family and friends, search engines are the greatest acquisition source, whereas for professional visitors like health care providers, scientist researchers, and advocates, journalists, public health officials, and policy makers, familiarity with NCI or cancer.gov is the number one source of traffic. US visitors are more familiar with NCI and cancer.gov than are non-US visitors.

Visitors to Español report a higher rate of looking for health information on a mobile device than do visitors to Main, and non-US visitors report a higher rate than do US visitors. On Main, health care providers report the highest rates of looking for health information on a mobile device.

Findings


Conclusions

Conclusions

Social media outreach efforts focused on Facebook, YouTube, and Wikipedia will be most effective at reaching visitors in all of the top role groups. Both US and non-US visitors are reachable via these platforms.

Specialty social media websites, such as Bebo, Delicious, Digg, Flikr, MySpace, Ning, Reddit, Stumble Upon, and Technorati may not be worth the investment at this time, as they reach only single-digit percentages of visitors, even those in professional roles.

Depending on the role of the visitor, anywhere from 13%-to-30% of visitors will be unreachable via social media initiatives, as they are not users of social media.

Online learning tools aimed at professional visitors may be more successful in reaching their intended audiences than those aimed at the public (patients, family, friends).

Maintaining a focus on search engine optimization is important, as search engines are a major source of traffic for cancer.gov.

Mobile devices are a key tool for international visitors, especially health care providers, seeking out health information.


Recommendations

Recommendations

Continue or expand social media outreach to Spanish-language visitors, as those who cite a social media influence in their decision to access cancer.gov en Español display higher satisfaction than do those who have no social media influence.

Focus domestic and international social media outreach initiatives on Facebook, YouTube, and Wikipedia, the platforms that are used by the most visitors from the top role groups.

Continue or expand search engine optimization efforts, given the large portions of visitors from the top roles who get to the site via a search engine.

When evaluating potential online learning tools to offer on the site, prioritize those aimed at professional visitors, as they are greater users of such tools than are patients, family, and friends.


Next Steps

Next Steps

Are the questions regarding Internet activities and social media sites visited still adding value, or has sufficient information from these questions been gained, such that they could be replaced with more detailed questions?

To take the social media and mobile analysis to the next level, consider adding custom questions to find out what content on cancer.gov visitors would be most interested in accessing via social media, and separately, via a mobile device.

Try the ForeSee mobile portal in order to access an at-a-glance snapshot of your satisfaction, model scores, and trend lines using a free iPhone / iPad or Android application. You can download the iPhone/iPad app from the iTunes App Store or the Android app from the Android Market (see page 49).

44

Appendix


AppendixModel Questions

ELEMENTS (drivers of satisfaction) CUSTOMER SATISFACTION FUTURE BEHAVIORSContent (1=Poor, 10=Excellent, Don't Know) Satisfaction Likelihood to Return (1=Not Very Likely, 10=Very Likely)Please rate your perception of the accuracy of information on the NCI Web site.

What is your overall satisfaction with the NCI Web site?(1=Very Dissatisfied, 10=Very Satisfied)

How likely are you to return to the NCI Web site if you needed information about cancer?

Please rate the quality of information on the NCI Web site.How well does the NCI Web site meet your expectations?(1=Falls Short, 10=Exceeds)

Recommend (1=Not Very Likely, 10=Very Likely)

Please rate the depth of information on the NCI Web site.

How does the NCI Web site compare to your idea of an ideal website?(1=Not Very Close, 10=Very Close)

How likely are you to recommend the NCI Web site to someone else if he/she needed information about cancer?

Please rate your perception of how current the information is on the NCI Web site.

Primary Resource (1=Not Very Likely, 10=Very Likely)

Functionality (1=Poor, 10=Excellent, Don't Know) How likely are you to use the NCI Web site as your primary resource for information relating to cancer?

Please rate the usefulness of the features provided on the NCI Web site.Please rate the convenience of the features on the NCI Web site.Please rate the variety of features on the NCI Web site.Look and Feel (1=Poor, 10=Excellent, Don't Know)Please rate the visual appeal of the NCI Web site.Please rate the balance of graphics and text on the NCI Web site.Please rate the readability of the pages on the NCI Web site.

Navigation (1=Poor, 10=Excellent, Don't Know)Please rate how well the NCI Web site is organized.Please rate the options that are available for you to navigate on the NCI Web site.Please rate how well the NCI Web site layout helps you find what you are looking for.Please rate the degree to which the number of steps to get where you want on the NCI Web site is acceptable.


AppendixCustom Questions

As of September 30, 2011

Skip From Question TextAnswer Choices

(limited to 50 characters) Skip to First time Daily About once a week About once a month Every few months Once or twice Message or recommendation from a friend on a social networkVideo I saw on YouTubeInternet blogs or discussion forumsAdvertising on social networks (Facebook, MySpace, Twitter)Message directly from the National Cancer Institute (NCI) on a social networkMobile phone text messages or alertsInstant Message from a friend or colleagueFamiliarity with Cancer.gov/National Cancer Institute (NCI)Promotional email(s) from the National Cancer Institute (NCI)Link from another websiteMobile AppsSearch engine resultsWord of mouth recommendation from someone I knowTV, radio, newspaper, or magazine advertisingInternet advertisingDon't knowOther

Rank 2 (Optional) Message or recommendation from a friend on a social networkVideo I saw on YouTubeInternet blogs or discussion forumsAdvertising on social networks (Facebook, MySpace, Twitter)Message directly from the National Cancer Institute (NCI) on a social networkMobile phone text messages or alertsInstant Message from a friend or colleagueFamiliarity with Cancer.gov/National Cancer Institute (NCI)Promotional email(s) from the National Cancer Institute (NCI)Link from another websiteMobile AppsSearch engine resultsWord of mouth recommendation from someone I knowTV, radio, newspaper, or magazine advertisingInternet advertisingDon't knowOther

Rank 3 (Optional) Message or recommendation from a friend on a social networkVideo I saw on YouTubeInternet blogs or discussion forumsAdvertising on social networks (Facebook, MySpace, Twitter)Message directly from the National Cancer Institute (NCI) on a social networkMobile phone text messages or alertsInstant Message from a friend or colleagueFamiliarity with Cancer.gov/National Cancer Institute (NCI)Promotional email(s) from the National Cancer Institute (NCI)Link from another websiteMobile AppsSearch engine resultsWord of mouth recommendation from someone I knowTV, radio, newspaper, or magazine advertisingInternet advertisingDon't knowOtherPatient with a cancer-related disease or condition A1A, A3A, B1AFamily member, Friend or Acquaintance of a Patient A1B, A3B, B1AHealth Care Provider/Health Professional C, D, E, C1AScientist Researcher AK, F, D1A, D1BGeneral Health Consumer AM, A1CNational Cancer Institute employee or contractorAdvocate AO, A1CMedia/JournalistPolicy MakerPublic Health OfficialStudent AQ, A1CEducator/Teacher A1CLibrarian or Information professional A1COther G

G Please specify what best describes your role in visiting Cancer.gov today.

How many times in the past 6 months have you visited the NCI Web site?

Which of the following sources drove you to visit the Cancer.gov site today? Please rank the top 3 (Rank 1 is most important).Rank 1 (Required)

In what role are you visiting the Cancer.gov website today?

Which best describes your specific role? Patient diagnosed with cancerPatient in treatment for cancerPatient diagnosed with cancer, post treatment/cancer survivorPatient with a cancer recurrencePerson concerned about being at risk for cancerOther A2A

A2A Please specify your role as a patient:Which best describes your specific role? Person concerned about someone else with cancer

Person concerned about being at risk for cancerOther A2B

A2B Please specify your role as a family member, friend or acquaintance.

B1A Are you the caregiver for a cancer patient? YesNo

A3A Give information to a family member or friendShare and talk about information with my health care providerFor my own personal useDidn't find what I wantedNot sure yetOther A4A

A4A Please specify how you plan to use the information you find on the site today:

A3B Give information to a family member or friendShare and talk about information with my health care providerFor my own personal useDidn't find what I wantedNot sure yetOther A4B

A4B Please specify how you plan to use the information you find on the site today:

Person with a health concernPerson interested in learning more about cancerOther AN

AN Please specify your role as a general health consumer:

AO Patient advocatePolicy advocate

12 Research advocateOther AP

AP Please specify your role as an advocate:AQ Elementary school

Middle schoolHigh schoolCollege/UniversityGraduate schoolPost-graduate school (including medical school)Other AR

AR Please specify your role as a student:A1C Give information to a family member or friend

Share and talk about information with my health care providerFor my own personal useUse to do homework or class assignmentUse as teaching material or for a classDidn't find what I wantedNot sure yetOther A2C

A2C Please specify how you plan to use the information you find on the site today:

C PhysicianNurse/Nurse practitionerCare coordinator/Case manager/Patient navigatorPharmacistPhysician assistantCounselor/Clinical social workerDietician/NutritionistOther K

K Please specify your role as a health professional:Are you involved in clinical research? Yes

NoD Do you specialize in the care of patients with

cancer?Yes

No

How do you plan to use the information you find on this site today?

Which best describes your specific role?



C1A

Which best describes your specific role as a student?

Which best describes your specific role as an advocate?

AM Which best describes your specific role as a general health consumer?

A1B

A1A


AppendixCustom Questions (continued)

E Keep myself up-to-date/Improve my understanding of clinical research findings Improve my understanding of basic research concepts and findingsPlan or verify cancer treatmentLearn how to become a clinical trial investigatorProvide cancer information to a patient or family memberProvide clinical trial information to a patient or family memberProvide information to other health care providersProvide information to studentsFor my personal useDidn’t find what I wantedNot sure yetOther L

L Please specify how you plan to use the information you find on this site today:

AK Which best describes your specific role? Post doctoral fellowProspective researcherFirst-time researcherNew Principal Investigator (PI)Long-time/Experienced researcherResearch manager/AdministratorOther AL

AL Please specify your role as a scientist researcher:

D1A Have you ever...? Applied for funding from NCIReceived funding from NCI

D1B Are you involved in… Clinical researchBasic researchBoth

F Learn about NCI research prioritiesImprove my understanding of basic research concepts and findingsKeep myself up-to-date/Improve my understanding of clinical research findingsDevelop a research proposal Provide information to other scientist/researchersProvide information to studentsFor my personal useDidn’t find what I wantedNot sure yetOther M

M Please specify how you plan to use the information you find on this site today:

How to prevent cancerEmotional concerns relating to cancerCancer risk factors or causes (including genetics/family history)Testing for cancer (detection/screening)How to find cancer treatment facilitiesStandard cancer treatments or therapies AAAlternative cancer treatments or therapiesManaging cancer pain and side effectsPhysical symptoms relating to cancer/cancer treatmentSymptoms related to cancer diagnosisMedicines or prescription drugsPhysicians or cancer specialistsCancer information to print out for a doctor's visitClinical trials ABQuitting smokingNews related to cancerCancer statisticsThe National Cancer InstituteCancer information for a report/paperCancer prognosis Financial concerns related to cancerResearch funding (grants, contracts, etc.) ACResearch tools and resources AEResearch training opportunities AGResearch partnership/collaborationJust browsing for cancer informationOther N

N If you answered "other" for visiting the site, please specify what information you were looking for.

A child or teenager with cancerAn adult with cancer

AB A child or teenager with cancerAn adult with cancer



Please complete this sentence: I am visiting the NCI Web site today to find information on _____ (Check all that apply).

I am looking for information about clinical trials for:

I am looking for information about the treatment of:AA

AC Are you looking for… How to apply for an NCI grantCurrent funding opportunitiesSmall Business Innovation Research (SBIR) / Small Business Technology Transfer (STTR)Research renewalSupplemental fundingOther AD

AD Please specify what other research funding information you were looking for. Are you looking for… Analytical tools

Cancer statistics/datasetsBiospecimensGenomic dataAnimal modelsChemicals/drugs/compoundsOther AF

AF Please specify what other research tools and resources information you were looking for. Are you looking for… Individual training/career development AI

Institutional training (support for large training programs)Other AH

AH Please specify what other research training opportunities information you were looking for. Are you interested in… Mentorship

Mid-career transitionPost-doctoral fellowshipMinority training programsOther AJ

AJ Please specify what other individual training or career development you were looking for.

YesPartially PNot yet, still looking R No S

P If you were unable to find all of the information you were looking for, please describe what you were trying to find.

R If you were <b>not yet able to find</b> the information you were looking for, please describe what you were trying to find.

S If you were unable to find the information you were looking for, please describe what you were trying to find.

Yes ZNoNot sure

Did you look at: Patient Health professional Both versions Neither version Not sure

From where are you accessing this site? United StatesCaribbeanEurope, including United KingdomCanada Mexico South America - ArgentinaSouth America - BrazilSouth America - ChileSouth America - UruguaySouth America - OtherCentral AmericaAustraliaNew ZealandAsia AfricaOther T

T Please specify where you are accessing this site from:

Z

Were you able to find the information you were looking for?

Did you notice that certain information is available in both patient and health professional versions?

AI

AG

AE



AppendixCustom Questions (continued)

Browsed Web sites using my mobile phone / device 1AText messaged using my mobile phone / deviceConducted an internet searchUsed personalized web pages (such as My Yahoo!)Added a widget or gadget to your personalized web page Visited a social networking site (such as LinkedIn, Facebook, MySpace etc.)Participated in virtual worlds (such as Second Life, Whyville, etc.)Used an online learning toolBookmarked or tagged websites (using social bookmarking sites as Digg or Delicious)Read blogs or WikisWrote a blog or contributed to WikisPosted comments, ratings or reviews on a Web siteListened to podcasts or audio on a Web siteWatched videos on a Web site (such as YouTube)Uploaded videos to a Web site (such as YouTube)Browsed photo galleriesSubscribed to RSS feedsUsed / posted to Slideshare Signed up to receive email updates from a websiteNone of the above

1A YesNoBeboDeliciousDiggFacebookFlickrLinkedInMySpaceNingRedditStumble UponTechnoratiTwitterWikipedia YouTubeNone/Not applicableOther

Please specify which social media site you have visited in the past 30 days.

FemaleMale17 and under18 - 2425 - 3435 - 4445 - 5455 - 6465 and over

If you could make one improvement to this site, what would it be?

In the last three months, which of the following have you done: (Check all that apply)

When you have browsed websites using your mobile phone/device, have you looked for health In the past 30 days, which of the following social media sites have you visited? (Check all that apply)

Please select the category that includes your age.

What is your gender?



AppendixMobile Portal: Access your data on the move

Key Features˃ Access an at-a-

glance snapshot of your Satisfaction, model scores and trendlines using a free iPhone / iPad or Android application.

˃ Data updated nightly

Anytime, Anywhere Access˃ Free Application˃ Supports iPhone 3,

3GS & 4˃ Android 2.0 and up


Benchmarking Your Site’s PerformanceBenchmarking provides a powerful context for your ACSI scores.

To most efficiently use benchmarks:1. Monitor your site performance on high-impact Elements identified by your Satisfaction

Research Analyst. Your most important benchmark is set by your site results. 2. Compare ranking within your industry peer group to make sure that you are keeping up with

your immediate competitors. 3. Assess your site score within the entire landscape of website satisfaction to see if you are

ahead of the game. There may be lessons that can be learned from particularly high-scoring categories.

Who’s in the Benchmarks?ForeSee Website Index:– All companies and organizations measured by ForeSee– More than 450 companies with more than 650 websites.

Benchmark SpecificityBecause ForeSee works with many different types of organizations throughout many different industries, we are able to offer over 145 categories of benchmarking to provide you with more comparability in benchmarking your scores.

AppendixBenchmark Descriptions


NIH

Health and Human Services - NIAIDNICHD-NIHNational Cancer InstituteNational Institute of Arthritis and Musculoskeletal and Skin DiseasesNational Institute of Dental and Craniofacial ResearchNational Institute of Diabetes & Digestive & Kidney DiseasesNational Library of Medicine

HHS

AHRQCMSCenters for Disease ControlDHHS Program Support Center (PSC)Dept of Health and Human ServicesFDAHHS Offi ce for Civil RightsHealth & Human ServicesHealth and Human Services - NIAIDNICHD-NIHNWHICNational Cancer InstituteNational Institute of Arthritis and Musculoskeletal and Skin DiseasesNational Institute of Dental and Craniofacial ResearchNational Institute of Diabetes & Digestive & Kidney DiseasesNational Library of MedicineOffi ce of Minority HealthSubstance Abuse and Mental Health Services AdministrationU.S. Offi ce of Disease Prevention and Health Promotion

AppendixBenchmark Participants

As of December 31, 2011


Govt Exit Survey

AHRQ Hoosier Lottery U.S. General Services AdministrationAdministrative Office of the U.S. Courts International Information Programs U.S. Pension Benefit Guaranty CorporationAlcohol and Tobacco Tax and Trade Bureau Kingdom of Bahrain UNICORAmerican Cancer Society MEDC US Department of Treasury - FLECAmerican Chemical Society MESSA US Dept of Homeland Security - CIS-WelcomeAmerican Forces Information Service MGoBlue US Offi ce of Personnel ManagementBedsider NARA - U.S. National Archives and Records Administration United States General Accounting OfficeBureau of Economic Analysis NHTSA United States Patent and Trademark Offi ce (USPTO)CMS NICHD-NIH Washington State Employee Credit UnionCPSC National Cancer Institute eRulemaking Initiative - PrivateCapitol Visitor Center National Institute of Dental and Craniofacial ResearchCareers.State.gov National Library of MedicineCenters for Disease Control National TrustChildFund Offi ce of Minority HealthCommonwealth of Massachusetts PBS KIDS SproutConsumer Reports Main Site PBS.orgDHHS Program Support Center (PSC) Partnership for Public ServiceDHS CIS Recovery BoardDOJ Main Save the ChildrenFDA SmithsonianFTC ComplaintAssistant Social Security AdministrationFannie Mae Substance Abuse and Mental Health Services AdministrationFederal Deposit Insurance Corporation (FDIC) TRICARE Management ActivityFederal Railroad Administration U.S. Department of Homeland SecurityFinancial Stability U.S. Department of StateFreddie Mac U.S. Department of TransportationGSA Auctions U.S. Department of Transportation (FHWA)General Services Administration U.S. Department of Veterans AffairsHHS Offi ce for Civil Rights U.S. Department of the Treasury, Bureau of the Public DebtHarvard Business School U.S. Dept. of Labor Offi ce of Labor-Management Standards

AppendixBenchmark Participants (continued)

As of December 31, 2011


AppendixACSI Methodology Survey Components

The ACSI methodology quantifies the performance of specified components of the site, their impact on satisfaction, and in turn, satisfaction’s impact on future behaviors. In addition, the custom questions establish a profile of site visitors and their behaviors.

Custom questions not only provide the information about key site audiences, but also allow for profiling each group with respect to the performance of the site’s components, satisfaction, and future behaviors.

Elements CustomerSatisfaction

Future Behaviors

NavigationSearch

Look & FeelContent

Etc…

Likelihood to:Return

RecommendPurchase

Etc…

RolePrimary reason for visit

Visit frequencyTask completion

Etc…

Custom Questions


American Customer Satisfaction Index (ACSI): The American Customer Satisfaction Index (ACSI) is a national economic indicator of customer evaluations of the quality of goods and services available to household consumers in the United States. It measures 7 economic sectors, 39 industries (including e-commerce) and approximately 200 companies and government agencies -- accounting for almost half of the U.S. economy. Established in 1994, the ACSI tracks trends in customer satisfaction and provides valuable benchmarking insights for the consumer economy. The Index is produced by a partnership of the University of Michigan Business School, American Society for Quality (ASQ) and CFI Group.

Custom Questions: Each Customer Satisfaction Survey includes two different types of questions: model questions and custom questions. Custom questions are unique to each survey and can take a variety of different forms, including radio buttons, drop-down menus, check boxes, and open-ended text. The custom question responses are not used to derive the results using the ACSI methodology; so custom question results are reported separately in the Online Reporting Facility. Custom questions provide an excellent opportunity for data segment analysis.

Elements: An element is a measure of the customer's experience with a defined and manageable property of a website (e.g., Look and Feel, Navigation, etc.). It is composed of several survey items that provide performance ratings about various aspects of the customer's experience with the website property being measured. The separate ratings are combined as part of the modeling process into a summary score reflecting the overall experience of the customer with the website property.

Random Sampling: We use a Sampling Percentage to determine the proportion of site visitors that will receive a survey invitation on your site. We try to attain a good balance between collecting enough information to provide you with in-depth analysis, while remaining unobtrusive to your site visitors.

Impact: Impact is the numeric representation of the cause-and-effect relationship between an element (e.g., Site Navigation) and customer satisfaction or customer satisfaction and a future behavior (e.g., Likelihood to Purchase). An impact represents the increase in customer satisfaction resulting from a 5-point increase in an element score. For example, if the impact of Site Navigation is 1.2, then a 5-point increase in Site Navigation’s score would lead to an increase in customer satisfaction by 1.2 points.

Loyalty Factor: The Loyalty Factor is a number that establishes how many pages with survey code a site visitor must visit before they are eligible to receive a survey. Loyalty Factors are utilized to ensure that respondents have experienced enough of a site before completing the Customer Satisfaction Survey.

Model Questions: In order to utilize the ACSI methodology to calculate Satisfaction and Future Behavior scores and impacts, ForeSee asks standard model questions. These model questions are targeted toward the key Elements (i.e., “main areas”) of the site that drive customer satisfaction (e.g., Functionality, Navigation, Product Browsing, etc.), Overall Satisfaction, and Future Behaviors such as Likelihood to Recommend or Likelihood to Purchase.

Future Behaviors: Future Behaviors are the things you want your customers to do more of as a result of visiting your website, such as recommend your site to others or complete a purchase. The cause-and-effect ACSI methodology enables us to quantify the impact that improving satisfaction with your site would have on increasing future behaviors such as site visitors’ likelihood to recommend your site.

AppendixGlossary of ForeSee Terminology


5.0

1.5

Y

X

Cus

tom

er S

atis

fact

ion

NavigationX

Y

X+5

Y+1.5

5-point change in Navigation = 1.5-point change in Satisfaction

Impact of Navigation on Customer Satisfaction

The rate of change between an element and satisfaction is represented by an Impact. In simple terms, an impact can be understood as something like a slope of a line defining the relationship between two variables.

AppendixUnderstanding Impacts


AppendixComprehensive Management System


Your ForeSee Team

Michelle HertzSenior Satisfaction Research [email protected]

Howard MerkelSenior Federal Account [email protected]

national cancer institute cancer.gov main site

Documents

elements of satisfaction

web satisfaction

satisfaction matter

satisfaction of site

acsi model results

en espaol survey results

priority aspect

key visitor groups