american community survey (acs) briefing on messaging and mail package assessment research f or the...
TRANSCRIPT
American Community Survey (ACS)Briefing on Messaging and
Mail Package Assessment Researchfor the
United Nations Economic Commission For Europe
Tasha Boone,United States Bureau Of the Census
April 29 – 30, 2015
Contents
American Community Survey Basics Challenges Research Goals Overview of Research Projects Focus Groups Key Message Findings Proposed Alternate Mail Designs Key Mail Package Findings Online Visual Testing & Key Findings Refined Alternate Mail Designs Key Cumulative Findings Reports Next Steps
2
American Community Survey Basics
Previously part of the 10-year Decennial Census
A large national mandatory survey that goes to 3.5 million addresses each year to produce over 11 billion estimates
Designed to produce critical information on small areas and small populations
Covers over 35 topics that cut cross demographics, economic, social and housing
Data Collection Modes: Internet/Web Mail Telephone Personal Visit
3
Month 1: Online ResponseMonth 1: Mail Response
Month 2: Computer-Assisted Telephone Interviewing (CATI)
Month 3: Computer-Assisted Personal Interviewing (CAPI)
Challenges
Lack of general awareness of the ACS
Confusion over the difference between ACS and the Decennial Census
Concerns from respondents/public about perceived intrusiveness and burden of the survey
A re-examination around making the ACS voluntary
Marketing a survey
44
Research Goals
Develop and test messages and mail package refinements to increase ACS self-response rates
Apply insights from ACS message testing to support general outreach, data dissemination, materials development, and call center and field operations
Help inform the 2020 Census Integrated Communications Program
5
Overview of Research Projects
Messaging Research1. Mental models interviews with internal stakeholders 2. Key informant interviews with stakeholders 3. Focus groups 4. Message testing - Benchmark survey 5. Message testing - Refinement survey
Mail Package Assessment6. Creative design of mail package alternatives and messages to test 7. Focus groups and one-on-one interviews 8. Quantitative online visual testing
Experimental Testing - Conducted by Census in 2015
Independent reviews – Don Dillman, PhD (survey methodologist), United States Census Bureau Advisory Committees
6
Messaging Research and Mail Package Assessment Focus Groups Focus groups were held for message testing and mail package testing
Focus groups were held in seven cities. Selected cities based on demographics, low response rates, and skepticism about the survey
The cities were diverse in their geographic and racial composition. Participants recruited from the following communities: Hispanic; African-American; Asian; American Indians living on tribal lands, and low-income
Each focus group had approximately 24 to 28 participants
The mail package focus group included 30-35 one-on-one interviews that spanned 45 minutes to 1 hour
7
Key Message Findings
America knows Census—but not American Community Survey
Why would the government need to know this?
The government has this information
Distrust of government is pervasive
Community is key
Language barriers
Good tools and training are key in the field
Local influencers are trusted messengers
8
Design of Alternative Mailing Packages
Description Use learnings from messaging research to inform language and designs
Approach Five initial themes presented, narrowed to three after input from Census
9
“Community” “Official” “Patriotic”
“Personal” “Historical”
Current ACS Design vs “Community” Mailing Design
Used messaging and mail package research, feedback from Dr. Don Dillman and Census advisory committee feedback to inform changes to language
Current ACS Envelope
“Community”
Proposed initial design
10
Current ACS Design vs “Official” Mailing Design
Current ACS Envelope
Proposed initial design
“Official”
Proposed initial design
11
Current ACS Design vs “Patriotic” Mailing Design
Current ACS Envelope
Proposed initial design
“Patriotic”
12
Current ACS Design vs “Personal” Mailing Design
Current ACS Envelope
Proposed initial design
“Personal”
13
Current ACS Design vs “Historical” Mailing Design
Current ACS Envelope
Proposed initial design
“Historical”
14
Key Mail Package Findings
The current package was seen as most effective
Consider a mix of effective design elements The name “American Community Survey” does not register
with the public
More direct messages that tell the respondent what action you want them to take
15
Current ACS Design vs Revised “Community” Designs
Current ACS Envelope
Design of “community mail design after message testing Design of “community mail design after mail package focus group testing
16
Current ACS Design vs Revised “Official” Designs
Current ACS Envelope
Design of “official” mail designafter message testing
Design of “official” mail design after mail package focus group testing
17
Current ACS Design vs Revised “Patriotic” Designs
Current ACS Envelope
Design of “patriotic” mail designafter message testing Design of “patriotic” mail design after mail package focus group testing
18
Online Visual Testing Survey Design
Mail Sort Exercise: an online simulation selection of print mail to assess what designs were noticed, opened, and recalled
Image Click Analyzer: explored how participants interacted with mail designs by asking respondents to click on areas of designs which caught their attention
Message Highlighting Tool: instructed respondents to highlight the words and phrases in the text of ACS letters that they found most compelling
19
Key Findings from the Online Visual Testing
Visual design elements and deadline messaging can have a significant impact on how people read and remember mail items
The U.S. Census Bureau logo should be prominently featured on mail items
Key words and phrases in letters should be emphasized using callout boxes, line spacing, and bolded text
“Your response is required by law” attracts more attention than any other message
It is possible to overdo the commercial “marketing” look and feel
20
21
Proposed Mail Designs for use in Field Testing
“Official” mail package: Builds on successful elements of existing ACS mail package Introduces improvements to visual design and messaging suggested by our
research
“Official” “Control”
Looks very similar
22
Proposed Mail Designs for use in Field Testing
“Blended” mail package:
Incorporates successful aspects of “Official” design with evocative visuals of “Patriotic” theme
Intersperses straightforward “governmental” pieces with more designed elements to catch recipients who may be drawn to more eye-catching pieces
“Blended” “Control”
Cumulative Findings
1. Emphasize the highly favorable Census brand in ACS materials
2. Use visual design principles to draw attention to key messages and help respondents better navigate materials
3. Use “governmental” appearance over a “marketing” approach
4. Emphasize effective “mandatory” messaging5. Demonstrate benefits of ACS participation to local
communities
23
Twenty eight cumulative findings……here are five key ones:
Reports
The following reports are final: Mental Models Interviews with Internal Stakeholders Key Informant Interviews with Stakeholders Deliberative Focus Groups with Stakeholders Message Testing: Benchmark Survey Message Testing: Refinement Study Mail Package Research
Access reports at: http://www.census.gov/acs/www/library/by_year/2014
24
Next Steps
Continue to research and test changes to messaging and the mailing package materials
Results will be used to enhance the mailing package, the data collection instrument and materials, and the survey educational materials
Continue to measure trends over time
25
Continue the ACS ConversationConnect on Social Media:
@uscensusbureau
facebook.com/uscensusbureau
Email:
Receive email updates: https://public.govdelivery.com/accounts/USCENSUS/subscriber/new
Find more information:
census.gov/acs
acsdatausers.org
26
27
Questions