ANALYSIS OF OUTREACHIN THE

RADON AND INDOOR AIR PROGRAMS

Region 10Policy, Planning and EvaJuation BranchMay, 1993

Background

Region 10's Radon and Indoor Air Program managers requested the Policy,Planning and Evaluation (PP&E) Branch to conduct an analysis of theeffectiveness of their outreach activities. The main goal was for PP&E to helpthe Radon and Indoor Air Program (RIAP) determine how they can optimize effortsto direct their messages to specific target audiences. Analysis success is thedevelopment of recommendations and information that allows RIAP to measure anddemonstrate that they are directing their limited resources in ways thatmaximize the reduction of risk.

The RIAP wanted recommendations that would aid efforts to target their outreachtoward geographic areas, audiences, and pollutants with the greatest risk.Overall Program guidance is established by HQ, but the Region has a wide rangeof methods available for disseminating and implementing the guidance. RIAP haslimited resources.

The PP&E Branch was to develop recommendations that can be implemented withoutforeseeable increases in resources, .and recommendations for RIAS timeallocation based on quantitative measures of program impact. RIAS requestedpreliminary results available in time to incorporate into their planning forFY94.

2

\

•••

TABLE OF CONTENTS

I. Executive Summary

Page4

II. Radon and Indoor Air Pollution 5

III. Bennett Model 8

IV. Explanation of Bennett's Hierarchical Steps 10

V. Bennet Model Applied in Washington 13

VI. Outreach in Other Government Agencies 17

VII. Outreach in Region 10 18

VIII. Recommendations 19

Appendices:

Appendix 1 22

Appendix 2 24

Appendix 3 26

EXECUTIVE SUMMARY

This analysis is intended to aid Region 10' s Radon and Indoor Air Programmanagers in their efforts to determine the effectiveness of their outreachactivities. The Radon and Indoor Air Program (RIAP) strives to optimize theirefforts to successfully target audiences and deliver messages to protect the public.The analysis presents recommendations and information that will help RIAP to measureand demonstrate that they are directing their limited resources in ways thatmaximize the reduction of risk and public protect.

\

The dichotomy between outreach that is "shotgunned" and that which is moreprecisely targeted need not necessarily be resolved entirely in favor of one or theother. This analysis will not offer recommendations that support or reject eitherapproach. A program that includes both approaches to outreach is much morerealistic. These recommendations are intended to suggest different approaches thatare not exclusive of one another. The follow"ing recommendations are offered toaddress the analysis goals (all recommendations are discussed in more detail in theRecommendations section):

1. Maintain a base program of general public information.

2. Build feedback/measurement systems into our grants and contracts.

3. Begin conducting our own feedback/measurement activities.

4. Develop strategic plans with our grantees and/or the states.

5. Develop an outreach capability within the Office of External Affairs.

Recommendations

Recommendation 1: Maintain a base program of general outreach designed to increasecontacts with the general public and with key participants.

Radon/Indoor Air Quality programs are in their infancy. A general disseminationof information will help to develop an infrastructure of informed people who canparticipate in a slow but steady shift of opinions and behavior. This approach isconsistent with the patterns followed by our grantees and contractors -- mainly thestates -- who have the power to reach more people than we do. Too narrow of anapproach reduces the number of potential allies who may at some point, through theirpurchasing habits, votes, word-of-mouth, or other behavior, contribute to betterindoor air quality.

Recommendation 2: Build feedback/measurement systems into EPA grants and contracts.

Disbursed funds are a primary lever to achieve better indoor air quality. Thisfunding is an excellent opportunity for determining how well our outreach is workingwithin the public sector.

This Recommendation advocates building limited feedback and measurement intogrants. Instead of devoting the entire grant to general public information, thegrantee can craft a proj ect of limited scope that lends itself to design andmonitoring.

This approach requires ga i.m.ng t;heapproval and cooperation of the grantee, whichmight be a challenge given their limited resources and preference for general public

4

information rather than targeted campaigns. EPA might gain cooperation by makingits staff more available.

Recommendation 3:activities.

EPA Region 10 can conduct its own feedback/ measurement

To date there have been many public presentations, training sessions,informational inquiries, distribution of materials, and various other people whomight be characterized as members of a radon/indoor air "infrastructure." We couldsurvey them to find out how useful our information was and how it led to changes inbehavior. More specifically, we could poll real estate brokers to develop a database of tested homes. We could also query radon testing and mitigation contractorsto determine their involvement in reducing radon risk.

Recommendation 4: EPA Region 10 can developing strategic plans with its granteesand/or the states.

While federal, state, and local government may still be struggling through thestart-up phase of radon and indoor air management, we can look ahead. This is auniquely federal role, given our four-state perspective, the nation-wide effort atthe national level, and the leverage we possess as grantor. Programs and projectsthat lend themselves to feedback and measurement may stretch over longer periodsthan the typical one-year grant. We need to persuade others to work with us todevelop a sense of what priorities are needed within our programs and how we canachieve measurable results.

This proposal is not limited to the states; there are non-profit organizations,such as the Washington Lung Association, who are potential collaborators. We canconsider forming alliances, in the form of cooperative agreements, with majorcollaborators. This encourages a common solution, as well as allows leveragedresources. Cooperative methods, driven by strategic planning, allow for feedbackand measurement of everyones' effectiveness.

Recommendation 5:Affairs.

Develop an outreach capability within the Office of External

Neither Radon nor Indoor Air have had much involvement with the Office ofExternal Affairs. There has been some press interest, but the interest fluctuates.Press is, by definition, concerned with the news value of an occurrence. Outreachis a continuing, rather than an episodic, activity. Outreach can be augmented byprofessional capabilities (writing, graphics, education) which the Office ofExternal Affairs could provide as needed by Radon, Indoor Air, and other programs.

The Policy, Planning and Evaluation Branch believes this analysis can be usefulto the Radon and Indoor Air Programs. Further analysis and/or support is availableshould the program desire it.

5

II. Radon and Indoor Air: National Program Design

The Indoor Air program was initiated by the Radon Gas and Indoor Air Quality Actof 1986, Title IV of the Superfund Amendment and Reauthorization Act (SARA). Thelegislation established a research program to identify, characterize, and monitorthe sources and levels of indoor air pollution, investigate the human health risks,and to develop control technologies or other mitigation measures. The statute alsoprovided for disseminating information to the public about the research activities.

EPA has expanded on the statutory mandate through its FY 93 program guidance,which emphasizes outreach activities at the Headquarters level: research,preparation of guidance documents for physicians and architects, documents fortargeted segments of the public, an information clearinghouse, written materials onassessment and mitigation with private sector organizations, and training. Theregional office role includes public information, technical assistance, training,and state and local program development.

The Indoor Radon Abatement Act, passed in 1988, relies on similar measures. Itprovides for the publication of a citizen's guide that outlines the health risk fromradon, mitigation costs and techniques, and testing methods. EPA is furtherdirected to develop model construction standards and provide technical assistanceto state radon programs, including training, an information clearinghouse,demonstration programs, and publication of public information materials concerningradon health risks and methods of radon mitigation.

A key aspect of the radon program, at least from the perspective of EPA regionaloffices, is state grant funding which is authorized to states for building andgeographic surveys, and purchase of measurement and analytic equipment. Otherpotential activities include development of public information and educationalmaterials, developing data systems, mitigation demonstration programs, and toll-freeradon hotlines to provide information and technical assistance.

EPA Headquarters guidance for FY 93 identifies five elements to be addressed bythe radon program:

1. Problem Assessment (identify areas with high radon levels)2. Mitigation and Prevention (methods to reduce or prevent radon contamination)3. Capability Development (improve state, tribal, private sector, and general

public in dealing with radon)4. Public Information (provide timely information about radon risk and methods

of risk reduction)5. Federal Coordination (work with other federal agencies and buildings)

While public information is presented as a separate prong of this five-partstrategy, the last three of these elements arguably constitute outreach activities.The guidance also iden-tifies regional office roles: overseeing regional trainingcenters, promoting new building codes; overseeing state grants; promoting realestate guides and addressing radon problems during real estate transactions;representing EPA to state, local, tribal, and private sector; providing technicalsupport; and expanding public outreach activities. This last role includesdeveloping public education materials (public service announcements, brochures,posters), providing EPA publications to states and tribes, and promoting modelbuilding codes to state and local governments. Again, public outreach is anenumerated function, but all of the listed regional office roles probably constituteoutreach.

6

r

It is important at the outset to settle on a definition of outreach. Anexpansive definition may be best since almost every activity in the radon and indoorair programs constitutes outreach. The term encompasses more than just publicinformation activities (e.g. production and distribution of written information,public service announcements, press releases). Training, grants, promotion ofchanges in building codes or real estate agent licensing requirements -- all areopportunities to educate and persuade, which I believe is what we intend when weemploy the word "outreach".

The national program structures for radon and indoor air invite some conclusions:

• They are voluntary programs that rely on research and outreach rather thanregulatory strictures.

• They reflect a national (some would say Beltway) approachthat emphasizes broad, sweeping measures.

to policy-making

• They provide no guidance on the precise nature of the outreach to be conductedby the regional offices. They offer none of the precision and narrow focus thatis conducive to gathering useful feedback and measuring program effectiveness.On the other hand, this leaves the regional offices with enormous latitude intheir outreach practices.

• Both statutes and programs are first-time, start-up efforts that forsake theregulatory approach in favor of behavior change by promoting "a better way"through education and persuasion.

EPA has further fleshed out its program goals through a radon program analysisto determine how well the program has been working. Draft Radon Program Review,Office of Policy, Planning and Evaluation, Environmental Protection Agency, May1992. The study found that EPA's radon program had raised public awareness, madeprogress in testing and mitigating, supported state programs, and collected data onradon distribution and health effects. However, the study found that only limitedtesting and mitigation had occurred; about six percent of homes had been tested andeven fewer of those needing mitigation had been mitigated. The analysis furtherfound that, although early program success compared favorably with similar effortsin anti-smoking, drunk driving, seat belt and crime safety, EPA wanted to have agreater impact on radon health risks. The study made four major recommendations:

1. EPA should focus on the greatest risks first (high radon potential areas andsmoking).

2. EPA should emphasize the long-term strategies of promoting radon-resistant newconstruction and testing/mitigation in connection with real estate transfers.

3. EPA should develop a coordinated research plan.

4. EPA should move beyond a nation-wide public information program and developa new strategy for public information, one that emphasizes high radon potentialareas and that focuses on building support for construction of radon-resistantnew homes and for testing/mitigation of existing homes when they are sold.

The fourth recommendation, regarding public information, directly concerns ushere. The study made several findings about public information, including:

7

• Most radon testing (70-90%) occurred in the context of real estatetransactions.

• State and local public information and incentive campaigns have yieldedlimited responses. A public information effort in Washington, D.C., resulted ina one-time testing rate of about four percent. Local direct mail and free ordiscount testing kit distribution have resulted in one-time testing rates of 15-30%.

• Government has a primary role of informing the public, particularly becauseradon is imperceptible without a public information effort.

• A foundation of public awareness is necessary to reduce radon risk.

• EPA's public information campaign has raised public awareness; 80% of thepopulation has heard of radon and 65-70% understand that it poses a health risk.

• Nevertheless, public information alone is not sufficient to make a significantimpact on radon risk reduction; EPA should be moving beyond a nation-wide publicinformation strategy.

• The localized nature of radon limits the federal role; statesbut federal energies can complement through research, pilot andprojects, and support for state activities. Other healthorganizations, at all levels, also have roles.

have the leaddemonstrationand consumer

The Headquarters analysis sends mixed signals about outreach. On the one hand,the report finds a need to move beyond a broad-brush public information program andidentifies a need to narrow the focus of outreach and recommends where it can bedone -- real estate transactions. This attempt to isolate an aspect of the overallradon problem seems consistent with the Bennett Model (see below), which works bestwhen the results sought are narrowly drawn.

8

III. The Bennett Model

A blueprint for planning and measuring outreach activities was developed byClaude Bennett of the Department of Agriculture in 1974. Bennett was trying toanalyze the effectiveness of the numerous outreach programs administered throughCooperative Extension. Washington State government has found the model useful; boththe Department of Wildlife and the Puget Sound Water Quality Authority have appliedit to their programs.

In creating the model, Bennett wanted to provide a useful tool for programmanagers to know whether their program was succeeding. Bennett's model assumes thatprogram evaluation (what we are calling measurement of the effectiveness ofoutreach) is built into every program from its outset, instead of juryrigging ameasurement system sometime down the road to provide hindsight.

The model consists of seven hierarchical steps. Each step represents a categoryof outcomes which, in turn, allows us to develop criteria for evaluating theoutcomes. The seven steps are arranged along a continuum, with "ends" at the higherend and "means" at the lower end. Each step is a prerequisite to the next, in termsof program design. However, actual program planning is conducted in reverse -- youestablish your desired results and work backward toward inputs. The essence of themodel is that you cannot design a program unless you know in advance, withspecificity, what your program is supposed to accomplish.

Bennett's Model is a simple one; it organizes obvious program components in alogical progression in a way that encourages a program planner or manager to see theconnections and integrate them in a result-oriented fashion.

[Bennett's Model is illustrated on the next page.]

9

I

Bennett's Model*

Results

(which are reached by)

Behavior Changes

(which result from)

KASA (Knowledge - Attitude - Skill - Awareness

(which are influenced by)

Reactions

(which are a function of)

Participants

(which happens due to)

Activities

(which are offered as a result of)

Inputs

* Designing an Effective Communication Program: A Blueprint forSuccess; University of Michigan: School of Natural Resources andEnvironment; Prepared for U.S. Environmental Protection Agency, Region5; September 1992; page 60. '

10

IV. Explanation of Bennett's Hierarchical Steps

Results: This step asks the essential question, how do you know whenyou have done your job? Someone must identify a problemand then enunciate a goal so that we know when the problemis solved. An important question to be resolved is, whodetermines the goal (s) -- policymakers or those chargedwith conduct of the outreach effort?

There are three kinds of results:

1. Environmental -- a change in the condition of the environment(e.g., a renewed fish run; a measurable improvement in opacity; a banon lead in gasoline).

2. Protective mechanism in place -- compliance with a measure that isdeemed to yield some benefit (e.g., obtaining a Clean Water Act §404 permit;implementation of the National Contingency Plan for Superfund sites).

3. Institutional capacity -- this is a pseudo-result, where a regulationis approved or a program is fully funded (e.g. listing a Superfund site onthe National Priority List)

When program design is initiated at the "Results" stage, the goal can beexpressed in terms of qualitative or quantitative measures. Much of EPA'soutreach appears to have considered this step in only the most general way --if we disseminate information widely then people will smarten up and we willhave a better environment. A glance through the Bennett model reveals howthis method has skipped many steps that provide the logical connectionsbetween program and results. EPA outreach resembles the reproductive methodsof many plant and animal species: scatter enough seeds and some will takeroot. While this may work for Douglas fir, it may not reflect the mostcareful husbandry of public resources, nor does it protect public health inthe most comprehensive or efficient manner.

Behavior Change:

This involves identification of some durable, persistent institutional orpersonal behavior change (e.g., stop smoking; buy recycled materials).Program designers should ask, what are the cultural enforcers or inhibitorsof behaviors? Behavior change can be difficult to measure it isimpossible to know how many have quit smoking. Then we need to look foranalogue or indicator behavior -- cigarette sales, expansion of anti-smokinglegislation. Follow-up may be necessary to determine behavior changes, suchas telephone surveys.

KASA (Knowledge - Attitude - Skill - Awareness)

This is the information component of program design, where we consider whatpeople think and know and what they should be thinking and knowing in orderto alter behavior and achieve program results. Ideally, we conduct a pre-test of KASA to determine what people knew prior to program implementation.A post-test after the program should then tell us the effect of the program.These are very resource and time-intensive measures.

11

What is essential at this stage of program design is that a clear, readable,technically accurate body of information be compiled. It may have toencompass a range of values in order to give everyone targeted what they needto change behavior.

Often, program design fails to go past this step in determining programeffectiveness Behavior Change and Results are not considered.Unfortunately, without feedback about what the outreach has accomplished, weare just "bean-counting" (e.g., we published X brochures and we reached apotential audience of Y through radio public service announcements).

Reactions

For outreach efforts to work, we have to create a climate of learning:timing, surroundings, consider the feelings of participants, appropriatebehavior by "teachers." Everything must be structured to achieve "buy-in" byparticipants.

Participants

The outreach audience must be targeted with prec1s1on. The larger and morediverse the audience, the less likely a message can be tailored that achievesthe behavior changes that will produce the result wanted. If the audience isthe general public, we will likely be reduced to a general "shotgunning" ofinformation accompanied by a leap of faith that appears to be based on thetheory that we know the facts and we understand the health risks, and,therefore, if we publicize what we know, everyone will know this and actwisely to protect their health. There is no evidence that broadbrushoutreach accomplishes this.

For an identified audience, we want to know what "pushes their buttons", howthey can best be approached. Perhaps affinity groups (labor unions,churches, professional societies) can be used as conduits to reach anaudience.

Activities

This step details the tools or levers available to augment a program:workshops, conferences, technical assistance, toll-free hotlines, brochures,speeches, testimony, etc. Activities encompasses more than the capabilitiesof our own agency; can we leverage additional performance through grants,contracts, cooperative agreements?

These activities can be arranged into a timeline to organize the entireprogram and envision what will occur.

Enputis ; The resources to be applied to the program include FTE, extramural funding,volunteers, etc. In a program planning sense, this step of the hierarchyrequires budgeting for results. Having designed the program, committingto outcomes, and identifying the resources will eventually lead to thoseresults.

12

Bennett recognized the limits of quantitative measurement -- the difficulties ofmeasurement and the substantial resource implications. In response, he alsodeveloped a qualitative measurement system for program evaluation. He called thisReflective Appraisal of Programs (RAP); program participants "reflect" upon thechanges effected by a program. In essence, participants are asked to respond to astandard questionnaire keyed to any of the four "higher end" outcomes of Bennett'shierarchy -- Reactions; K.A.S.A. Change; Behavior Change; and Results.

The RAP approach encourages use of the findings from the interviews to drawconclusions about and evaluate results of the program. These conclusions then leadto recommendations about how decision-makers can apply the findings.

The Bennett Model offers a systematic, logical, easily understood framework forprogram design. Its emphasis on backwards planning, with project results servingas the key to project design, encourages a focus on project feedback or measurement.

13

V. The Bennett Model Applied in Washington

Two agencies of Washington state government have applied the Bennett Model indifferent ways. The Department of Wildlife conducted a retrospective analysis ofits Project WILD, an environmental education effort that had been in operation foreight years. The goal of Project WILD was to

"...assist learners of any age in developing awareness, knowledge, skills, and.commitment to result in informed decisions, responsible behavior, andconstructive actions concerning wildlife and the environment upon which alllife depends." Evaluation of Project WILD: The State of Washington 1984 -1992, by Margaret Tudor, Washington Department of Wildlife, 1992.

When measured against the Bennett Model, Project WILD was found to have many ofthe preliminary prerequisites for program success. For example:

Inputs: 1 or 2 FTE throughout program life; 1,000 elementary guides, 300secondary guides, 300 aquatic guides produced annually.

Activities: 376 introductory and advanced workshops held; 2 conferences held(total attendance 800).

Participation: 6,300 teachers (15.5% of total) participated.

Beyond this point in Bennett's Hierarchy, program evaluation of Project W+LD tendedto be more anecdotal:

Reactions: Survey found response to workshops favorable; teachers reportedcurricular resources useful.

K.A.S.A.: Teachers report change in students' attitude, but students',knowledge, attitude and skill change not measured.

Behavior change: No strong evidence directly linking Project WILD withwildlife protection and habitat enhancement; anecdotal evidence that advancedworkshops using Project WILD as resource has led to habitat enhancement by.teachers.

Results: Project WILD -- no effects measured; anecdotal evidence that habitatenhancement education program linked to habitat grants protected or enhanced3,000 acres.

The analysis concluded that Project WILD needed further development of its goals,and that the goals had to be expressed in terms of measurable outcomes. Targets tomeet the goals needed to be set at the outset of the program.

14

The Puget Sound Water Quality Authority (PSWQA), originally established by theLegislature as an independent agency and now a part of the Department of Ecology,has conducted an ambitious outreach program for several years. Through the PublicInvolvement and Education (PIE) Fund, PSWQA has made an experimental effort toinvolve the public in managing Puget Sound through a number of creative outreachactivities.

PSWQA conducted a study of 42 PIE Fund projects, measured in terms of Bennett'shierarchy. Publicly Financed Voluntary Initiatives for Public Involvement andEducation as a Natural Resource Management Tool, An Evaluation 1989-1991 WashingtonState Public Involvement and Education Projects, Funded by the Puget Sound WaterQuality Authority, July 1991. The study include the following findings:

• All projects (100%) fulfilled Levels 1,2 and 3 (Inputs; Activities;Participation) in Bennett's model.

• 90% of the projects were fulfilled through Level 4 (Reactions).

• At Level 5 (K.A.S.A. changes) 71% were fulfilled.

• 33% reached Level 6 (Behavior/Practice changes).

• 14% reached Level 7 (Results) -- some form of water quality protection orimprovement.

The PSWQA and Department of Wildlife studies cry out for additional analysis --so here goes. The lower end, early stage steps in Bennett's hierarchy are morereadily quantifiable. At that point in program planning we are measuring resourcelevels, numbers of people, workshops, conferences, and hotline calls answered.

As we move up the scale away from "means" toward "ends", our methods ofmeasurement are few -- we can conduct surveys, send out questionnaires. Thesemeasurements tend to give us qualitative, subjective responses which usually tellus that we are having some effect; but we have no way of knowing whether we haveorchestrated an efficient use of public resources. Low return rates on thesemeasurements leave us questioning whether we have a representative sample or whetherwe really even know what effect our program has had. It is even more difficult todetermine what behavior changes we have effected; notice how the results in thePSWQA study drop from 71% (K.A.S.A. changes) to 33% for Behavior changes. TheProject WILD study, while not as quantitatively precise, shows a dramatic change inthe nature of the outcomes at the higher end of Bennett's scale.

In both studies, the evaluations of Results suggest that the projects ran out ofgas; Project WILD had inadequate anecdotal information, and PSWQA found that only14% of the projects fulfilled their purpose. Some possible explanations for thisdropoff in measurable performance are:

• Program design has not included more precise measurement.

• Environmental and natural resource outreach proj ects are inherentlydifficult to measure if they focus on the effects on the naturalenvironment. Puget Sound or the ambient atmosphere are too big and thereare too many potential intervening factors that may mask our projects,however well-intentioned, -designed, and -implemented they may be.

15

• Environmental outreach may be difficult to measure during the life of aproj ect. Proj ects tend to reflect funding cycles and budgetappropriations. Perhaps measurable effects are longer term phenomena thatdo not readily fall within these timeframes.

• Perhaps it is too early to tell. Many government outreach programs arestill in their infancy. Managers and staff may not have been trained inthe importance of outreach or in its intricacies. Development of outreachskills may elevate the outcome of future projects.Measurable results maynot be available until some time in the future.

• Measurement at the higher end of Bennett's hierarchy requires substantialresources. It may involve significant socialfbehavioral science fieldwork. At the KASA level this involves the art of learning assessment.Project evaluation of this magnitude is normally beyond the means (fundingand time) of public agencies.

• The lack of quantifiable data at the higher end of the model may simplyreflect a bias of environmental/natural resource agencies: our naturalbent is to look for things to measure. When we cannot find them, weregard whatever feedback we get as suspect or inadequate. Maybe we needto reassess our own attitudes and accept qualitative evaluations.

• There may be a form of entropy in human affairs that applies the longeran undertaking lasts, or the greater the span of control among theindividuals involved. In government agencies, resource cuts, personnelturnover, and shifting priorities may take the bloom off the rose beforea proj ect is completed. Outreach depends on influencing people andactivities that are not within the control of a government agency andwhich have other demands on their time, enthusiasm, and loyalty. Perhapsthere is simply a natural enthusiasm at the outset of a project that isinevitably drained <}:waybefore completion. Perhaps measuring programsuccess requires more, or different energies and attitudes, than settinga project in motion does.

16

• Program design, on the macro level, may inhibit the accomplishment ofgoals. For example, one of the Radon program's primary tools is grantfunding to the states. However, federal law requires a 50% state matchfor federal funding. Scarce radon funding on the state level has resultedin less than 100% of available federal grant money being awarded in Region10. If we are not using all of the admittedly low level of programresources, it should not surprise us that our outcomes fall short.

PSWQA has taken steps to overcome the difficulties of measuring outreacheffectiveness. The Authority has designed a questionnaire for use by PIE Fund grantrecipients to analyze the effectiveness of a grant. (See Appendix 1) PSWQA usesa pre-proj ect and post-proj ect questionnaire for grant recipients. The twoquestionnaires differ essentially only in verb tense. They are closely keyed to theBennett Model. The questionnaires are designed to have grantees thinking in termsof results at the inception of the project, and to reinforce the importance ofthorough program design as the project is completed.

17

VI. Outreach in Other Government Agencies

Most of the EPA regional offices, along with a few other government agencies,were informally surveyed by telephone to determine what outreach techniques theywere using arid their success in measuring outreach effectiveness. (See Appendix 2)As far as outreach methods, nothing novel emerged -- they were using anticipatedtools to predictable audiences. Almost without exception they claimed no successin measuring outreach. The Great Lakes Program (EPA, Region 5) relies on acarefully designed survey of visitors to a ship that sails among Great Lakes portsto ascertain visitors' knowledge of water quality issues. They are surveying acaptive audience for information purposes only; no behavior change is sought. Thereis little applicability of that effort to radon and indoor air. A few of the otheroffices surveyed count basic information such as the number of people reached byvarious outreach methods, but none has found a way to measure changes in behavior.Those offices polled expressed an interest in this study as a means of findingbetter ways to chart their effectiveness.

18

VII. Outreach in Region 10

Region 10 performs an impressive amount of outreach in the Indoor Air and Radonprograms. Staff are fielding public inquiries, developing contacts in public andprivate sectors, organizing and presenting training, meeting with key groups,circulating written materials, appearing at horne shows, etc. There have beenattempts to target audiences, such as appearances before real estate brokers'associations regarding radon, which is consistent with the recommendation in theHeadquarters Radon Program Review that real estate transactions be a primary focusof outreach. Staff are maintaining mailing lists and identifying attendees atvarious events. There has been no lack of energy in trying to promote bothprograms.

What is less apparent is whether the programs are using a measured, result-oriented approach in their outreach -- the Bennett Model or something similar.Their efforts seem more concerned with reaching as many different audiences in asmany different ways as possible, rather than targeting an audience with a particularresult in mind and then trying to measure the result. For example, the Indoor AirCoordinator participated in the presentation of the training program "Orientationto Indoor Air Quality" in Spokane. If this particular outreach event is analyzedin light of the Bennett Model, we discover the following:

Inputs: The Region's lone program coordinator (1 FTE) disseminatingprogram knowledge.

Activities: Three-day interactive training program.

Participants:Federal, state, and local employees, representing a variety of agencies,non-profit organizations, and some private sector representatives, manyfrom Washington and Idaho.

The implicit purpose of this training is to impart information, raiseconsciousness, develop networks of information and allies, and generally bring keyindividuals to a higher level of understanding of indoor air issues. However, thiseffort did not include any attempt to determine the Reactions of the participants,the Knowledge - Ability - Skills - Attitudes of the participants before and afterthe training, their Behavior Changes, and the program Results of the training. Anattempt to measure the effects of the training would have included these steps. (Nocriticism of any individual is intended by this analysis; it is offered solely tosuggest program improvement, or at least to illustrate a different approach.)

The traditional broadbrush approach to outreach and the more focused approachoffered by the Bennett Model highlight the fundamental tensions in performingoutreach, which is the main point of the Recommendations offered here.

19

VIII. RECOMMENDATIONS

Recommendation 1: Maintain a base program of general outreach designed toincrease contacts with the general public and with key participants.

Government has an educational function. Increasing the number of educatedpotential allies who may at some point, through their purchasing habits, votes, orother behavior, contribute to better indoor air quality, is an educational role.

Radon/Indoor Air Quality programs are in their infancy. A general disseminationof information to develop an infrastructure of informed people who can participatein a slow but steady shift of opinions and behavior. This approach is consistentwith the patterns followed by our grantees and contractors -- mainly the states --who have the power to reach more people than we do.

We can explore ways to make this general information function more efficient thanit is. Perhaps we can make morejbetter use of Senior Executive Employees (SEE) bygiving them primary responsibility for responding to general telephone inquiries.This frees the program coordinators to perform new duties suggested within theseRecommendations.

An alternative that could produce a similar result would be to use the PublicInformation Center in the Office of External Affairs for answering public inquiries.This would require giving PIC staff more training and more authority in answeringquestions, instead of only furnishing the public with publications.

There are other difficulties: these are contractor personnel and we would haveto decide whether contractors should be performing this function. On the otherhand, this might be a preliminary step in creating a more omnibus public informationfunction in OEA, which would broaden that Office's involvement in Regional affairsand free program staff for other, more narrowly focused outreach activities.

Recommendation 2:contracts.

Build feedback/measurement systems into EPA grants and

Disbursed funds are a primary lever to achieve better indoor air quality. Thisfunding is an excellent opportunity for determining how well our outreach is workingwithin the public sector.

Appendix 3 is a summary of the Washington State Department of Health's activi tiesunder an~PA grant in the Radon Program for a one-year period. Staff made a hostof presentations, attended meetings, drafted and updated fact sheets, operated atoll-free information line, issued press releases, and published articles.

20

I

But very little of their activity shows any results. True, they did have someinvolvement in the passage of legislation that requires local building officials toprovide radon testing devices in new home construction. That is a result of sorts;of course, a more meaningful result would describe the mitigation that occurred asa consequence of testing.

Other activities suggest tantalizing possibilities of feedback/measurement. Theyconducted a testing survey and mailing survey (of what? to what end?). They backedaway from a plan to test and mitigate economically disadvantaged homes when thecontractor failed to produce matching funds. Instead a general public informationeffort was introduced.

This Recommendation advocates building limited feedback/measurement into grants.Instead of devoting the entire grant to general public information, the grantee cancraft a project of limited scope that lends itself to design and monitoring asenvisioned by the Bennett Model. For example, isolation of real estate transactionswithin a given area, involving local realtors, offering test kits or services at areduced rate, conducting public information campaigns, and then measuring tests atthe time of real estate transactions can create feedback.

This approach requires gaining the approval and cooperation of the grantee, whichmight be a challenge given their limited resources and preference for general publicinformation rather than targeted campaigns. EPA might gain cooperation by makingits staff more available (with the time available through the use of Office ofExternal Affairs (OEA) personnel to field- inquiries -- see #l above).

Recommendation 3:activities.

EPA Region 10 can conduct its own feedback/ measurement

To date there have been enough public presentations, training sessions,informational inquiries, distribution of materials, and various other people whomight be characterized as members of a radon/indoor air "infrastructure." We couldsurvey them to find out how useful our information was and how it led to changes inbehavior. More specifically, we could poll real estate brokers to develop a database of tested homes. We could also query radon testing and mitigation contractorsto determine their involvement in reducing radon risk. Again, the available stafftime through public inquiries fielded by individuals other than program staff couldallow focus on this activity.

21

Recommendation 4: EPA Region 10 can developing strategic plans with its granteesand/or the states.

While federal, state, and local government may still be struggling through thestart-up phase of radon and indoor air management, we can look ahead. This is auniquely federal role, given our four-state perspective, the nation-wide effort atthe national level, and the leverage we possess as grantor. Programs and projectsthat lend themselves to feedback and measurement may stretch over longer periodsthan the typical one-year grant. If we are serious about gathering thisinformation, we need to persuade others to work with us to develop a sense of whatpriorities are needed within our programs and how we can achieve measurable results.

This proposal is not limited to the states; there are non-profit organizations,such as the Washington Lung Association, who are potential co~aborators. We canconsider forming alliances, in the form of cooperative agreements, with majorcollaborators. This encourages a common solution, as well as allows leveragedresources. Cooperative methods, driven by strategic planning, allow for feedbackand measurement of everyones' effectiveness.

Some of our grantees have continued with us for two or more consecutive years.The Alaska Health Project (a non-profit) has been a grantee for FY 91-93, as has theWashington State Department of Health. The Oregon Department of EnvironmentalQuality has been a grantee for FY 92-93. These patterns suggest some commitment,assuming continued grant funding. There are potential pitfalls -- grantees aretaking the risk that EPA funding will continue, they will be chosen as grantrecipients, and program priorities will remain constant enough for multi-yearundertakings seen through to conclusions.

Recommendation 5:External Affairs.

Develop an outreach capability within the Office of

Neither Radon nor Indoor Air have had much involvement with the Office ofExternal Affairs. There has been some press interest, but not recently. Press is,by definition, concerned with the news value of an occurrence. Outreach is acontinuing, rather than an episodic, activity. Outreach can be augmented byprofessional capabilities (writing, graphics, education) and could be located in OEAfor use as needed by Radon, Indoor Air, and other programs.

22

Appendix 1Puget Sound Water Quality Authority

PIE Fund Questionnaire

Pertinent portions of the post-project questionnaire are listed below.

Section 1: Improving Water Quality in Puget Sound

1.1 What condition has your project helped correct?1.2 Has that condition been corrected?1.3 How do you know the condition has been corrected?1.4 How has your project contributed to correcting the condition?

Section 2: Individual and Group Behavior

2.1 What behaviors (or lack of) on the part of individuals or groups contributed to thecondition described in Section 1?

2.2 What changes in behavior do you hope have occurred as a result of your project?2.3 How do you know if the behavior of project participants has changed as a result of

your project?2.4 Do you plan any followup activities to estimate long-term changes in behavior of

individuals or groups involved in your project?

Section 3: Knowledge, Attitudes, Skills, or Awareness

3.1 What information did project participants receive as a result of your project?3.2 How was that information transmitted to them?3.3 How did you find out if they received, understood, or used the knowledge, attitudes,

skills or awareness that they received during your project?

Section 4: Reactions Expected from Project Participants

4.1 Under what situations did learning occur in your project?4.2 How do you hope project participants felt about your project or Puget Sound after

being involved in your program?4.3 How do you know how they feel?4.4 Were project teachers (or others presenting intormatlon, if applicable) evaluated by

participants or audience? If so, how?

Section 5: Project Participants

5.1 Who was the audience for your project (be as specific as possible)?5.2 What was the most effective way you found to publicize your project?5.3 How many direct participants (workshop attendees, viewers, listeners, publication

readers, site visitors, etc.) were involved in your project?5.4 How many indirect participants were involved in your project?5.5 What products (publications, etc.) did your project produce? How many?5.6 Who participated in technical review or program oversight?

Section 6: Project Activities

6.1 What activities or events were involved in your projects?6.2 What did participants do at activities or events?

23

6.3 When did these activities occur?6.4 What external factors affected the success of these activities? (bad weather, no

competing events, etc., cooperation of another party or organization)?

Section 7: Resources Put to Work

7.1 What was the total budget for your project?7.2 How much money was raised from sources other than PSWQA?7.3 What other kinds of contributions were made to the project?7.4 How many paid employees worked on the project? For what percentage of their workload?7.5 How many volunteers worked on the project?

Section 8: Problems and Challenges

What problems or challenges were faced in completing the project and were they dealt with?

Section 9: Unanticipated Results

9.1 Describe unanticipated successes (other than planned goals and objectives) thatyou achieved in completing this project.

9.2 To what do you attribute these?

24

r~'I

Appendix 2

Survey of Outreach in Other Government Agencies

Agency Name Staff Outreach Methods Employed Effectiveness How Measured

Army Corps of Eng'rs Press News releases Good questionl Counting beans (# of news Count # of papers that pick upOfc. releases, speeches, or public inquiries) not news releases or inquiries that

accurate result from news release; hiringP.R. firm to research O/R veryexpensive

Seattle City Light Com Newsletter; public presentations Trade shows and community events allo~ Customer Service Survey showsmunit them to meet their customers how many customers they inform;y use signup sheets at presentationsRelati to count # reached; don't knowons (5 how to measure whether public isstaff) "getting" their Info

WA Dept of Ecology Need to plan measurementupfront, In program design; easyto measure # of contacts, harderto measure changes in people'sbehavior; try to measure changesin attitudes and awareness?

EPA Reg. 1 Presentations for groups; booths at Most effective when they target specific Count # of cases; use signupfairs; speeches; written groups sheets at presentations - sendpublications; show test kit questionnaires; track calls -

correlate w/groups they havegiven presentation; clip articlesfrom newspapers that use theirinfo

EPA Reg. 2 Respond to calls; offer tech asst; Want to use Am. Lung Assoc. to followup IAQ: program too new to tellsend written materials; bldg. AQ course to avoid "survey" issue;

problem w/O/R is that it leads to more callsif effective, requiring more staff; no singleform of O/R works -- need combo

25

EPA Reg. 3 Presentations to comm'ity grps; Setting up tables at malls not very effective; Don't measure effectiveness, butdeveloped school curriculum - haven't identified audience have good data restudents take home test kits and testing/mitigation - see directmust return; hot line for public correlation between D/R andinquiries; full color maps for testing/mitigdistrib.; use questionnaire forspeakers: who/how many attendedpresentations; lots of radio PSA's;work w /Better Breather's Clubs;health fairs at hospitals and firms;work w/states, realtors; combinew /EED program in work w /schools;coord w/ALA re Rn Awareness Mo.

EPA Reg. 4 Provide speakers; presentations for Most effective when targeting specific grps. Have no way of measuringgps.

EPA Reg. 5 Use sophisticated survey Survey measures what peopletechniques re Great Lakes project learned from ship - nothing

specific re Rn/IAQ

EPA Reg. 6 1 FTE Presentations to schools Unknown Don't measure

EPA Reg. 7 Brochures, talks in schools, PSA's D/R aimed at children seems effective, based Don't have resources to measureon experience re smoking and seatbelts;PSA's waste money - played in dead time

EPA Reg. 8 _ IAQ program just trng state staff; Too early to determine effects ofstates do most of Rn D/R; fed. IAQ program; WY has mosteffort is hodge-podge: try dif. aggressive Rn D/R and bestthings results - even so, awareness level

is 85% while testing level is 16%

"',.

26"

Appendix 3

WSDOH State Indoor Radon Grant ProgramFinal Report: July 1, 1991 - June 30, 1992

• Report period represent Year 2 of EPA grant• Staffing during grant period was 7 FTE, plus 1 temporary• Participation on committees and symposiums

• WSDOH convened meeting of state-wide Radon Task Force• Attended bimonthly meeting of SBCC tech. adv. gp. during 4-mo. period; attended Energy

Code Comm. and 2 SBCC meetings• Advise key gps. re Rn

• Clover Park S.D. Innovative Sch. Proj.: numerous meetings w/staff, 2 publ. mtgs.;presentation to ESD

• Advised and made 1 presentation to CRPDC committees• 7 presentations to various conferences and committees• 3 presentations to state gov't bodies; aided in passage of legislation that requires local bldg.

officials to supply testing device in new home construe.• Continued to augment and update database• Eliminated 1 fact sheet, updated 4 others• Updated telephone info service; 1,059 calls received during yr. - residential info was primary request,

then basic Rn info and where to purchase test kits• Lending library has various materials, which were used by public and contractors• PSA's

• Issued 3 press releases• Issued press releases and articles re Clover Park S.D. project• Published 4 articles in various newsletters• Rn promotion by Haggen Food Stores -- may have contributed to increase in calls (54 to

133) in Feb.• Various program staff attended at least 10 training sessions/conferences

• Conducted testing survey and mail survey• Produced and distributed brochure for builders to distribute to new home occupants, and

collaborated in brochure for local building officials to distribute to new home owners• Began, but never completed, project to send letters to new home owners re radar detector to be

provided by builders (project superseded by new law requiring local officials to provide detector)• Gave contract money to NE WA Rural Resources Dev't Assoc. to conduct publ. info. program to test,

diagnose, and mitigate in at least 100 econ. disadv'd homes in 3 counties (project never completeddue to lack of matching funds from contractor); project converted to publ. info. effort through variousmedia and presentations '

• CSRAT Project compiled materials, offered training re Rn in schools (113 total attendees plus 38more in Clover Park S.D.)

• 40 buildings te~ted in Clover Park

27

I