master of public administration program an evaluation of a proximity-based approach to environmental...

Master of Public Administration Program

An Evaluation of a Proximity-Based Approach to Environmental Education:

The Section 319 Burnt Mill Creek (BMC) Outreach and Demonstration Project

Mark T. Imperial, Ph.D.Lloyd Jones, Ph.D.

Master of Public Administration ProgramDepartment of Political Science

University of North Carolina at Wilmington

&

Jennifer D. ButlerStorm Water Services

City of Wilmington


Burnt Mill Creek Watershed

• Watershed: 4,274 acres

• 64% of the watershed’s surface is impervious surface

• Most impaired creek in Wilmington, NC and is on the state’s 303(d) list

• Primary pollutants are fecal coliform bacteria, nutrients, and low dissolved oxygen


The BMC Outreach & Demonstration Project

• Duration of the project:– 3-year, $200,000 project funded in part with a Section 319 grant

• Goal of the project:– Increase awareness about watershed issues and motivate

residents and businesses in selected areas of the watershed to adopt responsible watershed practices on individual properties

• Tested the effectiveness of a proximity-based approach to environmental outreach and education– Used an intensive outreach and education effort focused on a

specific target audience located in close proximity to a stormwater demonstration site that contains BMPs representing behavior changes desired by watershed residents


BMC Outreach & Demo Project• Installed a stormwater

demonstration site in close proximity to the target audience featuring examples of BMPs such as– Rain garden

– Habitat garden

– Pet waste stations

– Rain barrels

– Pervious pavement & walkways

– Grassy swale

– Native plants


BMC Outreach & Demo Project

• Direct mailings to residents in target area– Education materials on such things as pet waste, lawn care,

structural and nonstructural BMPs, and other information watersheds and NPS pollution

– Workshop announcements

• Public workshops– 16 held at the demonstration site– 9 other workshops of various types

• Mass media– PSAs, radio spots, print adds, cable access TV

• 10 ecowalks at demonstration site• 12 presentations to students in BMC watershed


BMC Outreach & Demo Project

• Web page with information on stormwater BMPs and pollution prevention

• 3 Creekkeeper trainings

• 3 awards to recognize persons/organizations doing “watershed friendly” activities

• 3 storm drain marking events to install awareness markers on storm drains in the BMC watershed

• 6 watershed clean up events organized and conducted by an area youth group

• Watershed poster featuring BMC watershed


What is the Theory Behind the Proximity-Based Approach?

• Targeting public education and outreach on residents located in close-proximity to watershed restoration and stormwater improvement projects will improve the efficacy of education efforts and increase residents’ motivations to adopt structural and nonstructural BMPs

• If effective, the target audience should have a higher level of awareness of watershed and NPS issues, higher rates of BMP adoption, and higher levels of message retention from outreach efforts than watershed or city residents


Research Design• Quasi-experimental design

– Pretest-posttest nonequivalent control group– Intact group of residents makes it impossible for the random

assignment of individuals to treatment and control groups– Powerful because it can detect changes due to treatment if there

are statistically significant changes

• Evaluation consisted of– Measuring a group of subjects (pretest)– Introducing a treatment to residents in the target area (BMC

Outreach and Demonstration Project)– Observing the same subjects again (posttest)

• BMC watershed and City residents served as the nonequivalent control (comparison) group– Control groups didn’t get direct mail but were exposed to media


Research Design• Telephone survey used for pretest and posttest

– 5 random samples: single & multi-family residents in target area, single & multi-family residents in BMC watershed, and the City

– Different procedures used to generate phone list for pretest and posttest samples

– Telephone survey provided savings in terms of time and cost– Allowed the use of a complicated skip pattern– CATI software helps standardized data entry– Limited length and questions had to be simple enough to be

understood by general public without visual aids

• Survey administration– UNC Wilmington Survey Research Laboratory– Pretest was administered in October and November 2002– Posttest was administered January – March 2005


Differences in the Pretest and Posttest Phone Sample for the Target Area

Land Use Phone Numbers for the Pretest Sample

Phone Numbers for the Posttest Sample

Single-Family 473 396

Multi-family 205 429

Business 67 149

Total 745 974


Total Number of Survey Responses for Pretest and Posttest Surveys

2002 Pretest 2005 Posttest

Single-family – Target Area 63 62

Multi-family – Target Area 44 27

Single-family – BMC Watershed 301 318

Multi-family – BMC Watershed 155 314

City of Wilmington 395 1293

Total 958 2014


Research Design

• Survey questions were open- and close-ended and collected data on:– Watershed awareness related to NPS pollution, watersheds, and

the values of specified BMPs

– Attitudes about the quality of local waterways

– Behavior changes associated with the adoption of structural and nonstructural BMPs

– Outreach effectiveness to see where residents received messages from and determine whether they recalled seeing mass media messages

– Demographics pertaining to housing type, income, race, education, and sex to facilitate data analysis


Potential Threats to Validity & Reliability• Attrition

– Potential problem when residents move in and out of target area and don’t get the full treatment

– This problem is most pronounced for multi-family residents

• Maturation & History– Potential problem when long time periods elapse between pretest and

posttest– 25 months elapsed– Problem controlled for because control groups are also likely to

experience any effects

• Testing– Potential problem if the testing instrument changes– Problem controlled for by using similar survey instruments for pretest

and posttest for all 5 samples


Potential Threats to Validity & Reliability

• Questions about unethical/illegal behaviors– Respondents may be unwilling to admit to behaviors such as

dumping motor oil in storm drains– Frequencies of these self-reported behaviors may be lower than

they actually are provided they know behavior is wrong– If they know its wrong and lie about their behavior, no amount

of education may change but they also know its wrong

• Telephone list Construction– Challenge to develop a geo-coded list of accurate telephone

numbers for residents in the target area and BMC watershed– Two different methods used to generate phone lists City’s utility

account information & Cross+Search Plus– Some statistically significant differences in the samples


Significant Demographic Changes Between the Pretest and Posttest Surveys

Changes in Demographic Characteristics Target Area BMC Watershed City

SingleFamily

MultiFamily

SingleFamily

MultiFamily

Have a dog N ** N N N

There is a creek, stream, or marshy area on or next to their property

*** N N

Home ownership N *** ** *** N

Education level N N N N **

Spanish or Hispanic origin N N N N N

Racial or ethnic group N N N * **

Household Income N N N ** N

Gender N N N N N

* p < .1; ** p < .05; *** p <. 01; (-) change in opposite direction; N = no change


% within Measurement period

34.5% 25.0% 31.1%

24.1% 43.8% 31.1%

3.4% 6.3% 4.4%

37.9% 25.0% 33.3%

100.0% 100.0% 100.0%

All the time

most of the time

Somtimes

Never

COLLECTSDOG WASTE

Total

pre target singlepost target

single

Measurement period

Total

Data Analysis• Compared pretest and posttest survey responses

– But how do you tell if these changes are important?

– Some increase in the desired direction, other don’t

– Sometimes changes are within sampling error


Chi-Square Tests

2.272a 3 .518

2.244 3 .523

.122 1 .727

45

Pearson Chi-Square

Likelihood Ratio

Linear-by-Linear Association

N of Valid Cases

Value dfAsymp. Sig.

(2-sided)

4 cells (50.0%) have expected count less than 5. The minimumexpected count is .71.

a.

Data Analysis• Pearson Chi-Square statistics

– Summary statistic that accounts for sample error and comparisons across categories

– Used p < .05 standard as recommended in social science literature

– Since p = .518 and results could occur by chance about 52 out of 100 times, there is insufficient evidence to assert that there were any significant differences in dog waste collection


Changes in Watershed Awareness

• Two sets of indicators were examined– Knowledge about the value of

selected BMPs

– Knowledge about water quality and NPS

Rain Barrels are a Structural BMP


Knowledge About the Value of BMPs

Indicators of a Change in Watershed Awareness

Target Area BMC Watershed City

SingleFamily

MultiFamily

SingleFamily

MultiFamily

Planting native plants N N N

Pervious materials N *

Rain gardens N ***()

N

Rain barrels N N N

Habitat gardens N N *()



Results – Value of BMPs

• The results of the comparisons of residents of single-family homes in the target area produced no statistically significant differences between the pretest and post test surveys.

Rain Gardens are a BMP


Knowledge About Water QualityIndicators of a Change in Watershed Awareness


SingleFamily

MultiFamily

SingleFamily

MultiFamily

All rain water is not absorbed by the ground before it gets to streams

N N N **() N

Rain falling on roads picks up pollutants from automobiles

N N N N N

Water from storm drains is carried to local waterways

N N *** N *

Major source of poor water quality is NPS runoff

** N *** N ***

Recall hearing the term watershed N N ** N ***

They recall the name of the watershed they live in

** * N N N



Results – NPS Knowledge• Two indicators produced statistically significant

results for single-family residents in the target area– Change in the percentage of respondents who knew that the

major source of poor water quality in local waterways was runoff from yards, streets, and parking lots (increased from 68.6 to 88 percent)

– The fact that similar changes were observed in the control groups may indicate that something else cause these changes, possibly mass media

– The percentage of respondents who knew the name of the watershed they lived in also changed from 20.9 to 44.4 percent

• The results for residents of multi-family homes in the target area produced no significant changes

• Positive changes in control groups’ knowledge of NPS


Changes in Attitudes about Water Quality

• Examined whether there were changes in attitudes about the quality of local waterways– Cape Fear River

– Greenfield Lake

– Intracoastal Waterway (ICW)

– Burnt Mill Creek


Attitudes About Water Quality

Indicators of a Change in Attitudes about Water Quality


SingleFamily

MultiFamily

SingleFamily

MultiFamily

Water quality in Cape Fear River N N * N N

Water quality in Greenfield Lake * N *** *** ***

Water quality in Burnt Mill Creek N N ** ** N

Water quality in Intracoastal Waterway

N N N N N



Results – Local Water Quality

• The results of the comparison of the single- and multi-family samples in the target area found no statistically significant changes in the four indicators

• Comparisons for the control groups produced some positive results– Residents in the BMC watershed and City increased the

percentages that correctly knew that water quality in Greenfield Lake is bad

– Residents in the watershed also changed their views of water quality in BMC

• While changes in the control groups are encouraging, it is unclear to what extent they were due to the BMC Outreach and Demonstration Project


Changes in Behavior

• Two sets of indicators were examined that asked whether residents installed– Structural BMPs like pervious

surfaces, plants, trees, rain gardens, rain barrels, habitat gardens, and buffers

– Nonstructural BMPs like proper disposal of dog waste, cooking grease, grass clippings, leaves, using soil tests, and not dumping oil, paint, or garbage into storm drains

Habitat Gardens are a structural BMP


Use of Structural BMPs

Indicators of Behavioral Change


SingleFamily

MultiFamily

SingleFamily

MultiFamily

Planted native plants N N ***

Installed paths with pervious materials

N ** N

Planted trees for shade N N N

Installed a rain garden N N ***()

Installed a rain barrel N N N

Have a habitat garden N N N

Planted a buffer or vegetation next to waterway

N N ***()



Results –Structural BMPs• Comparisons of the pretest and posttest samples for

single-family residents in the target area produced no statistically significant changes

• Comparisons of the pretest and posttest samples for the comparison groups were also disappointing– Single-family residents in the BMC watershed who installed

paths or driveways and used pervious materials increased from 32.7 to 53.6 percent

– For City residents who planted plants, the percentage who planted native plants increased from 64.6 to 77 percent

– However, City residents were also less likely to install rain gardens or plant buffers next to local waterways


Use of Nonstructural BMPsIndicators of Behavioral Change


SingleFamily

MultiFamily

SingleFamily

MultiFamily

Collect your dog’s waste N N * N **

Wash your car in proper location N N ** N N

Properly dispose of grass clippings ** ** * *** N

Properly dispose of leaves or pine needles

N N N *** **

Properly dispose of cooking grease N N ** N N

Did something to improve water quality

N N N N N

Planted grass to eliminate brown spots

N N *()

Got a soil test for their lawn N *** N

Proper application of fertilizer N N N



Use of Nonstructural BMPs (Cont.)Indicators of Behavioral Change Target Area BMC Watershed City

SingleFamily

MultiFamily

SingleFamily

MultiFamily

Put grass clippings and leaves into a storm drain or drainage ditch

N N *** N ***

Poured old or used engine oil or antifreeze into a storm drain or drainage ditch

** N N N *

Emptied paint into a storm drain or drainage ditch

** N N N N

Hosed down a driveway, sidewalk, or parking lot into a storm drain or drainage ditch

N *** ** N ***

Put garbage or litter into a storm drain or drainage ditch

** * ** N **



Results – Nonstructural BMPs• No significant changes for single-

family residents in target area for– Proper collection of dog waste– Car washing location– Disposal of grass clippings– Disposal of leaves and pine needles– Disposal of cooking grease– Taking action to improve water quality– Planting grass to eliminate erosion– Getting soil tests– Proper application of fertilizer– Not putting grass clippings and leaves

in storm drains– Hosing down driveways and parking

areas into storm drains

Pet Waste Disposal is a Nonstructural BMP


Results – Nonstructural BMPs (Cont.)• Fewer single-family residents reported that they:

– Improperly disposed of grass clippings from lawns– Poured engine oil or antifreeze into a storm drain or drainage

ditch – Emptied paint into a storm drain or drainage ditch – Put garbage or litter into a storm drain or drainage ditch

• Significant changes in BMC sample• However, substantive changes were small. For

example– While 1.7 percent acknowledged pouring oil or antifreeze into a

storm drain, no one reported the behavior in the posttest– However, 10.3 percent volunteered without being prompted that

they “would never do it” – No one volunteered this response in the pretest

– Messages may have been received but there is little behavior to change


Results – Nonstructural BMPs (Cont.)

• For multi-family residents in target area, significant changes– Properly disposing of grass clippings from lawns

• Mixed result

• Significant changes in BMC sample

– Hosing down a driveway or parking lot into a storm drain

• Positive but better sample could be reason

• City sample positive change casts additional doubt


Results – Nonstructural BMPs (Cont.)• Changes in single-family BMC sample

– Using the proper car washing location - Mixed

– Properly disposing of cooking grease - Mixed

– Getting a soil test for their lawn

– Putting grass clippings and leaves into a storm drain

– Hosing down a driveway or parking lot into a storm drain

– Putting garbage or litter into a storm drain

• Changes in multi-family BMC sample– Properly disposing of grass clippings

– Properly disposing of leaves and pine needles

– Positive but better sample could be reason


Results – Nonstructural BMPs (Cont.)• Changes in the City sample

– Proper collection of dog waste – Properly disposing of leaves and pine needles – hard to interpret– Putting grass clippings and leaves into a storm drain – small

substantive change– Hosing down a driveway, sidewalk, or parking lot into a storm

drain – small substantive change– Putting garbage or litter into a storm drain – small substantive

change

• While changes in BMC and City samples are encouraging it is difficult to see how they could be due to the project when similar changes are not observed in the target area


Evidence of Outreach Effectiveness

• Two sets of indicators were used to assess whether residents in the target area– Received and acted upon

outreach messages

– Received messages sent by various forms of mass media


Message Delivery and Action

Indicators of Outreach Effectiveness


SingleFamily

MultiFamily

SingleFamily

MultiFamily

Recalled receiving direct mail about water quality

*** N N N **

Recalled seeing a local watershed sign N N ** N N

Looked on the internet for information about local water quality and things to do to improve it

N N N N N

Read a brochure, fact sheet, or newsletter with information about local water quality

*** * * * N

Attended a workshop on local water quality

* N N N N

Are likely to read stories about local water quality in the newspaper

**() N ** N N



Results – Message Delivery• Comparisons for single-family residents in the target

area produced three significant changes – Percentage of respondents who reported receiving direct mail

increased from 29.3 percent to 75.4 percent

– Significant increase in the residents reporting that they read a brochure, fact sheet, or newsletter (44.8 percent to 74.1 percent)

– Negative change in the percentage reporting that they would be more likely to read stories about local water quality in the newspaper

• No significant changes in the multi-family sample– Lack of a positive change in direct mail question could be due to

transitory nature of population and difficulty getting an accurate mailing list


Mass Media Campaign

Indicators of Outreach EffectivenessTarget Area BMC Watershed City

SingleFamily

MultiFamily

SingleFamily

MultiFamily

Recalled seeing PSAs about water quality on TV

N N N N ***

Recalled seeing news stories about water quality on local television

N N N N **

Recalled seeing news stories about water quality in the Star News

N N N N N

Recalled hearing news stories or PSAs about water quality on the local radio

N N ***() N N



Results – Mass Media• If mass media were effective, positive changes should

be observed across all five geographic areas• Unfortunately, there were only three statistically

significant changes and none in the target area– The percentage of City residents who saw a PSA about water

quality on TV increased from 46.7 to 56.7 percent– There was a small increase in City residents that recalled seeing

news stories about local water quality on local television (53.3 to 59.4 percent)

– The percentage single-family residents in BMC watershed who reported hearing news stories or PSAs on the radio declined from 31.9 to 22.1 percent

– Results suggest limited effectiveness of campaign or that proximity based approach does not make residents more receptive to messages


Conclusions of Evaluation

• Project was particularly effective at delivering information via direct mail to single-family residents in target area

• Project was ineffective in changing watershed awareness, changing attitudes about local water quality, or motivating behavior changes– There does appear to be some general improved understanding

of NPS issues across all 5 samples but cause is unclear

– For many nonstructural BMPs, there was little negative behavior to change


Conclusions of Evaluation

• The data suggests that a proximity-based approach is ineffective– Targeting public education and outreach on residents located in

close-proximity to watershed restoration and stormwater improvement projects does not improve the effectiveness of education efforts

– It also does not significantly increase residents’ motivations to adopt structural or nonstructural BMPs

• Local officials can take some consolation– Few respondents reported that they put grass clippings, leaves,

paint, engine oil, antifreeze, or garbage in storm drains

– Much needs to be done to educate about other practices


Can the Findings Be Generalized?

• Results are for one project and a target area with a particular set of demographic characteristics

• But demographics are not unusual for area

• Resources, time, and effort expended are more substantial than occurs by countless other local governments as part of their environmental education efforts

• Quasi-experimental design suggests results should be generalizable to other areas but more research is needed


What could explain poor results?

• Treatment quality was comparable or exceeded what you find in similar efforts

• Direct mail may be a poor technique– Those interested and knowledgeable read materials and those

who are not don’t– No ability to educate or change behavior because those

interested already no the issues and what to do

• Mass media at a local level may be unable to get through the noise– Need “Madison Avenue” level of sophistication – Coordinated national campaign, not a large number of

“amateur” efforts


Implications & Lessons Learned


Implications & Lessons Learned

• Important public policy questions– Is environmental education that is oriented around geographic

areas and demonstration projects a waste of resources?

– Is NPDES Phase II requirements for additional investments in environmental education a waste of resources?

– Does mass media need to be conducted at a higher level like national media to cut through the noise


Questions?

master of public administration program an evaluation of a proximity-based approach to environmental...

Documents

creek watershed watershed

bmc watershed slide

watershed residents

public education

watershed issues

watershed restoration

creek bmc outreach

watershed friendly activities