evidence-based intervention:1viuw040k2mx3a7mwz1lwva5-wpengine.netdna-ssl.com/wp... · web viewupon...

Process for New Hampshire Service to Science Classification

The NH Bureau of Drug and Alcohol Services (BDAS) recognizes that over the years, many prevention agencies, organizations and practitioners have developed programs independent of large-scale research projects in order to meet the unique needs of their communities.

BDAS also recognizes that federal agencies and other oversight, policy and funding organizations are beginning to require interventions that have been proven successful through research or rigorous evaluation. Funders and policy makers are more vocal than ever that interventions being implemented in the field need to have outcomes establishing efficacy.

To ensure the utilization of interventions with sound evidence of effectiveness while recognizing the contributions and outcomes of interventions that have been developed within the field of practice rather than research, BDAS has developed a process by which interventions seeking endorsement as evidence-based for the purposes of funding priorities and program replication may apply for “NH Service to Science” classification.

The term “Service to Science” is used by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) to identify interventions developed by local and community organizations that have yet to establish their efficacy through research and/or rigorous evaluation. These programs have been implemented with focus populations to support the prevention of alcohol and other drug use and abuse, thus in “service” to their communities, but have not established measurable outcomes through the “science” of rigorous evaluation.

“Service to Science” endorsements indicate that an intervention has a strong likelihood of producing statistically significant intermediate and/or long-term outcomes and are candidates for resources and capacity to conduct formal evaluation to determine their efficacy.

The application that follows is aligned with SAMHSA’s Service-to-Science nomination process and has been developed to determine the strength of theoretical frameworks used in the development of the intervention and the feasibility of determining and evaluating intermediate outcomes relevant for the intervention’s intended purpose and for the risk and protective factors and cultural context influencing alcohol and other drug behaviors in New Hampshire.

Service to Science Applciation 2017 1

I. THE APPLICATION PROCESS

The NH Service to Science application process involves the following components and stages:

1. Intention to Apply: Prevention programs that are designed to influence change in risk and substance misuse behavior that are uniquely influencing change in participants should consider submitting an application for a review of evidence and practices. The intent to apply form is a document that triggers a program review/technical assistance call from a member of the Cetnter for Excellence and sets the stage for presenting to the NH Expert Panel.

2. The Application: This application is for interventions designed to prevent or reduce alcohol or other drug abuse or to influence contributing factors that may influence risk or resiliency related to alcohol or other drug abuse. The application can be found at: http://bit.ly/2rIux85

3. Establishment of Theoretical Framework: Research often establishes theories that scientists then use to create hypotheses that are tested in laboratories or in real-world scenarios. The theory behind the content, structure and delivery method of an intervention provides the logic to establish why and how the intervention “should” work.

4. Demonstration of Replicability: Replicability refers to the availability and quality of resources that other organizations or practitioners would need to be able to replicate an intervention, such as training (including qualified trainers and access to training), curriculum materials, templates, and other resources. Although a sponsoring organization may not be interested in promoting the replication of an intervention, the existence of formalized materials and resources that would facilitate replicability is an indicator of sound practice and design. Furthermore, the existence of formalized materials and resources allows for assessment of their quality and content.

5. Establishment of Intermediate and Long-Term Indicators Being Influenced: Establishing intermediate and/or long-term indicators that are both measurable and relevant to the intervention’s intended outcome is a requirement for NH Service to Science endorsement. Applicants are expected to use the theoretical framework of the intervention’s design to establish intermediate and long-term indicators that are expected to be influenced by the implementation of the intervention. If an intervention is selected for endorsement, the NH Center for Excellence will work with the intervention’s sponsoring organization or practitioner to review and finalize core indicators for measurement. It is important to note that BDAS is considering core indicators across multiple intervention categories. Information on these pre-determined core indicators is available from the Center.

6. Establishment of an Evaluation Design and Methodology to Determine Efficacy: An evaluation design and methodology must generate data that can determine whether an intervention has had its intended effect on intervening variables to a statistically significant degree. Although data need not have been gathered prior to the NH Service to Science application process, the development of an evaluation design and methodology provides information about analytical approach and


conceptual logic that serve as indicators of sound practice and design. As with the establishment of intermediate and long-term indicators above, the NH Center for Excellence will provide the resources and capacity required to make adjustments to evaluation methodology and implement the evaluation design to ensure data will be generated that will determine efficacy.

7. Initial Review and Technical Assistance: A staff member at the NH Center for Excellence meets with applicants to discuss their intervention and determine what actions, if any, should be taken to strengthen the application in preparation for Expert Panel review. Actions may include more thorough literature reviews, an improved evaluation design, formalization of training or materials, or other activities. Whether or not the intervention is currently funded by the NH Bureau of Drug and Alcohol Services may determine whether technical assistance can be provided by the Center to strengthen an application or if external consultation is required.

8. Expert Panel Review: Expert Panelists review application materials for satisfactory evidence of SAMHSA-recommended criteria outlined in the Review Process and Scoring Rubric (see Appendix A). These criteria are recommended by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) for review of evidence-based practice.1

Two phases of SAMHSA’s Service to Science guidance are utilized by the Expert Panel as a means to determine if a program or practice is “promising” (Phase I) or “evidence-based” (Phase II). Initial applications that adequately meet at least 80% of the criteria for Phase I will be endorsed as a “Promising Practice”. Upon review, applicants will receive recommendations from the Expert Panel around evaluation design, data analysis, etc. During Phase II, programs present practice refinements and evaluation outcomes for NH Evidence-Based Practice endorsement and inclusion on NH’s Registry of Evidence-Based Interventions.

Applications that adequately meet at least 80% of criteria for Phase I and II AND who have outcomes that show statistically significant change in indicators that research has supported as influencing risk and resiliency related to alcohol and other drug abuse will be endorsed as a “NH Evidence-Based Intervention” and be included on NH’s Registry of Evidence-Based Interventions.

Please note: Those NH Service to Science applications for “Promising Practice” typically enter into a 12 to 18 month process during which the following technical assistance may be sought through the NH Center for Excellence or independently. The NH Center for Excellence has limited funds to support non-state funded interventions; therefore, technical assistance may be capped, after which programs or agencies may choose to continue technical assistance for a fee or may seek other consultants or research centers for their additional needs.

II. THE DETERMINATION OF EFFICACY1 http://www.samhsa.gov/grants/2005/standard/Srv2Sci/srv2sci_01.aspx


For those interventions that are successful meeting the above criteria during the application process, the NH Center for Excellence in partnership with BDAS, the Center’s Expert Panel, and the Northeast Center for Applied Prevention Technology’s Regional Expert Team will engage sponsoring organizations in the implementation of an evaluation design to determine the efficacy of the intervention.

III. THE NH REGISTRY OF EVIDENCE-BASED INTERVENTIONS

Those interventions that are successful in their application for “Service-to-Science” endorsement and that have produced statistically significant outcomes through formal evaluation will be listed on the NH Registry of Evidence-Based Interventions (NH-REI). Sponsoring organizations will be required to update their NH-REI endorsement each year, with renewal applications required every five years.

APPLICATIONPlease provide the following information for your program or intervention to be considered for NH Service to Science endorsement.

Name of Intervention:

Name of Sponsoring Organization or Practitioner:

Primary Contact for this application:

Address of primary contact:

Email of primary contact:

Web site for intervention if any:

How long has this intervention been implemented? yearsHow many individuals or communities have been participants in the intervention since its initial implementation? Individuals Communities

Target Population: The target population is the population in which long-term change such as 30 day use will be realized. Check all that apply.

Age Race/Ethnicity0-4 year olds White, non Hispanic Native American or Alaska Native

5-11 year olds Asian/Asian American African American

12-17 year olds Native Hawaiian/Pacific Islander Multi-racial or Multi-ethnic

18-20 year olds Hispanic or Latino Specify Other Race:

21-24 year olds Other defining characteristics25-44 year olds

45-64 year olds

65+


Please provide a detailed description of the population that the practice is designed to serve, and demographic characteristics of the people served by the practice over the past year.

Focus Population: The focus population is the population toward which the intervention is focused. Below is a description of three population categories defined by the National Academy of Science’s Institute of Medicine (IOM). Please review the definitions before selecting the focus population for which the intervention was designed.

The IOM population categories are as follows:

Universal (U): populations recruited without consideration of specific risk factors or behaviors, such as the “general public or a segment of the entire population with average probability of developing a disorder, risk or condition.” 2

Selective (S): Populations whose risk of a disorder is significantly higher than average, either imminently or over a lifetime3 based on a specific risk factor or set of risk factors such as children of alcoholics, students with high truancy or other school difficulty, or trauma victims.

Indicated (I): Individuals, groups or populations who have minimal but detectable signs or symptoms suggesting a disorder.4

Focus Population of Intervention: Please indicate the focus populations which your intervention is designed to impact. If the intervention is designed to impact selective or indicated populations, please also indicate the risk factors or indicated behaviors of those populations. Check all that apply.

2 Springer, J.F. and Phillips, J. The Institute of Medicine Framework and Its Implications for the Advancement of Prevention Policy, Programs and Practice, p.4. http://www.ca-cpi.org/Document_Archives/IOMArticle3-14-07fs.pdf3 Springer, J.F. and Phillips, J. The Institute of Medicine Framework and Its Implications for the Advancement of Prevention Policy, Programs and Practice, p.4. http://www.ca-cpi.org/Document_Archives/IOMArticle3-14-07fs.pdf4 Springer, J.F. and Phillips, J. The Institute of Medicine Framework and Its Implications for the Advancement of Prevention Policy, Programs and Practice, p.4. http://www.ca-cpi.org/Document_Archives/IOMArticle3-14-07fs.pdf


IOM Focus Population NameFor Selected or Indicate Populations, please check the primary risk factors or indicators of the focus populationCheck all that apply

U Business – EmployeesU Business - Alcohol Retailers/Servers/SellersU Business – OtherU Medical/Hospital/Health Clinic staffU Treatment/Recovery staffU Mental Health providersU Youth – Universal

S Youth – Selective (At least one risk factor)

Low socio-economic status Family history of substance use Past involvement in Criminal Justice System Delinquency/Truancy Other school difficulty Housing instability Other:


I Youth – Indicated

Indication of current or recent substance use Indication of current or recent criminal activity Indication of other court involvement Other indicator:

U Parents/Families– Universal

S Parents/Families – Selective(At least one risk factor)


I Parents/Families – Indicated


U Parents/Families – Other

U Young Adults (College) – Universal

SYoung Adults (College) – Selective (At least one risk factor)


I Young Adults (College) – Indicated



U Young Adults (Non-College) – Universal

SYoung Adults (Non-College) – Selective (At least one risk factor)


I Young Adults (Non-College) – Indicated


U Elder Adults – Universal

S Elder Adults – Selective (At least one risk factor)



I Elder Adults – Indicated


U Other Adults – Universal

S Other Adults – Selective (At least one risk factor)


I Other Adults – Indicated


U Other Community-Based OrganizationsU Community-at-largeU Law Enforcement/Courts/Safety Staff/PersonnelU School Staff/PersonnelU Faith-Based Organizations

Please provide participant recruitment methods appropriate for the IOM population type for which the program is designed.


Please select the category and sub-category of intervention that most aptly characterizes the intervention. Definitions are provided below. Please check all that apply.

DIRECT: This category of interventions is typically characterized by two-way information exchange such as educational programs, community teams, trainings, guided discussion groups, and services to individuals.

Educational Programs: This sub-category relates to educational programming orcurriculum delivered to groups of individuals. Groups might include parents,employees, seniors, students, or other populations recruited for or assigned anintervention.

Empowerment/Leadership Teams: This sub-category of interventions relates tosupervised, facilitated teams of individuals who are both the target and deliverer ofprevention efforts. These include teams of parents, recovery community members oryouth or others who are leading prevention activities in their school or community.

Specialized training: This sub-category of interventions refers to training or other two-way communication or information exchange for groups of people who have a relevant, established commonality, such as Responsible Server/Seller trainings for employees of alcohol establishments; Screening, Brief Intervention and Referral to Treatment (SBIRT) training for medical staff; or a one-time skill-building sessions for parents of school aged children.

Guided discussion groups: This sub-category refers to interventions that involve guided discussion groups but that are not primarily educational. Small group sessions may include on-going guided discussion groups with Children of Alcoholics, individuals in recovery, parents of substance using children, or other focus populations.

Individualized Services: This category includes interventions that involve sustained one-on-one interaction over a period of time that may involve screening and brief intervention, mentoring, or problem identification and referral.

INDIRECT (Environmental): This category of interventions is typically characterized by one-waycommunication strategies such as media campaigns or mailings or influencing policy.

Media 5: This sub-category includes general or targeted media campaigns/social norms campaigns, or community messaging. i

Information Dissemination: This sub-category refers to one-time, often event-based,distribution of prevention materials or publications

Policy: This sub-category includes reviews of, revisions to, or newly established policies,ordinances, rules, codes of conduct or enforcement of such. It can involve schools,

5 Media: messaging to broad audience using visuals (poster, advertisement, video clip, etc) or sounds (radio) that broadcast a prevention message to an audience.


workplaces, athletic or extracurricular groups, local ordinances, or other organizations orfunctions.

Targeted Communications: This sub-category includes communications with specificgroups such as Remind/Reward letters to alcohol retailers, Safe Homes registrationmaterials mailed to parents of high school students, prevention speakers addressingaudiences, or other one-way direct communication targeting a specific audience orpopulation.

IV. CORE ELEMENTS

The National Institute on Drug Abuse (NIDA) recommends three core element categories (structure, content, and delivery) that may be used to establish the foundation for fidelity and performance monitoring. The categories have been expanded to the five established core element categories below for all NH Evidence-Based Interventions (EBIs). For the purposes of the NH Service to Science classification process, the following core elements will be operationalized in collaboration with the NH Center for Excellence to support the building of an intervention’s evidence base.

Core Element 1: Staffing/TrainingCore Element 2: Intensity/Dosage/DurationCore Element 3: ContentCore Element 4: LocationCore Element 5: Method of Delivery

Please provide detail on the intervention following the core element categories below.

NIDA CoreElement Category

CFEx CoreElement Category

Provide Detail for the Service-to-Science Intervention

Structure Method of DeliveryWhat is the primary method of delivery of this intervention?

Content ContentProvide Detail on core content areas (knowledge and skill areas, etc).

Delivery Intensity/Dosage/DurationWhat is the intensity, dosage or duration of one iteration of the intervention (e.g. weekly 30 minute meetings throughout a school


year? One 45 minute session per week for 8 weeks? Etc)

Staffing/TrainingWhat are the recommended staffing and training needs? Does the intervention require a Bachelor’s level position? How many training hours would be required to replicate the intervention? Etc.

Staffing Requirements:

Education and Training Requirements:

LocationWhat is the recommended location for delivering the intervention? Community setting? School classroom? In homes? Varied settings in community?

Please describe how critical stakeholders were included in the development of the practice.


V. ESTABLISHING THEORETICAL FRAMEWORK

To establish the theoretical framework of the proposed intervention, please provide a theory of change by filling in the logic model below to demonstrate the inputs, activities, outputs and outcomes of the intervention. In the subsequent section, provide information and citations from research supporting your theory of change.

GOAL: What is the ultimate goal of the intervention? This often parallels the desired impact in the right hand column below and should provide a period of time in which change will be observed.

Implementation of Intervention Outcomes of Intervention Impact

Inputs Activities Outputs Immediate Intermediate What do you need in order to implement the intervention?

What will be done during implementation?

What will you be able to show/produce after implementation?

What immediate change will be observed?

What intermediate change will be observed?

What long term impact will be realized?

Human ResourcesClientelePartnersInstruments, curricula, information

Advisory teams or boardsCommunity resources or complementary servicesFundingData, data collectionTraining and TAOther inputs

Identify and engageRecruitTrainData collectionOther activities

Service delivered# served% of people who remained in program through completion

# trained# engaged

# youth have been instructed in …# families have been introduced to …# of community orgs or

schools…Satisfaction levels

Change observed in participants (knowledge, perception, attitude)Changed observed in services or capacities

Other change observed/measured

Behavior Change of participantsBehavior Change of larger communityBehavior change of

community partners

Contextual Factors (e.g. political, economic, cultural, school climate, etc.):

11

Research Citations: For the purposes of engaging in a process to determine the evidence base on an intervention, literature reviews of prior research are often used by program developers to build a hypothesis of efficacy for an intervention and to establish aspects of design, content, and delivery. For each NIDA core element category below, please provide prior research that supports the intervention’s theory of change. For example, if the program is delivered in an after-school setting, you may provide information about research done that establishes after school hours as an effective setting for prevention education. A program delivering content to adolescents relative to communication skills may present research on the impact positive communication skills have had on other adolescent risk behavior. Please tab through the examples below to document as many research articles you wish to present to support your theory of change.

List CoreElement Category ( content, structure or delivery) or Focus Population for Intervention

Established Research to Support Core Element CitationEvaluation Design of Research Cited

Number (N) of research citation

Content Example:

1. Autonomous Decision-Making

Previous research on the relationship between motivational orientations and drinking behavior among college students has suggested that students who are more autonomously oriented consume less alcohol whereas those who are more control oriented consume more alcohol.6

Chawla, N., Neighbors, C., Logan, D., Lewis, M. A., & Fossos, N. (2009). Perceived Approval of Friends and Parents as Mediators of the Relationship Between Self-Determination and Drinking. Journal of Studies on Alcohol & Drugs, 70(1), 92-100.

Descriptive Correlational Design

N=818

Focus Population Example:

2. Peers

This research also focused on the extent to which autonomy vs control orientations are mediated by the perceived approval of friends and parents and showed friend approval as a significant influence but not parent approval.7

Same as above Same as above Same as above

1.

2.

More space is available on the next page.

6 http://www.higheredcenter.org/research/perceived-approval-friends-and-parents-mediators-relationship-between-self-determination-an7 http://www.higheredcenter.org/research/perceived-approval-friends-and-parents-mediators-relationship-between-self-determination-an

12




3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.




13

14.

15.

16.

17.

18.

19.

20.

14

VI. DEMONSTRATION OF REPLICABILITY

MATERIALS

Please list materials that are required or recommended for implementation of the intervention. Include copies of essential print materials with this application.

TRAINING

Please provide information about training requirements or recommendations, trainer contact information, length of training, and location and accessibility of trainings in the intervention. Include trainings that may be provided by the program developer, by the Service to Science applicant, or other entities or individuals. Document the number and percentage of staff trained in the practice, and a mechanism for ongoing training for any new staff.

OTHER RESOURCES

Please provide information on other resources required or recommended for successful implementation of the intervention.

VII. INTERMEDIATE AND LONG-TERM INDICATORS

- 15 -

Establishment of Intermediate and Long-Term Indicators Being Influenced: Establishing intermediate and/or long-term indicators that are both measurable and relevant to the intervention’s intended outcome is a requirement for NH Service to Science endorsement. Applicants are expected to use the theoretical framework of the intervention’s design to establish intermediate and long-term indicators that are expected to be influenced by the implementation of the intervention. If an intervention is selected for endorsement, the NH Center for Excellence will work with the intervention’s sponsoring organization or practitioner to review and finalize core indicators for measurement. It is important to note that BDAS is considering core indicators across multiple intervention categories. Information on these pre-determined core indicators is available from the Center.

The NH BDAS is in the process of establishing core indicators for each sub-category of intervention. The chart below provides anticipated core indicators.

Please indicate which indicators are anticipated to be influenced by the intervention. Check all that apply. Please enter indicators that are not listed but that are relevant to the theoretical framework of the intervention.

INDICATOR

Knowledge

Knowledge of Physical and Development Effects of AOD Use and AbuseKnowledge of Addiction ProgressionKnowledge of effective strategies to increase resiliencyOther: Other:

Perception

Perception of Risk/Harm (multiple levels/types of risk)Perception of knowledge related to participation in program (self-efficacy) NOT COREPerception of skill related to participation in program (self-efficacy) NOT CORE Other: Other:

Attitude

Perception of Wrongness (Disapproval)Intention to communicate with others such as child, parent, law enforcementCommunity Engagement (likelihood to increase involvement in community prevention efforts)Prediction of future use (e.g .likelihood that participant will use alcohol or other drugs in the next month) Other: Other:

SkillsRefusal skillsOther: Other:

Behavior

30 day useAge of OnsetOther: Other:

VIII. EVALUATION DESIGN AND METHODOLOGY

- 16 -

Establishment of an Evaluation Design and Methodology to Determine Efficacy: An evaluation design and methodology must generate data that can determine whether an intervention has had its intended effect on intervening variables to a statistically significant degree. Although data need not have been gathered prior to the NH Service to Science application process, the development of an evaluation design and methodology provides information about analytical approach and conceptual logic that serve as indicators of sound practice and design. As with the establishment of intermediate and long-term indicators above, the NH Center for Excellence will provide the resources and capacity required to make adjustments to evaluation methodology and implement the evaluation design to ensure data will be generated that will determine efficacy.

Please provide information below on current or anticipated outcomes and data collection methods to support the development of evaluation designs and methodology if the intervention is selected for Service to Science consideration.

OUTCOME MEASURES AND DATA COLLECTION METHODS

Immediate (Process) Outcomes(Include service satisfaction)

Intermediate Outcomes Long-term Impact

Outcomes Anticipated

What data will be collected?

Who will data be collected from?

What data collection method or measurement instrument (e.g. survey) will be used?*

How will data be analyzed and stored?

How will data be used?

*Please attach measurement instruments when submitting this application.

Existing data and outcomes: If the intervention has already collected data on efficacy, please share outcomes achieved to date. If available, please reference and attach existing data reports from previous implementation.

OUTCOMES TO DATE

- 17 -

Dates of data collection period:

Immediate (Process) Outcomes(Include service satisfaction) Intermediate Outcomes Long-term Impact

Outcomes Achieved

# of participants from whom data was collected

Description of population from whom data were collected

Data collection method or measurement instrument (e.g. survey) used?

IX. NIDA COMPATIBILITY

The National Institute on Drug Abuse (NIDA) also provides guiding principles for evidence-based prevention practice. 8

Please indicate which principles are met by the proposed by the intervention.

1. Prevention programs should enhance protective factors and reverse or reduce risk factors(Hawkins et al. 2002).

2. Prevention programs should address all forms of drug abuse, alone or in combination, including the underage use of legal drugs (e.g., tobacco or alcohol); the use of illegal drugs (e.g., marijuana or heroin); and the inappropriate use of legally obtained substances (e.g., inhalants), prescription medications, or over-the-counter drugs (Johnston et al. 2002).

3. Prevention programs should address the type of drug abuse problem in the local community,target modifiable risk factors, and strengthen identified protective factors (Hawkins et al. 2002).

4. Prevention programs should be tailored to address risks specific to population or audiencecharacteristics, such as age, gender, and ethnicity, to improve program effectiveness (Oetting etal. 1997).

5. Family-based prevention programs should enhance family bonding and relationships andinclude parenting skills; practice in developing, discussing, and enforcing family policies onsubstance abuse; and training in drug education and information (Ashery et al. 1998).

6. Family bonding is the bedrock of the relationship between parents and children. Bonding canbe strengthened through skills training on parent supportiveness of children, parent-childcommunication, and parental involvement (Kosterman et al. 1997).

8 http://www.drugabuse.gov/pdf/prevention/redbook.pdf

- 18 -

7. Prevention programs can be designed to intervene as early as preschool to address risk factorsfor drug abuse, such as aggressive behavior, poor social skills, and academic difficulties (WebsterStratton 1998; Webster-Stratton et al. 2001).

8. Prevention programs for elementary school children should target improving academic andsocial-emotional learning to address risk factors for drug abuse, such as early aggression,academic failure, and school dropout. Education should focus on the following skills (Ialongo et al.2001; Conduct Problems Prevention Work Group 2002b):

self-control; emotional awareness; communication; social problem-solving; and academic support, especially in reading.

9. Prevention programs for middle or junior high and high school students should increaseacademic and social competence with the following skills (Botvin et al.1995; Scheier et al. 1999):

study habits and academic support; communication; peer relationships; self-efficacy and assertiveness; drug resistance skills; reinforcement of antidrug attitudes; and strengthening of personal commitments against drug abuse.

10. Prevention programs aimed at general populations at key transition points, such as thetransition to middle school, can produce beneficial effects even among high-risk families andchildren. Such interventions do not single out risk populations and, therefore, reduce labeling andpromote bonding to school and community (Botvin et al. 1995; Dishion et al. 2002).

11. Community prevention programs that combine two or more effective programs, such asfamily-based and school-based programs, can be more effective than a single program alone(Battistich et al. 1997).

12. Community prevention programs reaching populations in multiple settings—for example,schools, clubs, faith-based organizations, and the media—are most effective when they presentconsistent, community-wide messages in each setting (Chou et al. 1998).

13. When communities adapt programs to match their needs, community norms, or differingcultural requirements, they should retain core elements of the original research-basedintervention (Spoth et al. 2002b), which include:

Structure (how the program is organized and constructed); Content (the information, skills, and strategies of the program); Delivery (how the program is adapted, implemented, and evaluated).

14. Prevention programs should be long-term with repeated interventions (i.e., booster programs)to reinforce the original prevention goals. Research shows that the benefits from middle schoolprevention programs diminish without follow-up programs in high school (Scheier et al. 1999).

- 19 -

15. Prevention programs should include teacher training on good classroom managementpractices, such as rewarding appropriate student behavior. Such techniques help to fosterstudents’ positive behavior, achievement, academic motivation, and school bonding (Ialongo et al.2001).

16. Prevention programs are most effective when they employ interactive techniques, such aspeer discussion groups and parent role-playing, that allow for active involvement in learning aboutdrug abuse and reinforcing skills (Botvin et al. 1995).

X. RISK FACTOR CATEGORIES

Core factors influencing risk of alcohol and other drug use in New Hampshire as determined by the Strategic Prevention Framework assessments in 2007-2008 are Retail Access/Availability, Perception of Risk, Social Access/Availability, Social Norms, Enforcement, and Alcohol Pricing and Promotion. Please indicate which of the factors are addressed by the intervention.

NH Risk Factors Addressed by this Intervention

Retail Access/Availability Perception of Risk/Harm Social Access/Access Availability

Social Norms Enforcement

Alcohol Pricing and Promotion

XI. RESOURCES REQUIRED

Please estimate the resource required to replicate this intervention.

A. StaffingWhat staffing level is required to implement this intervention? You may respond with a percent of a full-time position, such as 1.0 FTE, or provide general context to staffing needs.

- 20 -

B. AccessWhat access, such as to focus populations, space, or special equipment, is necessary to implement the intervention? You may consider the following:

Focus populations

Existing media and communication channels Community institutions and local leaders for policy influence and other environmental

activities Space for meetings, forums and community-level trainings

Multi-sector representatives for leadership teams Technical assistance, resources, consultation and materials based on needs of locally

developed action plan

C. Other ResourcesWhat other resources may be needed to successfully implement the intervention?

Thank you for your application. You will be contacted shortly to review the application and determine next steps.

APPENDIX A. REVIEW PROCESS AND SCORING RUBRIC

NH’s SERVICE TO SCIENCE PROMISING PRACTICE AND EVIDENCE-BASED PRACTICE ENDORSEMENT

REVIEW PROCESS AND SCORING RUBRIC

The NH Center for Excellence, an initiative of the NH Bureau of Drug and Alcohol Services (BDAS), facilitates the determination of evidence-based interventions for the prevention of alcohol and other

- 21 -

drug abuse that emerge from the field of practice. There are two levels of evidence-based endorsements: 1) Promising Practice (Phase I); and 2) Evidence-Based (Phase II). These endorsements are made by the NH Bureau of Drug and Alcohol Services and the NH Center for Excellence Expert Panel for interventions who apply for NH Service to Science endorsement.

The following is the process by which NH Service to Science applicants are reviewed for Promising Practice determination:

Step One: The ApplicationInterventions designed to prevent or reduce alcohol or other drug abuse or to influence contributing factors that may influence risk or resiliency related to alcohol or other drug abuse complete an application for consideration as a promising practice. Applications are available at www.nhcenterforexcellence.org.

Step Two: Initial Review and Technical AssistanceA staff member at the NH Center for Excellence meets with applicants to discuss their intervention and determine what actions, if any, should be taken to strengthen the application in preparation for expert panel review. Actions may include more thorough literature reviews, an improved evaluation design, formalization of training or materials, or other activities. Whether or not the intervention is currently funded by the NH Bureau of Drug and Alcohol Services may determine whether technical assistance can be provided by the Center to strengthen an intervention’s application or if external consultation is required.

Step Three: Expert Panel ReviewThe following are considerations for evidence-based practice review recommended by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA).9 Expert Panelists for NH’s Service to Science process review application materials for satisfactory evidence of SAMHSA-recommended criteria detailed below.

The following two phases of SAMHSA Service to Science guidance are utilized by the expert panel as a means to determine if a program or practice is “promising” (Phase I) or “evidence-based” (Phase II). Applications that adequately meet at least 80% of the criteria for Phase I will be endorsed as a “Promising Practice”. Applications that adequately meet at least 80% of criteria for Phase I and II AND who have outcomes that show statistically significant change in indicators that research has supported as influencing risk and resiliency related to alcohol and other drug abuse will be endorsed as a “NH Evidence-Based Intervention” and be included on NH’s Registry of Evidence-Based Interventions. Please note: Those NH Service to Science applications for “Promising Practice” typically enter into a 12 to 18 month process during which the following technical assistance may be sought through the NH Center for Excellence or independently. The NH Center for Excellence has limited funds to support non-state funded interventions; therefore, technical assistance may be capped, after which programs or

9 http://www.samhsa.gov/grants/2005/standard/Srv2Sci/srv2sci_01.aspx

- 22 -

http://www.nhcenterforexcellence.org/

agencies may choose to continue technical assistance for a fee or may seek other consultants or research centers for their additional needs.

Information below is extracted and adapted from SAMHSA’s Service to Science process10:

Phase I provides limited support to stabilize and document an existing practice that fills an identified gap. During Phase I, applicants endorsed as a “Promising Practice” may:

further develop or refine the promising practice; develop training and practice manuals;

train persons who are implementing the practice; more systematically implement the practice;

develop improved measurement instruments; and ensure that the intended target population is being reached by the practice.

The desired endpoint of Phase I is readiness to conduct a high-quality, systematic evaluation and to have the evaluation collecting and reporting data that determine its effectiveness on indicators highly correlated with reducing or preventing alcohol and other drug abuse. The evaluation may use a pre-post approach, an open trial model, other quasi or non-experimental model, or an experimental model.

Phase II supports further refinement of the practice and of the protocols and processes to strengthen its replicability and evaluation. Phase II culminates in the outcomes of a high-quality, systematic evaluation documenting short-term and intermediate outcomes.

The desired endpoint for Phase II is readiness to submit an application for inclusion in NH’s Registry of Evidence-Based Interventions and/or to submit applications to various research institutions for additional research.

During Phase I, programs have been endorsed as a Promising Practice and will further develop and document the practice.

During Phase II, programs present practice refinements and evaluation outcomes for NH Evidence-Based Practice endorsement and inclusion on NH’s Registry of Evidence-Based Interventions.

10 http://www.samhsa.gov/grants/2005/standard/Srv2Sci/srv2sci_01.aspx

- 23 -

PROMISING PRACTICE CRITERIA: As stated earlier, applications for Promising Practice must adequately meet 80% of the following criteria for endorsement. Criteria are reviewed by a minimum of five expert panelists and are determined by the panel’s consensus.

To be completed by the Expert Panel.

Name of Practice: Date of Review:

SAMHSA CRITERIA (Phase I) or 0* Panel commentsA logic model depicting the principles and concepts underlying the practice. Documentation of how critical stakeholders were included in the development of the practice.

A detailed description of the population that the practice is designed to serve, and demographic characteristics of the people served by the practice over the past year.

Documentation of the number and percentage of staff trained in the practice, and a mechanism for ongoing training for any new staff.

A process evaluation demonstrating that the practice is in full operation and that a routine service delivery process is in place.

Pilot outcome results. (Note: Collection of these data need not include an extensive set of outcomes systematically collected on all participants, but quantitative project data should provide some indication that key outcomes are being achieved.)

Measurement instrument development/selection Participant recruitment methods appropriate for IOM population type for which program was designed

Development of quality assurance and accountability mechanisms and data collection to support these mechanisms

Implementation of the practice Process evaluation (what process data is collected, and how it is used)

- 24 -

SAMHSA CRITERIA (Phase I) or 0* Panel commentsManagement information system development (how information is collected and stored)

Key outcome measures clearly identified and defined Collection of pilot outcome data Evidence of service satisfaction Systematical collection intermediate outcome measures that will indicate meaningful results

Participant data collection systems are in place that include: - Demographic characteristics - Practice outcomes - Service utilization Other NH CriteriaDocumentation of core elements of the practice that can support fidelity measures: - Content relevant to alcohol and other drug abuse risk and resiliency - Staffing/training requirements - Intensity (dosage and duration) - Method of Delivery - Location

* = adequately meets criteria; 0 = missing or inadequate

- 25 -

By the end of Phase II, the evaluation of the practice must have demonstrated adequate fulfillment of all Phase I criteria in addition to 80% of the following Phase II criteria:

SAMHSA CRITERIA (Phase II) or 0* Panel commentsA manual describing the practice and its core elements in detail that would allow others to replicate the practice.

Documentation that the number of people being served by the practice has been stabilized. A fidelity scale has been developed for assessing the integrity of the practice, and the practice has been implemented with fidelity according to the scale.

Use of quality assurance and accountability mechanisms to improve and refine practice Key outcome measures have been clearly identified and defined. Participant data collection systems are in place that include: Demographic characteristics Practice outcomes Service utilization Service delivery costs Satisfaction with services Demographic characteristics of participants, as well as the types of services that participants have received, are consistent with expectations based on the logic model for the practice.

Service delivery patterns are stable. A fidelity scale has been developed and implemented for assessing the integrity of the practice, and the practice has been implemented with fidelity according to the scale.

Systematically collected short-term outcome measures indicate meaningful results. Consumers, family members, and other critical stakeholders are satisfied with the practice. Systematically collected intermediate outcome measures indicate statistically significant change on indicators relative to alcohol and other drug abuse risk prevention and resiliency promotion

- 26 -

SAMHSA CRITERIA (Phase II) or 0* Panel commentsDissemination of results to external partners, stakeholders, other providers to strengthen visibility and sustainability of the practice

Dissemination of results internally to staff, administration, participants and others to support quality improvement

Methods and processes to determine service delivery costs Methods and processes to determine service delivery patterns

* = adequately meets criteria; 0 = missing or inadequate

- 27 -

SCORE SUMMARY

Name of Practice: Date of Review:

PHASE I: /24 = % SUMMATIVE COMMENTS:

Endorsed as Promising Practice Not Endorsed as Promising Practice

PHASE II:

Phase I Criteria: / 24 + Phase II Criteria /20

TOTAL: /44 = %

SUMMATIVE COMMENTS:

Endorsed as NH Evidence-Based Intervention

Not Endorsed as NH Evidence-Based Intervention

- 28 -

evidence-based intervention:1viuw040k2mx3a7mwz1lwva5-wpengine.netdna-ssl.com/wp... · web viewupon...

Documents