2009 siym theory, research, practice, and profession evidence_final

70
Theory, Research, Practice, Profession: Institute objectives and themes Thomas Keller

Upload: kay-logan

Post on 18-Dec-2014

200 views

Category:

Documents


4 download

DESCRIPTION

 

TRANSCRIPT

Page 1: 2009 siym theory, research, practice, and profession evidence_final

Theory, Research, Practice, Profession:

Institute objectives and themes

Thomas Keller

Page 2: 2009 siym theory, research, practice, and profession evidence_final

Introduction

Field of youth mentoring Interplay of theory, research, practice Ideas and aims of summer institute Participation and feedback Research principles and statistical analysis Conceptual frameworks for mentoring and

mentoring relationships

Page 3: 2009 siym theory, research, practice, and profession evidence_final

Field of youth mentoring Origins

Juvenile court probation (late 1800’s) Big Brothers Big Sisters (1904)

The Kindness of Strangers (Freedman, 1993) Interest and growth Positive youth development Potential benefits “Fervor without infrastructure”

Continued expansion MENTOR (1990) PPV study (1995) OJJDP/gov’t funding (1990’s) Handbook of Youth Mentoring (2005)

Page 4: 2009 siym theory, research, practice, and profession evidence_final

Development of field

Attributes of a profession (Greenwood, 1957): Skill based on theoretical knowledge Period of formal training and education Occupational organization Code of ethics or conduct Some form of community-sanctioned license

Page 5: 2009 siym theory, research, practice, and profession evidence_final

Continuing challenges

Expanding and solidifying base of knowledge (practice knowledge and research)

Sharing knowledge and education (bi-directional communication)

MENTOR National Research & Policy Council (September, 2006): “What do practitioners need to know and what is

he best way to disseminate needed information to them?”

Page 6: 2009 siym theory, research, practice, and profession evidence_final

Interrelations of theory, research, practice

Theory Research

Practice

Page 7: 2009 siym theory, research, practice, and profession evidence_final

Sequence Define need/problem Identify associated factors Develop explanatory model Create intervention based on model Pilot test intervention Refine intervention Efficacy trial Effectiveness trials Dissemination

Page 8: 2009 siym theory, research, practice, and profession evidence_final

Alternative sequence

Theory Practice

Research

Page 9: 2009 siym theory, research, practice, and profession evidence_final

Common language

Idea Action

Observation

Page 10: 2009 siym theory, research, practice, and profession evidence_final

Everyday example What should I do?

You have a reason or rationale based on past experience and/or logic (explanation)

You expect a certain result (prediction) Did it work?

Determine result Confirming evidence (reinforces your idea/approach) Disconfirming evidence (need to remember this and try

something different next time) Other examples

Learning and survival Scientist in the crib Internal working model Reflective practice

Page 11: 2009 siym theory, research, practice, and profession evidence_final

Personal vs. formal

Personal theory Intuition Common sense

Personal evaluation Observations and

recollections Interpretations and

meanings Result: convince

yourself (belief)

Formal theory Stated propositions Logical case

Formal evaluation Systematic collection of

data Systematic analysis of

data Result: convince others

(evidence)

Page 12: 2009 siym theory, research, practice, and profession evidence_final

Cautions The main difference:

Making it explicit, documentation The common risk:

Want to hold on to our ideas/beliefs Avoid any evidence Pay attention to confirming evidence Disregard disconfirming evidence Twist and stretch theory to match disconfirming

evidence

Page 13: 2009 siym theory, research, practice, and profession evidence_final

Research and practice

Internal validity—study has solid design that permits strong conclusions (quality)

External validity—study conclusions apply beyond specific sample (relevance)

Page 14: 2009 siym theory, research, practice, and profession evidence_final

Different roles

Theory

PracticeResearch

Page 15: 2009 siym theory, research, practice, and profession evidence_final

Modified model

Ideas

Action

Observation

Action

Observation

Practice Research

Page 16: 2009 siym theory, research, practice, and profession evidence_final

Real-life examples

Idea Action

Observation

Page 17: 2009 siym theory, research, practice, and profession evidence_final

Example: School-based mentoring

Opportunities and potential (Herrera, 1999) Recruitment of volunteers (time, convenience) Reach children with greater needs/challenges (school

referrals) Fewer staff resources (less screening, easier supervision) Focus on academic development

Rapid growth in BBBSA (Herrera, 2004) 1999=27,000; 2002=90,000 233% increase

Careful evaluation of effects (Herrera, et al., 2007)

Page 18: 2009 siym theory, research, practice, and profession evidence_final

Example: Summer Institute Goal: To improve services for youth who participate

in formal youth mentoring programs Premise: A sustained dialogue between

experienced professionals and researchers stimulates research with relevance to the field and enhances its translation to practical application

Strategy: A direct relationship between researchers and professionals

Model: A series of highly interactive discussions that provide an in-depth view of the research and examine its implications for program policies and practices

Page 19: 2009 siym theory, research, practice, and profession evidence_final

Model elaborated People

Professionals eager for cutting edge research and exchange of information and ideas

Researchers who want work to be relevant and want findings translated into improvements in services for youth

Size Small-group format to encourage active exchange

Time Ample time to think critically and creatively about issues

and explore opportunities for innovation Get away and go back to school

Dynamics Cohort effect Mixing within and between roles

Page 20: 2009 siym theory, research, practice, and profession evidence_final

Mixture model

Advocates Program Leaders

Researchers

YOU

Page 21: 2009 siym theory, research, practice, and profession evidence_final

Professional development and career

Training tied to transitions (Caplan & Curry, 2001) Internship: Transition from student to worker

Transfer knowledge from class to real-life Entry-level: Transition to professional

Learn skills and tasks of position Leadership development: Transition to leader in

organization Preparation for supervision and management

Master practitioner: Transition to leader in the field Special opportunities for experienced professionals

Page 22: 2009 siym theory, research, practice, and profession evidence_final

Leaders Wisdom and insight to share at institute Wisdom and insight to share in communities Transfer of learning

Hold positions of influence Training and supervision of staff Development of program models Implementation of service delivery changes

Task Think about new program models, program policies and

procedures, training materials for program staff, training materials for volunteer mentors and youth participants

Page 23: 2009 siym theory, research, practice, and profession evidence_final

Summer Institute aims Contribute to the development of policy and

practice in the field of youth mentoring Convene leaders in the field for substantive

discussion of practices, policies, and new directions

Create new networks of peer relationships among professionals and researchers from different programs and backgrounds

Promote professional identity and commitment of participants and researchers

Page 24: 2009 siym theory, research, practice, and profession evidence_final

Moving forward

Idea Action

Observation

Page 25: 2009 siym theory, research, practice, and profession evidence_final

Preliminary observations

Support of advocacy agencies providing training and technical assistance? Yes! (support with plans and announcements)

Interest on part of researchers? Yes! (eager to attend)

Interest on part of professionals? Yes! (competitive application process)

Ability to reach target participants? Yes! (look around)

Page 26: 2009 siym theory, research, practice, and profession evidence_final

Study

Encourage reflection Provide feedback Respond to questions at end of institute Respond to questions after 6-12 months Informed consent

Page 27: 2009 siym theory, research, practice, and profession evidence_final

Questions What types of opportunities for collaboration

among colleagues (research-practice, practice-practice, research-research) do you see emerging from the institute: a) this week and b) continuing into the future?

What types of initiatives or changes could you undertake upon returning to your program?

How will you share the new information and ideas with others in your agency or community?

Page 28: 2009 siym theory, research, practice, and profession evidence_final

Assignment

Assignment for each presentation Summarize (few sentences) Follow-up questions (2-3) Implications for program (example)

Page 29: 2009 siym theory, research, practice, and profession evidence_final

SIYM Themes

Year 1: School-based Mentoring Reprise in year 3:

Symposium on Monday Guest speakers on Friday

Year 2: Diversity in Mentoring Year 3: Use of Evidence for Practice

MENTOR—Elements of Effective Practice, 3rd Ed. BBBSA program models (SBM, CBM) New Meta-analysis Standards and accreditation discussions

Page 30: 2009 siym theory, research, practice, and profession evidence_final

Research-Practice Survey

WT Grant Distinguished Fellows Use of high quality research in practice Marc Wheeler, VP Programs BBBS Alaska (SIYM

alum) to PSU David DuBois, to BBBSA

Web-based Survey Opinion survey Needs assessment Descriptive data

Page 31: 2009 siym theory, research, practice, and profession evidence_final

Survey: sample

Sample Responses: n=455 Positions

ED/CEO (39%) VP of Program/Program Director (18%) Program Coordinator (25%)

Mostly with BBBS agencies (54%) Average employment in field=8.9 yrs.

Page 32: 2009 siym theory, research, practice, and profession evidence_final

Survey: current assessment

Current level of EB decision making in field A lot more needed (30%) Somewhat more needed (55%) About right (10%) Less (5%)

Personally comfortable using EB decision making Very/somewhat uncomfortable (26%) Neutral (9%) Somewhat comfortable (33%) Very comfortable (32%)

Page 33: 2009 siym theory, research, practice, and profession evidence_final

Survey: type of data

Use of published research Haven’t (11%) Small steps (37%) Creating systems (16%) Have systems to

routinely do this (37%)

Use of internal agency data Haven’t (7%) Small steps (31%) Creating systems (22%) Have systems to

routinely do this (40%)

Page 34: 2009 siym theory, research, practice, and profession evidence_final

Survey: evidence used

Use of the following: Little/none Somewhat Substantial Extensive

Published external research 18% 37% 34% 11%

Data on client/stakeholder preferences

17% 28% 42% 14%

Professional experience/expertise

7% 17% 50% 27%

Data on local trends/needs 13% 31% 40% 16%

Performance data on program operation/outcome

7% 21% 40% 33%

Page 35: 2009 siym theory, research, practice, and profession evidence_final

Survey: reasons to use evidenceGoals: Un-

importantNeutral Somewhat

importantVery important

Improve youth outcomes 5% 1% 11% 83%

Demonstrate impact to funders

4% 3% 14% 78%

Prevent negative outcomes for youth

7% 8% 26% 60%

Focus resources on effective areas

4% 4% 33% 59%

Design new programs for specific populations

6% 8% 38% 49%

Guide decisions of board and stakeholders

6% 6% 41% 47%

Page 36: 2009 siym theory, research, practice, and profession evidence_final

Survey: Greatest needs (very imp.)

1. How to analyze data and report findings (59%)2. Step-by-step guide for EB decision making (56%)3. How to find/select measures/metrics (56%)4. How to find, read, use existing research (54%)5. Description of different types of evidence (42%)6. How to collaborate with researchers/colleagues on

EB decision making (38%)7. Glossary of common research terms (37%)8. Guidance on ethical issues with research (36%)9. Description of scientific method and applications

(29%)

Page 37: 2009 siym theory, research, practice, and profession evidence_final

Research principles and statistical analysis

Briefly….

Page 38: 2009 siym theory, research, practice, and profession evidence_final

Classic questions

What works for whom under what circumstances? Why? Does program work better/differently for certain types of

mentees (age, gender, race, stress, aptitude)? Does program work better/differently in certain settings

(community, school, etc.)? Does program work better/differently with certain types

of volunteers (age, gender, occupation, personality)? What are the essential processes that yield the results?

Page 39: 2009 siym theory, research, practice, and profession evidence_final

Research paradigms

Qualitative approach Complex data Smaller samples

Strengths Reflects complexity of

experience Captures contexts and

processes Good for discovering what

is happening Good for within-system view

Limitations Subjective interpretations Translating results to others

Quantitative approach Defined data Larger samples

Strengths Reflects clear definitions

and theories Captures relations between

variables Good for demonstrating

what is happening Good for comparisons

Limitations Less nuanced Sampling biases

Page 40: 2009 siym theory, research, practice, and profession evidence_final

Research Process

Introduction Framing the issue Motivation, rationale Theory/model Research questions

Method Design Sample Procedures Measures Analysis plan

Results Data Findings

Discussion Conclusions/insights Interpretations Limitations Next steps

Page 41: 2009 siym theory, research, practice, and profession evidence_final

Outcomes Criteria for selecting “outcomes”

Outcome can reasonably be expected to change during period, given intensity of intervention

Proximal (intervening) vs distal (ultimate) Number of other uncontrolled factors Portion of variance explained

Outcome is measurable and assessment is sensitive enough to detect likely change

Clearly and narrowly defined Reliable measure Valid measure

Page 42: 2009 siym theory, research, practice, and profession evidence_final

Measures

Validity

XXX

X

X

OOO

OO

Z

Z Z

Z

Z

Reliability

Page 43: 2009 siym theory, research, practice, and profession evidence_final

Sampling

Defining population of interest Representative members Random sampling (vs. assignment)

Each individual has equal chance of being selected

Population parameter vs. sample statistic Inferences apply to population

Page 44: 2009 siym theory, research, practice, and profession evidence_final

Sampling distribution Unlikely to get a sample statistic exactly equal to

population parameter (sampling error). Imagine a hypothetical sampling distribution if you

took multiple samples from population and plotted all the sample statistics.

Central Limit Theorem:If a population has a mean of μ and a standard deviation of σ, then

the sampling distribution of the mean based on sample size N will have a mean of μ and a standard deviation of σ/sq root (N) and will approach a normal distribution as the sample size N upon which it is based becomes larger (regardless of population distribution).

Page 45: 2009 siym theory, research, practice, and profession evidence_final
Page 46: 2009 siym theory, research, practice, and profession evidence_final

Evaluation Design Logic

We can determine the true effect of a program (or experience) if we compare what happens to an individual who is in the program versus what would have happened to that individual if he or she were not in the program (impossible in one lifetime)

Problem We always lack the ideal counterfactual (the outcome in the

what if situation…) Missing data solution

Compare participants to non-participants who are as similar as possible in every way except for having or not having the intervention

Page 47: 2009 siym theory, research, practice, and profession evidence_final

Evaluation Design

How do we get comparison group? Experimental design (optimal)

Researcher controls exposure to intervention through random assignment to intervention

Quasi-experimental designs Research tries to get a non-participant comparison

group as equal/similar as possible

Page 48: 2009 siym theory, research, practice, and profession evidence_final

Random assignment means everyone has equal chance of being in program

Imagine this dimension is motivation to succeed. We would have equal distribution of low, middle, high

among participants and “control group”With random assignment, this would be true on EVERY dimension

(observed and unobserved).

Without random assignment, they may differ on important dimensions. For example, program participants may have higher motivation to succeed

than non-participants—that’s why they signed up

Page 49: 2009 siym theory, research, practice, and profession evidence_final

Experimental design

Pre-test Post-test

Program (assume equivalence) Xp

Control Xc

Program X Xp

Control X Xc

Test of effect = mean (Xp) – mean (Xc)

Page 50: 2009 siym theory, research, practice, and profession evidence_final

Experimental comparisonPre-test Post-test

Program group

Control group

Program group

Control group

Page 51: 2009 siym theory, research, practice, and profession evidence_final

No control groupPre-test Post-test

Program group

Program groupControl group

Page 52: 2009 siym theory, research, practice, and profession evidence_final

Comparing Group Means

State a null hypothesis (e.g. X1 – X2 = 0) Create sampling distribution for difference

between means. Compare observed difference between

means to null hypothesis. If difference is relatively small, it could be due

to sampling error (p>.05, FAIL to reject null). If difference is relatively large, it is unlikely

due to sampling error (p<.05, REJECT null) and conclude actual difference exists.

Page 53: 2009 siym theory, research, practice, and profession evidence_final

Errors

Conclusions are based on probabilities Conclusions can be incorrect

Reject null hypothesis when we shouldn’t (freaky sample)

Fail to reject null hypothesis when we should (don’t detect actual difference) Low statistical power, need larger sample

Page 54: 2009 siym theory, research, practice, and profession evidence_final
Page 55: 2009 siym theory, research, practice, and profession evidence_final
Page 56: 2009 siym theory, research, practice, and profession evidence_final

Observational studies

No imposed difference between groups--naturalistic observation

Can see how certain variables correspond Correlation Regression Multiple regression (can evaluate one factor controlling for

others) Sampling distribution for each estimate (assumption

of no association) Causal inference depends on several considerations

(temporal order, ruling out other explanations, etc.)

Page 57: 2009 siym theory, research, practice, and profession evidence_final

PSI--Total stress, T7

14012010080604020

CT

S_

7

50

40

30

20

10

0

-10

Page 58: 2009 siym theory, research, practice, and profession evidence_final
Page 59: 2009 siym theory, research, practice, and profession evidence_final

Model Summary

.415a .172 .146 .95353Model1

R R SquareAdjustedR Square

Std. Error ofthe Estimate

Predictors: (Constant), bm_wsafa.

Coefficientsa

.083 1.455 .057 .955

.607 .239 .415 2.542 .016

(Constant)

bm_wsaf

Model1

B Std. Error

UnstandardizedCoefficients

Beta

StandardizedCoefficients

t Sig.

Dependent Variable: fm_clowa.

Page 60: 2009 siym theory, research, practice, and profession evidence_final

Orienting frameworks

Central ideas….

Page 61: 2009 siym theory, research, practice, and profession evidence_final

Development

Universal processes vs. individual differences Successive adaptations

Past experience Current circumstances

Continuity and change Change possible at any time Change constrained by prior adaptation

Diversity in process and outcome Equi-finality Multi-finality

Page 62: 2009 siym theory, research, practice, and profession evidence_final

Developmental adaptation

Source: Sroufe, L.A. (1997). Psychopathology as and outcome of development. Development & Psychopathology, 9, 251-268.

Page 63: 2009 siym theory, research, practice, and profession evidence_final

Mentoring relationships

What distinguishes relationships? (Laursen & Bukowski, 1997)

Permanence Voluntary, kinship, committed

Social power Resources, experience/knowledge, rank

Gender Male-male, female-female, cross-gender

Page 64: 2009 siym theory, research, practice, and profession evidence_final

Relationship dimensions

Permanent

(obligation)

Voluntary

(mutual)

Unequal social power

(vertical)

Parent Mentor

Equal social power

(horizontal)

Cousin Friend

Page 65: 2009 siym theory, research, practice, and profession evidence_final

Systemic model

Child

Mentor

Parent Worker

Program/Agency

Page 66: 2009 siym theory, research, practice, and profession evidence_final

Systemic model Conceptual points

Wholeness and order Parts are interconnected and interdependent

Hierarchical structure Composed of sub-systems with boundaries

Practical points Intervention goes beyond mentor-child relationship Caseworker, parent, teacher contribute to success or

failure of relationship Mentoring effects can be indirect, through multiple

pathways of influence

Page 67: 2009 siym theory, research, practice, and profession evidence_final

Systemic model

Analytical uses Direct (M C) Reciprocal (M C) Transitive (W M, M C) Parallel (W M, W C, M C) Circular (C W, W M, M C)

Page 68: 2009 siym theory, research, practice, and profession evidence_final

Developmental stages

Contemplation

Initiation

Growth & Maintenance

Decline & Dissolution

Redefinition

Page 69: 2009 siym theory, research, practice, and profession evidence_final

Stage Conceptual features Program practices

ContemplationAnticipating and preparing for relationship

Recruiting, screening, training

InitiationBeginning relationship and becoming acquainted

Matching, making introductions

Growth and maintenance

Meeting regularly and establishing patterns of interaction

Supervising and supporting, ongoing training

Decline and dissolution

Addressing challenges to relationship or ending relationship

Supervising and supporting, facilitating closure

RedefinitionNegotiating terms of future contact or rejuvenating relationship

Facilitating closure, rematching

Stage model

Page 70: 2009 siym theory, research, practice, and profession evidence_final

Final thoughts

Human beings of all ages are happiest and able to deploy their talents to best advantage when they are confident that, standing behind them, there are one or more trusted persons who will come to their aid should difficulties arise. --John Bowlby