DEMYSTIFYING REFLECTIVE PRACTICE: USING THE DATA

MODEL TO ENHANCE EVALUATORS’ PROFESSIONAL

ACTIVITIES

Tiffany L. SmithJohn M. Peters

Gary J. SkolitsPatrick B. Barlow

Expert LectureAmerican Evaluation Association 2013

Friday, October 18th, 1:45-2:30pm

“Insights and innovation await us only if we are capable of stepping outside the frenzied worlds of data and distraction that wash over us…

time for reflection is an open invitation to discover what awaits us…” (Forrester, 2011, pp. 216-217)

INTRODUCTION: WHO ARE WE?

TIFFANY SMITH A doctoral candidate in the Evaluation, Statistics and Measurement program at the University of Tennessee Knoxville. She has taken two courses on Reflective Practice and has interned with the Institute for Reflective Practice at the University of Tennessee with John Peters. Her primary research interests include stakeholder communication and involvement as well as reflective practice in evaluation.

JOHN PETERS The Director of the Tennessee Teaching and Learning Center’s Institute for Reflective Practice. The primary mission of the Institute is to promote reflective practice by individuals and organizations served by the University of Tennessee, Knoxville. Its service is based on the belief that addressing and managing change in individual lives and organizations demands flexible thinking processes by persons capable of proactive interpersonal communication.

GARY SKOLITS The Director of the Institute for Assessment and Evaluation at the University of Tennessee’s in the College of Education, Health, and Human Sciences. He is a tenured associate professor in the Evaluation, Statistics, and Measurement Ph.D. program. His research interests include evaluation methods, the evaluation of educational interventions, P-16 and college access program evaluation as well as higher education accountability.

PATRICK BARLOW A doctoral candidate in the Evaluation, Statistics, and Measurement program at the University of Tennessee. He works as a Statistical and Research Design Consultant for the University of Tennessee Graduate School of Medicine. His primary research areas are in higher education assessment and teaching statistics and research methods in the social sciences, epidemiology, and clinical medical fields.

WHY REFLECT?

“Reflective practice is more than spending ten minutes at the end of an evaluation

congratulating oneself on getting the damn thing done” (Patton, 2012, p. 401).

REFLECTIVE PRACTICE IN EVALUATION

• One of six Essential Competencies for program evaluators, defined as:– “being acutely aware of personal evaluation

preferences, strengths, and limitations; self-monitoring the results of actions intended to facilitate effective evaluation studies; and planning how to enhance future endeavors” (Stevhan, King, Ghere, & Minnema, 2005, p. 46)

HOW DOES REFLECTION IMPROVE EVALUATION PRACTICE?

• Can be used as a means for self-awareness, professional growth and development, improved ethical practice, dialogue and stakeholder communication, and for learning in organizations

• The reflection process can happen while engaging in daily practice as well as after the fact, either alone or with others.

• Reflection involves being a student of the actions that you make, and the study of those actions is systematic and analytical

Patton (2012) reports that “in speeches and workshops at professional evaluation

association meetings, I like to ask for a show of hands of those who systematically reflect

on evaluations they have conducted for learning and further professional

development. Few hands go up; typically, in fact, no one raises a hand” (p. 400).

A SHOW OF HANDS?

Who systematically reflects on evaluations they have conducted for learning and further

professional development?

Why Not?

Why is reflective practice not a part of regular evaluation practice?

WHY IS RP NOT A PART OF REGULAR EVALUATION PRACTICE?

• Perhaps there is a lack of awareness of the purpose of reflection in evaluation?

• Unclear what reflective practice is in the first place?

According to Peters (1991), “reflective practice involves more than simply thinking about what

one is doing and what one should do next. It involves identifying one’s assumptions and feelings associated with practice, theorizing

about how these assumptions and feelings are functionally or dysfunctionally associated with

practice, and acting on the basis of the resulting theory of practice” (p. 89).

WHAT IS RP, ANYWAY?

THE DATA MODEL FOR REFLECTION

• DESCRIBE• ANALYZE• THEORIZE• ACT

HOW HAS DATA BEEN USED AS A REFLECTIVE PRACTICE EFFORT?

• Mainly to guide individual and group reflection on a variety of work-related issues, train teams of workers, and for supervisory training.

• Also as major component of action research projects in community education programs, business applications, and in higher education settings

DESCRIBE

• A detailed account of the situation, task, or incident that has happened in one’s practice.

• Concerned with identifying the specifics of the situation at hand. – Context of the situation, the setting in which it occurred,

who was involved, etc.• Use as much detail as possible, so as to reflect on

the whole of the problem and its setting.• Everything that you know about the situation and

context are key to this step of the reflective process.

DESCRIBE

• This might prove to be harder than one might expect– A description only involves accounts of what actually

occurred• Description is devoid of any why inclinations or

explanations in order that a clear picture of the problem is painted (Peters, 2009).

• This portion of the DATA model could prove to be beneficial if one includes other colleagues, be it evaluators or stakeholders.

ANALYZE

• The why that it was necessary to avoid during the Describe phase gets its turn.

• Identification of factors contributing to the situation.

• Examination of one’s assumptions, biases, and preconceived notions

ANALYZE

• Beliefs, rules, motives, and facts should be things that the practitioner is aware of during this stage of reflection.

• Through the first two steps of DATA, problem is at least tentatively outlined– The practitioner should have a very clear and holistic

understanding of the problem, its context, and why it has come up in their practice.

• This will in turn produce a practical question of how to solve the problem.– How, specifically, can I change my practice to produce a better

outcome?

THEORIZE

• The focus is on answering the practical question introduced in the analysis.

• Refers to deriving a practical theory from the description and analysis in order to improve one’s practice.– A practical theory refers to understanding the structure and

implications of a theory’s use in professional practice.• This does not need to be a scholarly theory, derived

from the literature, although previous literature may influence this section

• This theory comes from the practice in which one works.

THEORIZE

• At the end of Theorize, the practitioner should be able to ask What am I going to do, and Why is this solution better than other potential solutions?

• How many options do you have? • Is there relevant literature or research to indicate that

this option is better over others?• In practice, what has worked before, and how well?• The Theorize portion of DATA is intended to provide

practitioners with a theoretical solution to their concern.

ACT

• Taking action.• What are you going to do about it, specifically?• Moving forward based on a practical theory which is

derived from reflection on the situation, including the what and the why.

• When one takes action, it is in the context and for the betterment of their practice.

• This action tests the practical theory that was developed via this reflective process.

• This can lead to further reflection on one’s practice and further professional development.

A CASE FOR REFLECTION

Michael is a seasoned evaluator who has worked many different evaluation projects over the past

20 years. The following is a case of Michael utilizing the DATA model of reflective practice in order to understand and solve a dilemma in his

evaluation work.

DESCRIBE

At the beginning of the evaluation project:

• Contract awarded for 3 years of funding– Serving a true community need

• Stakeholders and evaluators come together to define the evaluation design, data collection, schedules, procedures

• Client not familiar with evaluation or working with evaluators and very reluctant and cautious– Displaying signs of confusing evaluator with auditor

• Client staffs project and project data collection and reporting based on evaluation design

DESCRIBE• About a year in, change in requirements from funding agency

– Reporting is vastly increased – Increase in the amount of data to be collected by tenfold– Complications for collecting the data are greatly increased– Project itself is being redefined by new data collection requirements

• Changes mandated at the policy level resulting from inner turmoil in funding agency

• This issue requires a total renegotiation of the client’s data collection process as well as the evaluation design, breaking all of the prior agreements.

• Data collection is complex, training implemented that they never counted on

Due to this issue, the evaluation is struggling. Evaluative progress is not being made.

ANALYZE

• Why is this situation occurring?– Evaluator’s assumption that no major policy change

would occur was totally inaccurate; after 20 years of practice, never has such a change been encountered

– Examination of how evaluator’s behavior could have helped to create the situation

– Examine angers and frustrations– Evaluator and client, and now multiple stakeholders

come to new understandings about what is expected and required

• Sense of a weak relationship, a weak tie, and a mistrust in the stakeholders

Practical Question: How can we work to reestablish trust in the evaluation and program

environment?

THEORIZE• Utilization of Program Evaluation Standards (as well as ethical

guidelines, Essential Competencies, Utilization Focused evaluation) to decide on the most effective means of establishing trust.– Some specific Standards: F1, F2, F4, U1, U2, U3, U4, U8, P2, P4, P7

• Discusses options with trusted colleagues

• After thinking about other options, including termination of the project, Michael decides that the best strategy would be:– Face to face discussion with primary stakeholders

• Review the status of the relationship prior to the change (contracting, working relationship)

• Review the relationship after the change• Gauge the perspectives of stakeholders• Is there a basis for continuing? Want? • Both the evaluation staff and the client must be willing to put the cards on the table

and understand that this was an imposed change.

ACT

• Michael acts on the basis of a practical theory.

• Renegotiation of agreement

• Re-establishment of trust through critical conversation with stakeholders

USING DATA IN EVALUATION

• How did reflection on the situation help Michael to come up with a plan of action?– How could it have improved his professional

practice? Way of thinking?• When is DATA practical for evaluators?• Should RP be an alone thing? Or is this best

done collaboratively?

USING DATA IN ACTION RESEARCH: THE DATA-DATA APPROACH

• DATA-DATA has been used to plan and conduct more than 125 projects in higher education, community education, and business.– Design– Analyze– Theorize– Act

• Using the second DATA when you want to have more confidence in the result of your thinking and action that results from the first DATA.

• The systematic design work and careful analysis in the second DATA accomplishes this.

WHAT DOES REFLECTION PROVIDE FOR EVALUATION PRACTITIONERS?

• A way to step back from daily practice and to create an encompassing understanding of the issues and situations that we face

• Reflecting helps us to learn more about context• Reflection provides us an opportunity to

investigate our own assumptions and how they influence our actions.

• How can RP be integrated into evaluation work?

Can anyone think of a way that the DATA model can be used in their evaluation work?

CONTACT INFORMATION:Tiffany Smith: tsmith92@utk.eduJohn Peters: jpeters@utk.eduGary Skolits: gskolits@utk.eduPatrick Barlow: pbarlow1@utk.edu

Institute for Reflective Practice: irp@utk.edu

QUESTIONS?