evaluating organizational change: how and why? dr kate mackenzie davey organizational psychology...
TRANSCRIPT
Evaluating Evaluating Organizational Organizational Change: How and Change: How and Why?Why?
Dr Kate Mackenzie DaveyDr Kate Mackenzie DaveyOrganizational PsychologyOrganizational PsychologyBirkbeck, University of LondonBirkbeck, University of [email protected]@bbk.ac.uk
22
AimsAims
Examine the arguments for evaluating Examine the arguments for evaluating organizational changeorganizational change
Consider the limitations of evaluationConsider the limitations of evaluation Consider different methods for Consider different methods for
evaluationevaluation Consider difficulties of evaluation in Consider difficulties of evaluation in
practicepractice Consider costs and benefits in practiceConsider costs and benefits in practice
33
Arguments for Arguments for evaluating evaluating organizational changeorganizational change Sound professional practice Sound professional practice Basis for organizational learning Basis for organizational learning Central to the development of Central to the development of
evidence based practiceevidence based practice Widespread cynicism about fads Widespread cynicism about fads
and fashionsand fashions To influence social or To influence social or
governmental policy governmental policy
44
Research and Research and evaluationevaluation Research focuses on relations between Research focuses on relations between
theory and empirical material (data)theory and empirical material (data)– Theory should provide a base for policy decisionsTheory should provide a base for policy decisions– Evidence can illuminate and inform theoryEvidence can illuminate and inform theory– Show what does not work as well as what doesShow what does not work as well as what does– Highlight areas of uncertainty and confusionHighlight areas of uncertainty and confusion– Demonstrate the complexity of cause-effect Demonstrate the complexity of cause-effect
relationsrelations– Understand predict controlUnderstand predict control
55
Pragmatic Evaluation: Pragmatic Evaluation: what matters is what what matters is what worksworks Why Why it works may be unclearit works may be unclear Knowledge increases complexityKnowledge increases complexity Reflexive monitoring of strategy links Reflexive monitoring of strategy links
to OL & KMto OL & KM Evidence and cultural contextEvidence and cultural context May be self fulfillingMay be self fulfilling Tendency to seek support for policyTendency to seek support for policy Extent of sound evidence unclearExtent of sound evidence unclear
66
Why is sound Why is sound evaluation so rare? evaluation so rare? Practice shows that evaluation is Practice shows that evaluation is
an extremely complex, difficult an extremely complex, difficult and highly political process in and highly political process in organizations.organizations.
Questions may be how many, not Questions may be how many, not what workswhat works
77
Evaluation modelsEvaluation models
1.1. Pre-evaluationPre-evaluation2.2. Goal based (Tyler, 1950)Goal based (Tyler, 1950)3.3. Realistic evaluation (Pawson & Tilley,1997; Realistic evaluation (Pawson & Tilley,1997;
Sanderson, 2002)Sanderson, 2002)4.4. ExperimentalExperimental5.5. Constructivist evaluation (Stake, 1975)Constructivist evaluation (Stake, 1975)6.6. Contingent evaluation (Legge, 1984)Contingent evaluation (Legge, 1984)7.7. Action learning (Reason & Bradbury, 2001) Action learning (Reason & Bradbury, 2001)
A study should be technically sound, A study should be technically sound, administratively convenient and politically administratively convenient and politically defensible. Alec Rodgerdefensible. Alec Rodger
88
1.1 Pre-evaluation 1.1 Pre-evaluation (Goodman & Dean, 1982)(Goodman & Dean, 1982)
The extent to which it is likely The extent to which it is likely that... A has an impact on bthat... A has an impact on b Scenario planningScenario planning Evidence based practiceEvidence based practice
– All current evidence thoroughly All current evidence thoroughly reviewed and synthesisedreviewed and synthesised
– Meta-analysisMeta-analysis– Systematic literature reviewSystematic literature review
Formative v summative Formative v summative (Scriven, 1967)(Scriven, 1967)
99
1.2 Pre-evaluation 1.2 Pre-evaluation issuesissues Based on theory and past Based on theory and past
evidence: not clear it will evidence: not clear it will generalise to the specific casegeneralise to the specific case
Formative: influences planningFormative: influences planning Argument: to understand a Argument: to understand a
system you must intervene system you must intervene (Lewin)(Lewin)
1010
2. 1. Goal based evaluation 2. 1. Goal based evaluation Tyler (1950)Tyler (1950)
Objectives used to aid planned changeObjectives used to aid planned change Can help clarify modelsCan help clarify models Goals from bench marking, theory or Goals from bench marking, theory or
pre-evaluation exercisespre-evaluation exercises Predict changesPredict changes Measure pre and post interventionMeasure pre and post intervention Identify the interventionsIdentify the interventions Were objectives achieved?Were objectives achieved?
1111
2.2 Difficulties with 2.2 Difficulties with Goal based evaluationGoal based evaluation
Who sets the goals? How do you identify Who sets the goals? How do you identify the intervention?the intervention?
– Tendency to managerialism (unitarist)Tendency to managerialism (unitarist)– Failure to accommodate value pluralismFailure to accommodate value pluralism– Over-commitment to scientific paradigmOver-commitment to scientific paradigm– What is measured gets doneWhat is measured gets done– No recognition of unanticipated effectsNo recognition of unanticipated effects– Focus on single outcome, not processFocus on single outcome, not process
1212
3.1 Realistic evaluation: 3.1 Realistic evaluation: Conceptual clarity Conceptual clarity (Pawson (Pawson & Tilley,1997)& Tilley,1997)
Evidence needs to be based on clear Evidence needs to be based on clear ideas about conceptsideas about concepts
Measures may be derived from theoryMeasures may be derived from theory Examine definitions used elsewhere Examine definitions used elsewhere Consider specific examples Consider specific examples Ensure all aspects are coveredEnsure all aspects are covered
1313
3.2 Realistic evaluation 3.2 Realistic evaluation Towards a theory: What Towards a theory: What are you looking for?are you looking for? Make assumptions and ideas explicit Make assumptions and ideas explicit
What is your theory of cause and effect?What is your theory of cause and effect?– What are you expecting to change What are you expecting to change
(outcome(outcome)? )? – How are you hoping to achieve this How are you hoping to achieve this
change (change (mechanismmechanism)?)?– What aspects of the What aspects of the contextcontext could be could be
important?important?
1414
3.3 Realistic evaluation 3.3 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome Context:Context: What environmental What environmental
aspects may affect the aspects may affect the outcome?outcome?– What else may influence the What else may influence the
outcomes?outcomes?– What other effects may there What other effects may there
be?be?
1515
3.4 Realistic evaluation 3.4 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome MechanismMechanism: What will you do to : What will you do to
bring about this outcome?bring about this outcome?– How will you intervene (if at all)?How will you intervene (if at all)?– What will you observe?What will you observe?– How would you expect groups to How would you expect groups to
differ?differ?– What mechanisms do you expect to What mechanisms do you expect to
operate?operate?
1616
3.5 Realistic evaluation 3.5 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome OutcomeOutcome: What effect or : What effect or
outcome do you aim for?outcome do you aim for?– What evidence could show it What evidence could show it
worked? worked? – How could you measure it? How could you measure it?
1717
4.1 Experimental 4.1 Experimental evaluation: evaluation:
Explain, predict and control by identifying causal Explain, predict and control by identifying causal relationshipsrelationships
Theory of causality makes predictions about Theory of causality makes predictions about variables eg training increases productivityvariables eg training increases productivity
Two randomly assigned matched groups: Two randomly assigned matched groups: experimental and controlexperimental and control
One group experiences intervention, one does One group experiences intervention, one does notnot
Measure outcome variable pre-test and post-Measure outcome variable pre-test and post-test (longitudinal)test (longitudinal)
Analyse for statistically significant differences Analyse for statistically significant differences between the two groupsbetween the two groups
Outcome linked back to modify theoryOutcome linked back to modify theory The gold standardThe gold standard
1818
4.2 Difficulties with 4.2 Difficulties with experimental evaluation experimental evaluation in organizationsin organizations Difficult to achieve in organizationsDifficult to achieve in organizations Unitarist viewUnitarist view Leaves out unforeseen effectsLeaves out unforeseen effects Problems with continuous change Problems with continuous change
processesprocesses Summative not formativeSummative not formative Generally at best quasi-Generally at best quasi-
experimentalexperimental
1919
5.1 Constructivist or 5.1 Constructivist or stakeholder evaluationstakeholder evaluation
Responsive evaluation Responsive evaluation (Stake, 1975)(Stake, 1975) or or Fourth generation evaluation Fourth generation evaluation (Guba & (Guba &
Lincoln, 1989)Lincoln, 1989) Constructivist interpretivist Constructivist interpretivist
hermeneutic methodologyhermeneutic methodology– Based on stakeholder claims concerns Based on stakeholder claims concerns
issuesissues– Stakeholders: agents, beneficiaries, Stakeholders: agents, beneficiaries,
victimsvictims
2020
5.2 Response to an IT 5.2 Response to an IT implementationimplementation(Brown, 1998)(Brown, 1998)
ThemeTheme The wardThe ward LaboratoLaboratoryry
IT TeamIT Team
GoalGoal Improve Improve quality to quality to patientspatients
Improve Improve quality for quality for ward staffward staff
Clinical Clinical and and financial financial benefitsbenefits
OutcomeOutcome Waste of Waste of time and time and energy on energy on a pointless a pointless systemsystem
No No improvemimprovement to ent to adequate adequate systemssystems
TechnicallTechnically y competent competent system - system - misconceimisconceived ved projectproject
2121
5.3 Constructivist 5.3 Constructivist evaluation issuesevaluation issues No one right answerNo one right answer Demonstrates complexity of Demonstrates complexity of
issues issues Highlights conflicts of interestsHighlights conflicts of interests Interesting for academics Interesting for academics Difficult for practitioners to Difficult for practitioners to
resolveresolve
2222
6 A Contingent approach to 6 A Contingent approach to evaluationevaluation(Legge, 1984)(Legge, 1984)
Do you want the proposed change Do you want the proposed change programme to be evaluated? programme to be evaluated? (Stakeholders)(Stakeholders)
What functions do you wish its What functions do you wish its evaluation to serve? evaluation to serve? (Stakeholders)(Stakeholders)
What are the alternative approaches to What are the alternative approaches to evaluation? evaluation? (Researcher)(Researcher)
Which of the alternatives best matches Which of the alternatives best matches the requirements?the requirements? (Discussion) (Discussion)
2323
7. Action research7. Action research
Identify good practiceIdentify good practice(Reason & Bradbury, 2001)(Reason & Bradbury, 2001) Action research Action research– Responds to practical issues in Responds to practical issues in
organizationsorganizations– Engages in collaborative relationshipsEngages in collaborative relationships– Draws on diverse evidenceDraws on diverse evidence– Value orientation - humanistValue orientation - humanist– Emergent, developmentalEmergent, developmental
2424
Problems with realist Problems with realist modelsmodels Tendency to managerialiseTendency to managerialise Over-commitment to scientific paradigmOver-commitment to scientific paradigm Context stripping, Context stripping, Over-dependence on measuresOver-dependence on measures Coerciveness: truth as non-negotiableCoerciveness: truth as non-negotiable Failure to accommodate value pluralismFailure to accommodate value pluralism Every act of evaluation is a political act, Every act of evaluation is a political act,
not tenable to claim it is value freenot tenable to claim it is value free
2525
Problems with Problems with Constructionist Constructionist approachapproach Evaluation judged by who for whom Evaluation judged by who for whom
and in whose interests?and in whose interests? Identify different views, then what?Identify different views, then what? Who has power?Who has power? Leaves decisions openLeaves decisions open May lead to ambiguityMay lead to ambiguity
2626
Why not evaluate?Why not evaluate?
Expensive in time and resourcesExpensive in time and resources De-motivating for individuals De-motivating for individuals Contradiction between “scientific” Contradiction between “scientific”
evaluation models and supportive, evaluation models and supportive, organization learning modelsorganization learning models
Individual identification with activityIndividual identification with activity Difficulties in objectifying and Difficulties in objectifying and
maintaining commitmentmaintaining commitment External evaluation ‘off the shelf’ External evaluation ‘off the shelf’
inappropriate and unhelpfulinappropriate and unhelpful
2727
Why evaluate?Why evaluate?(Legge, 1984)(Legge, 1984)
OvertOvert– Aids decision Aids decision
makingmaking– Reduce Reduce
uncertaintyuncertainty– Learn Learn – ControlControl
CovertCovert– Rally Rally
support/oppositionsupport/opposition– Postpone a Postpone a
decisiondecision– Evade Evade
responsibilityresponsibility– Fulfil grant Fulfil grant
requirementsrequirements– SurveillanceSurveillance
2828
ConclusionConclusion
Evaluation is very expensive, Evaluation is very expensive, demanding and complexdemanding and complex
Evaluation is a political process: need Evaluation is a political process: need for clarity about why you do itfor clarity about why you do it
Good evaluation always carries the risk Good evaluation always carries the risk of exposing failureof exposing failure
Therefore evaluation is an emotional Therefore evaluation is an emotional processprocess
Evaluation needs to be acceptable to Evaluation needs to be acceptable to the organizationthe organization
2929
Conclusion 2Conclusion 2
Plan and decide which model of Plan and decide which model of evaluation is appropriateevaluation is appropriate
Identify who will carry out the Identify who will carry out the evaluation and for what purposeevaluation and for what purpose
Do not overload the evaluation Do not overload the evaluation process:judgment or development?process:judgment or development?
Evaluation can give credibility and Evaluation can give credibility and enhance learningenhance learning
Informal evaluation will take place Informal evaluation will take place whether you plan it or notwhether you plan it or not