shcr review 2015 - appendix 3 impact measurement

1 | S H C R i m p a c t r e v i e w – A p p e n d i x 3

The School for Health and Care Radicals: what impact has it had?

Appendix 3: Measuring the impact of the SHCR

Ksenia Zheltoukhova, CIPD


Contents Contents .................................................................................................................................................. 2

Note ......................................................................................................................................................... 2

Introduction ............................................................................................................................................. 3

Methodology ....................................................................................................................................... 3

Measures ............................................................................................................................................. 7

Results ..................................................................................................................................................... 8

Refining the SHCR impact model ......................................................................................................... 8

Engagement with the School and feedback on the SHCR experience ............................................... 13

Self-reported outcomes ..................................................................................................................... 14

Change in individuals’ attitudes and behaviours ............................................................................... 15

Change in energy ............................................................................................................................... 17

Change in organisation-level outcomes ............................................................................................. 19

Inter-group comparisons ................................................................................................................... 22

References ......................................................................................................................................... 37

Note This appendix forms part of the review of the 2015 School for Health and Care Radicals run by NHS Improving Quality. The review was conducted by the research arm of the CIPD between November 2014 and September 2015. The main report, which presents an overview of the findings, is also available.


Introduction The launch of the School for Health and Care Radicals (SHCR) in 2014 was met with strong enthusiasm by the learners within the NHS and abroad, with a number noting positive changes in their attitudes to change, and ability to lead radical change in their organisations. Although students’ reflections provided some evidence on the value added by SHCR, the impact of the School was not measured systematically in its first iteration. So, one of the aims of evaluating SHCR in 2015 was to quantify a range of outcomes at individual, community, and organisational level, that result from learners’ participation in the School.

To measure the impact of the School the CIPD conducted two surveys – one immediately before the School started and one 6 months later. This approach aimed to:

• Assist in refining the model of SHCR impact (see Appendix 2). • Collect feedback on learners’ experience of participating in the School. • Compare individual-level outcomes before and after learners took part in the School. • Compare organisation-level outcomes before and after learners took part in the

School. • Appreciate the link between change in individuals’ behaviours and attitudes and

change in organisation-level outcomes.

This document outlines the methodology of measuring the impact of SHCR 2015 and presents the findings of the two surveys.

Methodology As the main purpose of this research was to evaluate the change in individual- and organisation-level outcomes as a result of taking part in the School of Health Care Radicals (SHCR), the evaluation survey relied on a “before-and-after” design. This approach (also referred to as quasi-experimental design) is a common technique for understanding the effect of an intervention, such as taking part in a course (Rossi et al., 2004).

Gathering survey data before students started the School helped establish a baseline of their attitudes and behaviours, as well as their experiences of leading change in their organisation. The second wave of the survey was conducted 6 months later to allow individuals time to develop their behaviours and implement change in their organisations. The responses to this wave then gave us an understanding of the incremental change since the start of the School.

One of the challenges of the “before-and-after” design is in distinguishing the impact associated with the intervention itself from any natural changes that can happen in individuals’ behaviours and attitudes, as well as in organisational environments over time. To differentiate between the two, a parallel “control” group is sometimes formed (Rossi et al, 2004). Members of the control group do not take part in the intervention but their behaviours are measured for between-group comparisons. However, drawing a control group for this evaluation was difficult, as individuals only contacted the School with the purpose of taking part, so the population of “non-learners” was not accessible. Even where individuals did not take part in the School, despite signing up for it, they were no longer engaged with SHCR enough to be encouraged to take part in the evaluation.

Although the “before-and-after” measures were the core focus of the impact evaluation, the second wave of the survey also assisted in collecting some after-only measures, for example, feedback on the learning experience. Similarly, the first wave of the survey was used to refine the evaluation model. Table 1 below summarises the purpose of the two waves of the survey.


Table 1. SHCR evaluation objectives and survey waves

Objective Survey wave Assist in refining the model of SHCR impact (see Appendix 2). Wave 1 Collect feedback on learners’ experience of participating in the School. Wave 2 Compare individual-level outcomes before and after learners took part in the School.

Waves 1 and 2

Compare organisation-level outcomes before and after learners took part in the School.

Waves 1 and 2

Appreciate the link between change in individuals’ behaviours and attitudes and change in organisation-level outcomes.

Waves 1 and 2

Sample and data collection The survey population came from the NHS IQ database of all participants who had signed up to the SHCR by 27 January 2015. The first wave survey was a census of these participants – in other words, attempting to survey them all instead of selecting a probability sample (Harding 2006). All participants were invited to complete the first wave of the survey via an email, containing information about the survey and a link to the online questionnaire. In total 1,232 individuals were invited. A reminder requesting to complete the survey was sent within the following week to all non-respondents.

The first wave of the survey gathered 639 valid responses, with the response rate of 51.9%. Of the respondents 71.7% were female and 26.3% were male. The average age was 42.81 (see Figure 1). This sample profile is typical of the healthcare workforce (see, for example, HSCIC, 2014).

Figure 1. Age distribution of Wave 1 survey respondents.

More than half of the respondents worked for a public health care provider (52.9%) – this was also the largest group of the respondents (see Table 2). 58.5% of respondents worked for the NHS.

3.5

18.0

33.9 32.6

12.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Up to 24 25-34 35-44 45-54 55 and over


Table 2. Type of organisation respondents worked for.

Type of organisation %

Public health care provider 52.9

Public Health/Healthcare improvement 6.9

Community care provider 6.1

Other public sector organisation 5.3

Care commissioning 5.2

Health care education and training 4.5

Consultancy (e.g. change management consultancy)

2.7

Private health care provider 1.7

Health care research 1.4

Patient advocacy 1.3

Other private sector organisation 1.3

Other 10.0

Don’t know .8

The second wave survey sample was drawn from the list of individuals who were registered with SHCR 2015 on 15 June 2015. This list could include individuals who were signed up to take part in the School before it started, as well as those who joined the School later. Once again the invitation to participate in the survey was sent via an email with the link to the questionnaire. Two reminders were sent one week and three weeks after the original email. However, as some of the potential survey participants did not complete the first wave of the survey, two different invites were sent:

1. A survey repeating questions from wave 1, as well as questions about the learning experience and self-reported outcomes – sent to those who took part in Wave 1.

2. A survey only containing questions about the learning experience and self-reported outcomes – sent to those who did not take part in Wave 1.

The first invite for Wave 2 was sent to the 639 individuals who took part in Wave 1 survey, returning 113 responses, which represents a response rate of 17.7%. The data from both survey waves was matched using unique identifying numbers assigned to the respondents by the researchers prior to surveying, and communicated via the invite emails. Only the matched responses were analysed when comparing changes in individual- and organisation-level outcomes, resulting from the School.

Of the respondents who completed both waves of the survey 72.1% were female and 25.6% were male. The average age was 43.81 (see Figure 2). 58.9% of respondents worked in the NHS.


Figure 2. Age distribution of “before-and-after” survey respondents.

The second invite was sent to 1,951 individuals who did not take part in Wave 1 survey, returning 65 valid responses, which represents a response rate of 3.3%. These responses were combined with the 113 of responses from the “before-and-after” sample for analysis of the learning experience and self-reported outcomes of the School. The total sample for this “after-only” analysis, therefore, was 178.

Of the respondents who completed both waves of the survey 70.9% were female and 27.8% were male. The average age was 44.37 (see Figure 3).

The respondents were invited to complete the survey on voluntary basis, and those who completed both waves of the survey were entered in a draw for one of eight non-cash prizes provided by NHS IQ.

Figure 4 below illustrates the types of questions and the samples used in the two waves of the survey.

Note on methods These methods complement the other methods used in this review, including case study research (see Appendix 5). Overall, we collected different types of data that, by the principle of triangulation, help us build a more complete picture of the impact of the SHCR. Thus, the before and after survey gives us our most rigorous assessment of the school’s impact across the board; yet while this focuses at the level of individuals’ capability as change agents, the in-depth case studies go further to investigate the types of impact the school can have at the level of how organisations function and the quality of healthcare services.

The evaluation survey emphasises the impact on individuals as change agents because this is the easiest level at which to obtain convincing evidence. By contrast, assessment of organisational impact of learning is complicated. Firstly, because the SHCR focuses on helping people lead change, the potential organisational outcomes are hugely diverse. Secondly, it is very difficult to attribute any change in the organisation outcomes to the experiences of participants in the school.Figure 3. Age distribution in the total Wave 2 survey sample.

1.8

18.2

33.6 31.8

14.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Up to 24 25-34 35-44 45-54 55 and over


Figure 4. Questions used in survey Waves 1 and 2

Measures Engagement with the School was assesses with a single question, where participants could select activities that they intended to take part in during the School (Wave 1), or the ones they did take part in (Wave 2).

1.2

18.7

31.6 31.0

17.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

Up to 24 25-34 35-44 45-54 55 and over

Individual behaviours and attitudes

Energy

Organisation-level outcomes

Individual behaviours and attitudes

Energy

Organisation-level outcomes

Self-reported impact

Feedback on the School

Group 1

Group 2

Wave 1, N= 639 Wave 2, N = 113 (before-and-after)

Wave 2, N = 178 (after-only)


Feedback on the learning experience was measured with 11 questions, asking about participants satisfaction with different aspects of the School (such as quality of resources, communications and others), as well as with each of the five modules delivered in the course of the School.

Individual behaviours and attitudes towards leading change were measured with 25 statements, using a 7-point agree/disagree Likert scale. Example of statements include “I am knowledgeable about models of organisational change” and “If I can see how to make a positive change, I will do it without waiting for permission”.

Energy was measured with 16 statements developed by SHCR, using a 7-point agree/disagree Likert scale. Three components of energy were measured: spiritual energy (5 items, α=.809), psychological energy (6 items, α=.739), and social energy (5 items, α=.847). Example of statements include “I feel depleted of energy when others express doubt about change in my organisation” and “I feel safe enough to do things differently”.

Self-reported outcomes were measured with two sets of questions, assessed by respondents retrospectively. One question asked the respondents about the different actions taken as a result of the School (for example, whether they “shared learning with others in your organisation”). Another set of questions asked the participants to reflect on the extent to which the School improved a range of behaviours and attitudes (such as Openness to other people’s points of view on change, Ability to challenge, and others).

Organisational outcomes were measured with 6 statements, using a 7-point agree/disagree Likert scale. Example of statements included “In my organisation it is acceptable to challenge the status quo” and “I respond to change in a way which increases organisation performance”.

Alongside other background questions (age, gender etc) level of activity on social media networks (Twitter, LinkedIn, blogs) was measured with three multiple-choice questions: “How often do you contribute original posts/content?”, “How often do you read posts/content contributed by others?”, and “How often do you connect/have conversations with other members?”.

Several free-text questions were included. In Wave 1 survey respondents were asked to record their expectations of the School in a free text format, as well as select the activities they intend to take part in as part of the School from the list of activities provided. In Wave 2 survey individuals were also able to submit any comments on their experience of SHCR.

Results This section presents the results of each stage of the data analysis. For the discussion of the findings see the main evaluation report.

Refining the SHCR impact model The first step of the analysis was to assess the strength of the SHCR evaluation model (see Appendix 2), and to reduce the 25 items, measuring individual behaviours and attitudes towards change, into a set of individual-level outcomes enabled by participation in the School.

Principal components analysis was applied on the original 25 items to identify any emerging components of the School’s impact model. KMO measure and Bartlett’s test of sphericity, as well as exploration of between-item correlations indicated that the data was appropriate for factor reduction. Components were first extracted, based on eigenvalues 1 and above, and a rotation applied to increase factor loadings on items. Both Varimax rotation with Kaiser normalisation and Direct oblimin


rotation models were attempted. The components were not strongly correlated with each other, so a model resulting from Varimax rotation was chosen to represent factor loadings (see Table 3).

Table 3. Factor loadings

Rotated Component Matrix

Component

1 2 3 4 5

I make my work visible in ways that help others. .755

I collect, organise and share useful resources. .728

I build support for change at work. .583

I inspire others to a shared purpose. .535

I help others find solutions to challenges in leading change. .521 .479

I get people interested in change initiatives. .486

I use storytelling to inspire change. .462

I am an effective change agent. .434 .420 .418

I am very aware of my natural leadership style. .378

If I can see how to make a positive change, I will do it without waiting for permission.

.796

I encourage others to lead change without waiting for permission. .792

I mobilise others to act on change. .548 .562

I challenge traditional ways of working when I think they can be improved.

.358 .510 .460

I lead change informally, by connecting with and influencing people.

.322 .446

I receive encouragement or support from like-minded people trying to lead change.

.362 .364

I know when to use planned or emergent approaches to change. .870

I am knowledgeable about models of organisational change. .834

I am naturally motivated to improve how we work. .782

I feel a sense of purpose in my day-to-day work. .626

I reflect on my work experience to help improve practice. .428 .556

I make suggestions on new ways of thinking or working. .441 .510

I clash with colleagues about how we should work. .744


I prefer to lead change by myself rather than collaborating with others.

.655

I tend to be careless in how my behaviour or words affect others. .615

I am careful to frame what I want to say so that it has a positive impact.

.407 .357 .434

Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 9 iterations.

Two items were removed from the model at this stage: “I receive encouragement or support from like-minded people trying to lead change” demonstrated poor factor loadings, and “I am an effective change agent” had similar but relatively low loadings across three components. It is likely that the first statement represented a construct different from individual behaviours and attitudes towards change, while the second statement was too general to be definitively associated with a single component. Factor loadings on other items were interpreted in the context of the School’s objectives to form five components (see Table 4).

Table 5. Five individual-level outcomes of the School

Factor Items α ICC(1) Ability to connect with others to build support for change

I make my work visible in ways that help others. I collect, organise and share useful resources. I build support for change at work. I inspire others to a shared purpose. I help others find solutions to challenges in leading change. I get people interested in change initiatives. I use storytelling to inspire change. I am very aware of my natural leadership style.

.851 .401

Rocking the boat If I can see how to make a positive change, I will do it without waiting for permission. I encourage others to lead change without waiting for permission. I mobilise others to act on change. I challenge traditional ways of working when I think they can be improved. I lead change informally, by connecting with and influencing people.

.780 .352

Knowledge about change

I know when to use planned or emergent approaches to change. I am knowledgeable about models of organisational change.

.843 .631

Sense of purpose and motivation

I am naturally motivated to improve how we work. I feel a sense of purpose in my day-to-day work. I reflect on my work experience to help improve practice. I make suggestions on new ways of thinking or

.647 .287


working. Staying in the boat I clash with colleagues about how we should

work. I prefer to lead change by myself rather than collaborating with others. I tend to be careless in how my behaviour or words affect others. I am careful to frame what I want to say so that it has a positive impact.

.595 .197

Table 6. Confirmatory factor analysis of the SHCR impact model

Chi sq. (213) = 760.413***, RMSEA = 0.066, CFI=0.900, TLI=0.881, CD=0.998

Factor Items Ability to connect with others to build support for change

I make my work visible in ways that help others (0.588) I collect, organise and share useful resources (0.515) I build support for change at work (0.713) I inspire others to a shared purpose (0.694) I help others find solutions to challenges in leading change (0.769). I get people interested in change initiatives (0.719). I use storytelling to inspire change (0.564). I am very aware of my natural leadership style (0.611).

Rocking the boat

If I can see how to make a positive change, I will do it without waiting for permission (0.380). I encourage others to lead change without waiting for permission (0.632). I mobilise others to act on change (0.857). I challenge traditional ways of working when I think they can be improved (0.658). I lead change informally, by connecting with and influencing people (0.501).


I know when to use planned or emergent approaches to change (0.822). I am knowledgeable about models of organisational change (0.888).


I am naturally motivated to improve how we work (0.482). I feel a sense of purpose in my day-to-day work (0.382). I reflect on my work experience to help improve practice (0.515). I make suggestions on new ways of thinking or working (0.698).

Staying in the boat

I clash with colleagues about how we should work (0.269). I prefer to lead change by myself rather than collaborating with others (0.208). I tend to be careless in how my behaviour or words affect others (0.356). I am careful to frame what I want to say so that it has a positive impact (0.923).

Item covariances

I make my work visible in ways that help others / I collect, organise and share useful resources (0.315) I build support for change at work / I make my work visible in ways that help others (0.226) If I can see how to make a positive change, I will do it without waiting for permission / I challenge traditional ways of working when I think they can be improved (0.221) If I can see how to make a positive change, I will do it without waiting for permission / I encourage others to lead change without waiting for permission (0.531) I challenge traditional ways of working when I think they can be improved / I make suggestions on new ways of thinking or working (0.410) I am naturally motivated to improve how we work/ I feel a sense of purpose in my day-to-day work (0.306)


I prefer to lead change by myself rather than collaborating with others / I clash with colleagues about how we should work (0.265)

Factor covariances

Ability to connect with others/Rocking the boat (0.963) Ability to connect with others/Knowledge about change (0.634) Ability to connect with others/Sense of purpose and motivation (0.896) Ability to connect with others/Staying in the boat (0.664) Rocking the boat/Knowledge about change (0.507) Rocking the boat/Sense of purpose and motivation (0.840) Rocking the boat/Staying in the boat (0.506) Knowledge about change/Sense of purpose and motivation (0.520) Knowledge about change/Staying in the boat (0.412) Sense of purpose and motivation/Staying in the boat (0.468)


Engagement with the School and feedback on the SHCR experience

This section reports on the intended and actual levels of engagement with the School, as well as feedback on various aspects of the School experience.

Table 7. Level of intended engagement with the School before the start of the course (N=639)

Activity % intending to take part in activity Download the materials 92.3% Participate in the online modules 90.8% Connect with other participants 74.6% Participate in other opportunities offered by the School

66.5%

Table 8. Actual levels of engagement with the School (post-SHCR)

Engagement level – actual (N= 178)

% of those who planned to do activity before the start of the School (N=112)

Downloaded the materials

81.5 83.8% of those who said they would download the materials

Listened to the online modules LIVE and/or RECORDINGS

78.7 95.2% of those who said they would participate in the online modules

Listened to the online modules LIVE


Listened to the RECORDINGS of the online modules


Took part in the online chats during the modules

51.7 60.7% of those who said they would connect with other participants

Connected with other participants via social media

48.9 61.9% of those who said they would connect with other participants

Other 6.2

Table 9. Actions taken as a result of the School

Action % Shared learning with others in your organisation 75.8 Shared learning outside of your organisation (e.g. via social media) 43.8 Recommended the School to others 82.6 Sought other types of training for change management and leadership

40.4

Became a certified change agent 44.9 None of these 5.6


Table 10. Satisfaction with the online modules

N Didn’t access this module

Dissatisfied or very dissatisfied

Neither satisfied nor dissatisfied

Satisfied or very satisfied

Module 1. Being a health & care radical: change starts with me 175 1.1 1.1 5.1 92.6

Module 2. Building alliances for change 174 4.6 1.1 4.6 89.7

Module 3. Rolling with resistance 170 5.3 1.2 4.1 89.4 Module 4. Making change happen 170 8.8 0.0 4.7 86.5 Module 5. Moving beyond the edge 169 13.6 0 5.3 81.1

Table 11. Satisfaction with different aspects of the SHCR experience

N Dissatisfied or very dissatisfied

Neither satisfied nor dissatisfied

Satisfied or very satisfied

Quality of additional resources (e.g. study guides) 171 0.6 4.1 95.3 Quality of community activities (e.g. live chats, Twitter, Facebook) 155 2.6 16.1 81.3

Technical aspects of the webinar platform 167 4.8 17.5 77.7 Timing of the modules (i.e. Friday morning UK time) 173 16.2 22.2 61.7

The way the School was run 128 0.0 4.0 96.0 The certification process 171 6.3 14.1 79.7

Self-reported outcomes In the second wave of the survey the participants were asked to retrospectively assess whether they had experienced change in any individual-level outcomes as a result of the School. Table 11 below presents participants’ responses against ten types of such outcomes.

Table 12. As a result of the School, to what extent has the following increased?

Not at all To little extent

To some extent

To a great extent

Your sense of purpose and motivation to improve practice 4.5 6.2 37.6 50.6

Your knowledge about organisational change 3.4 12.4 48.3 34.3 Your ability to challenge 6.2 19.1 48.9 23.6 Your ability to inspire and build support for change 6.2 19.1 48.9 23.6

Your ability to maintain relationships and keep people onside 4.5 10.7 51.7 29.8 Your perseverance and resilience 3.9 21.3 41.6 30.3 Your sense of being supported by others 5.1 17.4 38.2 37.1 How often you seek other people’s views on change or the need for change 8.4 16.3 43.8 28.7


Your openness to other people’s points of view on change 5.6 20.8 45.5 24.7

Your confidence talking about the need for change 5.1 15.2 36.0 41.0

Change in individuals’ attitudes and behaviours Individual-level outcomes were measured in both survey waves. This section presents the results by individual item, as well as by component, identified in the first phase of the analysis.

Table 13. Individual behaviours and attitudes, before and after taking part in the School, % of respondents agreeing/disagreeing

Statement Wave 1 (N=638) Wave 2 (N=178)

% agreeing or strongly agreeing

% neither agreeing nor disagreeing

% disagreeing or strongly disagreeing




I am naturally motivated to improve how we work.

98.4 .5 .9 100.0 0.0 0.0

I feel a sense of purpose in my day-to-day work.

91.5 4.7 3.3 93.8 2.8 3.4

I am knowledgeable about models of organisational change.

64.6 16.0 18.6 92.7 6.2 1.1

I know when to use planned or emergent approaches to change.

41.2 22.1 34.0 81.9 13.0 5.1

I reflect on my work experience to help improve practice.

97.0 2.0 .9 98.3 .6 1.1

I am very aware of my natural leadership style.

85.8 8.0 5.6 93.2 5.6 1.1

I tend to be careless in how my behaviour or words affect others.

9.4 7.8 82.5 6.8 5.6 87.6

I lead change informally, by connecting with and influencing people.

87.3 8.3 3.9 92.1 5.1 2.8

I prefer to lead change by myself rather than collaborating with others.

13.3 14.9 71.4 12.6 12.0 75.4


If I can see how to make a positive change, I will do it without waiting for permission.

64.5 13.5 21.6 73.3 11.4 15.3

I challenge traditional ways of working when I think they can be improved.

91.5 5.2 3.3 96.6 2.8 .6

I get people interested in change initiatives.

84.7 12.5 2.7 94.9 4.5 .6

I make suggestions on new ways of thinking or working.

95.9 3.1 .9 98.9 .6 .6

I clash with colleagues about how we should work.

32.7 23.3 43.7 29.5 21.6 48.9

I inspire others to a shared purpose.

79.8 15.3 3.3 89.7 7.4 2.9

I use storytelling to inspire change. 68.5 16.4 13.6

83.5 10.2 6.3

I am careful to frame what I want to say so that it has a positive impact.

81.5 13.6 4.7 94.3 2.8 2.8

I build support for change at work.

81.5 13.8 3.3 94.3 5.1 .6

I make my work visible in ways that help others.

81.7 12.2 5.0 93.2 5.1 1.7

I collect, organise and share useful resources.

89.0 8.3 2.5 96.1 2.2 1.7

I encourage others to lead change without waiting for permission.

64.5 17.7 16.6 75.0 17.0 8.0

I help others find solutions to challenges in leading change.

84.8 11.3 3.1 92.7 5.6 1.7

I mobilise others to act on change. 80.9 13.8 4.4

90.4 8.5 1.1

I receive encouragement or support from like-minded people trying to lead change.

77.0 11.6 10.8 83.1 5.6 11.3

I am an effective change agent. 62.1 26.4 9.9 84.6 13.1 2.3


Table 14 below provides the results of the statistical comparison in the levels of individual outcomes before and after the School. Wilcoxon signed ranks test for paired samples showed significant differences in ‘before’ and ‘after’ levels for ability to connect with others to build support for change, rocking the boat, sense of purpose and motivation, knowledge about change, and responses to the statement “I am an effective change agent”. However, there were no significant differences for staying in the boat and responses to the statement “I receive encouragement of support from like-minded people trying to lead change”.

Table 14. Change in individual-level, by component, average scores (matched sample)

Outcome Before (N=113) After (N=112) Wilcoxon signed ranks test

Effect size

Mean SD Mean SD Z r=Z/√N Ability to connect with others to build support for change

5.40 .67 5.83 .62 -6.33*** 0.60

Rocking the boat 5.15 .90 5.58 .75 -5.41*** 0.51 Staying in the boat 5.09 .77 5.17 .75 -1.28 0.12 Sense of purpose and motivation 6.20 .54 6.31 .54 -1.99* 0.19 Knowledge about change 4.53 1.25 5.59 .77 -7.13*** 0.67 I receive encouragement of support from like-minded people trying to lead change

5.24 1.25 5.38 1.31 -1.521 0.14

I am an effective change agent 4.79 1.21 5.54 .92 -6.004*** 0.57 *p<0.05; **p<0.01; ***p<0.001

Change in energy Spiritual, social and psychological energy were measured in both survey waves. This section presents the results by individual item, as well as by energy scale.

Table 15. Energy, before and after taking part in the School, % of respondents agreeing/disagreeing








Social energy

I feel a sense of solidarity with those around me.

72.9 14.7 11.9 75.3 13.5 11.2

I don’t feel appreciated by others at work.

23.9 15.5 59.6 25.6 10.2 64.2

I feel isolated from others. 20.0 13.3 66.5 22.0 13.6 64.4

I feel disconnected from others.

20.2 13.0 66.7 20.8 10.1 69.1


I feel part of a group engaged in a change.

72.9 14.7 11.9 73.4 9.6 16.9

Psychological energy

I feel safe enough to do things differently.

80.9 13.8 4.4 78.1 10.7 11.2

I will be blamed if I try something new and it fails.

35.5 17.7 45.2 28.0 15.4 56.6

I sense openness about the potential to change.

57.7 19.2 22.2 68.2 14.2 17.6

I am able to keep expressing hope for change in my organisation when presented with setbacks.

80.3 10.5 6.9 92.4 4.7 2.9

I feel depleted of energy when others express doubt about change in my organisation.

45.9 16.9 35.8 44.8 14.9 40.2

I feel fearful about change in my organisation.

12.8 9.9 76.2 10.9 9.1 80.0

Spiritual energy

I am committed to our common vision for the future.

80.0 13.8 3.6 87.9 8.6 3.4

I am driven by shared values. 85.6 11.0 3.0 91.4 5.1 3.4

I am not driven by a shared purpose for change.

5.9 17.2 74.6 6.3 12.6 81.0

Current change in my organisation does not fit with my sense of purpose.

25.7 17.5 53.5 19.8 16.9 63.4

I feel current change in my organisation may conflict with my values.

24.6 15.8 57.0 20.5 15.8 63.7

Table 16 below provides the results of the statistical comparison in the levels of energy before and after the School. Wilcoxon signed ranks test for paired samples showed significant differences in ‘before’ and ‘after’ levels for social, psychological and spiritual energy.


Table 16. Change in energy, by scale, average scores (matched sample)

Scale Before (N=113) After (N=112) Wilcoxon signed ranks test

Effect size

Mean SD Mean SD Z r=Z/√N Social energy 4.80 1.18 5.04 1.18 -3.09** 0.29 Psychological energy 4.67 .94 5.10 .89 -5.04*** 0.48 Spiritual energy 5.07 .97 5.39 1.04 -3.20** 0.30 *p<0.05; **p<0.01; ***p<0.001

Change in organisation-level outcomes Organisation-level outcomes and organisational context were measured in both survey waves. This section presents the results by individual item, as well as correlations between change in organisation-level outcomes and change in individual-level outcomes.

Table 17. Organisation-level outcomes, before and after taking part in the School, % of respondents agreeing/disagreeing








I respond to change in a way which increases organisation performance.

93.0 4.4 1.1 97.7 1.1 1.1

I would recommend care at my place of work to friends and family.

62.6 12.2 7.2 83.5 11.5 5.0

My organisation regularly reviews its performance to identify areas for improvement and change.

67.9 12.4 16.6 74.4 8.9 16.7

In my organisation it is acceptable to challenge the status quo.

56.0 14.7 26.8 65.1 16.9 18

My organisation is resistant to change.

37.9 13.8 45.9 49.4 18.8 31.8

My organisation is good at building employee commitment to its change programmes.

38.5 19.2 38.2 33.5 28.8 37.6


Table 18 below provides the results of the statistical comparison in the levels of energy before and after the School. Wilcoxon signed ranks test for paired samples showed significant differences in ‘before’ and ‘after’ levels for four types of organisation-level outcomes highlighted in the table.

Table 18. Change in organisation-level outcomes, average scores (matched sample)

Statement Before (N=113)

After (N=112) Wilcoxon signed ranks test

Effect size

N Mean SD N Mean SD Z r=Z/√N I respond to change in a way which increases organisation performance.

101 5.96 .71 103 6.05 .73 -8.50*** 0.85

I would recommend care at my place of work to friends and family.

83 5.48 1.36 78 5.81 1.21 -6.02*** 0.68

My organisation regularly reviews its performance to identify areas for improvement and change.

99 5.02 1.42 97 5.26 1.57 -6.09*** 0.62


98 4.44 1.57 100 4.86 1.54 -3.55*** 0.36


98 4.20 1.46 98 4.33 1.52 -0.53 0.05


97 3.97 1.40 98 4.08 1.29 -0.42 0.04

*p<0.05; **p<0.01; ***p<0.001

Table 19 below reports Pearson r for the different pairs of individual-level and organisation-level outcomes. The change in the outcome level (the score reported after the School minus the score reported before the School) is used.

Table 19. Correlations between changes in organisation-level outcomes, and changes in individual-level outcomes (matched sample)

Key: Org1: I respond to change in a way which increases organisation performance. Org2: I would recommend care at my place of work to friends and family. Org3: My organisation regularly reviews its performance to identify areas for improvement and change. Org4: In my organisation it is acceptable to challenge the status quo. Org5: My organisation is resistant to change. Org6: My organisation is good at building employee commitment to its change programmes.

Org1 Org2 Org3 Org4 Org5 Org6

Ability to connect with others to build support for change

Pearson Correlation .435** -.133 -.004 .132 -.027 .185

Sig. (2-tailed) .000 .255 .968 .199 .797 .075

N 101 75 96 97 95 94

Rocking the boat Pearson Correlation .317** -.026 .014 .055 .027 .189

Sig. (2-tailed) .001 .826 .895 .591 .794 .068


N 101 75 96 97 95 94

Staying in the boat

Pearson Correlation .096 -.006 .088 .026 .263** .014

Sig. (2-tailed) .338 .958 .392 .802 .010 .895

N 101 75 96 97 95 94


Pearson Correlation .342** -.049 .161 .098 .104 .127

Sig. (2-tailed) .000 .677 .117 .338 .317 .224

N 101 75 96 97 95 94


Pearson Correlation .152 -.116 .208* .167 .044 .058

Sig. (2-tailed) .130 .321 .042 .103 .669 .576

N 101 75 96 97 95 94

Social energy

Pearson Correlation .106 -.027 .079 .216* .169 .139

Sig. (2-tailed) .291 .816 .447 .034 .102 .183

N 101 75 96 97 95 94

Psychological energy

Pearson Correlation .184 -.110 .170 .145 .274** .158

Sig. (2-tailed) .066 .346 .099 .155 .007 .128

N 101 75 96 97 95 94

Spiritual energy

Pearson Correlation .335** -.302** -.028 .094 .136 .050

Sig. (2-tailed) .001 .008 .784 .360 .188 .629

N 101 75 96 97 95 94

I am an effective change agent

Pearson Correlation .262** .001 .093 .115 .040 .255*

Sig. (2-tailed) .009 .994 .371 .268 .699 .014

N 98 75 94 94 94 93

I receive encouragement or support from like-minded people trying to lead change

Pearson Correlation .221* -.221 .097 .221* .041 .138

Sig. (2-tailed) .026 .057 .345 .030 .693 .186

N 101 75 96 97 95 94

*p<0.05; **p<0.01; ***p<0.001


Inter-group comparisons This section reports the results of inter-group comparison between the changes in individual-level and organisation-level outcomes before and after the School. Because of the non-normal distribution of variables in the sample non-parametric tests were applied. For comparison of 2 groups this was Wilcoxon Mann-Whitney test, and for 3 and more groups - Kruskal-Wallis Test, with a follow-up Games-Howell post-hoc test appropriate for small groups with unequal variances.

In the tables below significant differences in outcomes are highlighted in bold, and further details on the average outcomes for each of the comparison groups are provided.

Age Table 20. Age groups

Age group % Up to 34 20.0% 35-44 33.6% 45+ 46.4% Total 100.0

Table 21. Kruskal-Wallis test for individual- and organisation-level outcomes by age group

Chi-Square

df Asymp. Sig.

Ability to connect with others to build support for change 1.918 2 .383

Rocking the boat 6.354 2 .042

Staying in the boat .126 2 .939

Sense of purpose and motivation 4.337 2 .114

Knowledge about change 3.070 2 .215

Social energy .619 2 .734

Psychological energy .696 2 .706

Spiritual energy .640 2 .726

I respond to change in a way which increases organisation performance. .904 2 .636

I would recommend care at my place of work to friends and family. .538 2 .764

My organisation regularly reviews its performance to identify areas for improvement and change. 1.030 2 .598

In my organisation it is acceptable to challenge the status quo. .660 2 .719

My organisation is resistant to change. .269 2 .874


My organisation is good at building employee commitment to its change programmes. .585 2 .746

Results of post-hoc test and means reported in Tables 22 and 23 below show that the significant differences in “Rocking the boat” outcome are between groups of respondents aged 35-44 and 45 and over, with the latter group experiencing least change in “Rocking the boat” since the start of the School.

Table 22. Post-hoc test for “Rocking the boat” by age group

Post-hoc test for Rocking the boat (I) Age group 1

(J) Age group 2

Mean Difference (I-J)

Std. Error Sig. 95% Confidence Interval Lower Bound Upper Bound

Up to 34 35-44 .00393 .25642 1.000 -.6228 .6307 45+ .41381 .23502 .202 -.1688 .9964

35-44 Up to 34 -.00393 .25642 1.000 -.6307 .6228 45+ .40988* .15537 .028 .0369 .7828

45+ Up to 34 -.41381 .23502 .202 -.9964 .1688 35-44 -.40988* .15537 .028 -.7828 -.0369

Table 23. Descriptive statistics for change in “Rocking the boat” levels by age group

Rocking the boat Age group Change in score (After-Before) N Std. Deviation Up to 34 .6364 22 1.03212

35-44 .6324 37 .80071 45+ .2225 51 .58935

Gender Table 24. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by gender

Mann-Whitney U

Wilcoxon W

Z Asymp. Sig. (2-tailed)

Ability to connect with others to build support for change 613.000 2566.000 -.702 .483

Rocking the boat 673.500 926.500 -.087 .931

Staying in the boat 617.000 2570.000 -.665 .506

Sense of purpose and motivation 650.000 2603.000 -.329 .742

Knowledge about change 642.000 895.000 -.412 .680

Social energy 583.000 2536.000 -1.012 .312


Psychological energy 674.000 927.000 -.082 .935

Spiritual energy 631.000 2584.000 -.521 .602

I respond to change in a way which increases organisation performance. 426.000 657.000 -2.440 .015

I would recommend care at my place of work to friends and family. 361.500 1586.500 -.547 .585

My organisation regularly reviews its performance to identify areas for improvement and change. 575.000 806.000 -.617 .537

In my organisation it is acceptable to challenge the status quo. 621.000 874.000 -.307 .759

My organisation is resistant to change. 514.500 2284.500 -1.177 .239

My organisation is good at building employee commitment to its change programmes. 515.500 2345.500 -.979 .328

There was a significant difference between males and females’ responses to the statement “I respond to change in a way which increases organisation performance” before and after the School. The average scores reported by males have went down since the start of the School, while the scores submitted by females went up.

Table 23. Descriptive statistics for change in “I respond to change in a way which increases organisation performance” levels by gender

I respond to change in a way which increases organisation performance. Gender Change in score (After-

Before) N Std. Deviation

Female .1639 61 .87902 Male -.3333 21 .65828

Working for the NHS No significant differences in individual- and organisation-level outcomes were found between those working for the NHS, and those working for other organisations.

Table 24. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by employment in the NHS

Mann-Whitney U

Wilcoxon W


Ability to connect with others to build support for change 1208.000 2289.000 -1.834 .067

Rocking the boat 1489.500 2570.500 -.169 .866





Social energy 1416.500 3627.500 -.602 .547

Psychological energy 1332.000 2413.000 -1.102 .270


I respond to change in a way which increases organisation performance. 1445.000 3590.000 -.113 .910






Participation in the School as a team Only 17 out of 113 respondents (15%) participated in SHCR as a team, which is insufficient for meaningful comparisons of the outcomes experienced by the respondents in this group with the outcomes reported by those who participated on their own.

CEO distance The number of respondents in each group (by level from CEO) was too small for inter-group comparisons (under 20).

Time working in the current organisation No significant differences in individual- and organisation-level outcomes were found. Table 25. Time spent working in the current organisation Time % Up to 2 years 30.2% 2-5 years 22.6% 5-10 years 22.6% 10+ years 24.5%


Table 26. Kruskal-Wallis test for individual- and organisation-level outcomes by time working in the current organisation

(Only those who stayed in the same organisation through both survey waves)

Chi-Square

df Asymp. Sig.

Ability to connect with others to build support for change 5.480 3 .140


Staying in the boat 7.189 3 .066

Sense of purpose and motivation 4.324 3 .229


Social energy 2.263 3 .520


Spiritual energy .719 3 .869

I respond to change in a way which increases organisation performance. 2.248 3 .522

I would recommend care at my place of work to friends and family. 2.048 3 .563

My organisation regularly reviews its performance to identify areas for improvement and change. .672 3 .880

In my organisation it is acceptable to challenge the status quo. 3.519 3 .318

My organisation is resistant to change. 2.664 3 .446

My organisation is good at building employee commitment to its change programmes. 1.665 3 .645

Time working in the current role No significant differences in individual- and organisation-level outcomes were found. Table 27. Time spent working in the current role Time % Up to 1 year 30.0% 1-2 years 20.0% 2-5 years 24.0% 5+ years 26.0%


Table 28. Kruskal-Wallis test for individual- and organisation-level outcomes by time working in the current role (only those who stayed in the same organisation through both survey waves)

Chi-Square

df Asymp. Sig.

Ability to connect with others to build support for change .688 3 .876


Staying in the boat 6.892 3 .075

Sense of purpose and motivation .660 3 .883


Social energy 7.337 3 .062


Spiritual energy 1.663 3 .645

I respond to change in a way which increases organisation performance. 2.513 3 .473

I would recommend care at my place of work to friends and family. 2.735 3 .434

My organisation regularly reviews its performance to identify areas for improvement and change. 1.181 3 .757

In my organisation it is acceptable to challenge the status quo. 2.268 3 .519

My organisation is resistant to change. 4.599 3 .204

My organisation is good at building employee commitment to its change programmes. 1.429 3 .699

Level of engagement with the School Table 29. Summary of the levels of engagement with the School and group sizes

Engagement with the School Yes(N) No (N) Can comparisons be made?

Downloaded the materials 95 18 No Listened to modules live 80 33 Yes Took part in the online chats during the modules

68 45 Yes

Listened to the recordings of online modules 67 46 Yes Connected with other participants via social media

66 47 Yes

Listened to modules live or listened to the recordings

106 7 No


Listened to modules live

Table 30. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Listened to modules live”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1174.000 1735.000 -.830 .407




Social energy 1025.000 1586.000 -1.783 .075







My organisation is resistant to change. 1159.500 3860.500 -.061 .952


Those who listened to the modules live reported a significantly greater change in their ability to connect with others to build support for change.

Table 31. Descriptive statistics for change in “Ability to connect with others to build support for change” levels by “Listened to modules live”

Ability to connect with others to build support for change I listened to modules LIVE Change in score (After-


No .2165 33 .61131 Yes .5217 79 .61863


Took part in the online chats during the modules

Table 32. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Took part in the online chats during the modules”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1427.000 3705.000 -.480 .632

Staying in the boat 1298.000 2333.000 -1.251 .211



Social energy 1089.500 2124.500 -2.489 .013





My organisation regularly reviews its performance to identify areas for improvement and change. 1034.000 1937.000 -2.082 .037

In my organisation it is acceptable to challenge the status quo. 1136.500 2082.500 -1.576 .115



Those who took part in the online chats during the modules reported an increase in their agreement with the statement “My organisation regularly reviews its performance to identify areas for improvement and change”. On the other hand, those who didn’t take part in the online chats reported slight decrease in this outcome.


Table 33. Descriptive statistics for change in “My organisation regularly reviews its performance to identify areas for improvement and change” levels by “Took part in the online chats during the modules”

My organisation regularly reviews its performance to identify areas for improvement and change. I took part in the online chats during the modules

Change in score (After-Before) N Std. Deviation

No -.0952 42 1.10010 Yes .3594 64 1.28936

Listened to the recordings of online modules

Table 34. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Listened to the recordings of online modules”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1449.500 3727.500 -.346 .730




Social energy 1435.000 3713.000 -.432 .666




I would recommend care at my place of work to friends and family. 680.000 1905.000 -1.180 .238


In my organisation it is acceptable to challenge the status quo. 1208.000 3353.000 -1.037 .300




Connected with other participants via social media

Table 35. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Connected with other participants via social media”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1224.500 2352.500 -1.793 .073


Sense of purpose and motivation 1259.000 2387.000 -1.601 .109


Social energy 1322.500 2450.500 -1.213 .225









Actions taken as a result of the School

Table 36. Summary of the types of actions taken as a result of the School and group sizes

Outcome Yes(N) No (N) Can comparisons be made?

Shared learning with others in your organisation 89 24 Yes Shared learning outside of your organisation (e.g. via social media)

61 52 Yes

Recommended the School to others 99 14 No


Sought other types of training for change management and leadership

52 61 Yes

Become a certified change agent 57 56 Yes

Shared learning with others in your organisation

Table 37. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Shared learning with others in your organisation”

Mann-Whitney U

Wilcoxon W



Rocking the boat 954.000 1254.000 -.726 .468




Social energy 722.500 1022.500 -2.373 .018


Spiritual energy 906.500 4822.500 -1.063 .288







Those who shared learning with others in their organisation reported an increase in social energy, while those who didn’t share learning reported a decrease.


Table 38. Descriptive statistics for change in “Social energy” levels by “Shared learning with others in your organisation”

Social energy Shared learning with others in your organisation


No -.1292 24 1.40325 Yes .3394 88 .77401

Shared learning outside of your organisation

Table 39. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Shared learning outside of your organisation”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1285.000 2611.000 -1.586 .113




Social energy 1464.500 2790.500 -.533 .594









Those who shared learning with others outside their organisation reported an increase in their agreement with the statement “I respond to change in a way which increases organisation


performance”. On the other hand, those who didn’t share learning reported slight decrease in this outcome.

Table 40. Descriptive statistics for change in “I respond to change in a way which increases organisation performance” levels by “Shared learning outside of your organisation”

I respond to change in a way which increases organisation performance. Shared learning outside of your organisation (e.g. via social media)


No -.1765 51 .86501 Yes .2881 59 .81051

Sought other types of training for change management and leadership

Table 41. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Sought other types of training for change management and leadership”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1215.000 3106.000 -1.997 .046



Knowledge about change 1269.000 3160.000 -1.692 .091

Social energy 1496.000 3387.000 -.349 .727



I respond to change in a way which increases organisation performance. 1395.000 3225.000 -.672 .502







Those who sought other types of training for change management and leadership reported a greater increase in their ability to rock the boat, compared with those who didn’t seek additional training.

Table 42. Descriptive statistics for change in “Rocking the boat” levels by “Sought other types of training for change management and leadership”

Rocking the boat Sought other types of training for change management and leadership

Change in score (After-Before)

N Std. Deviation

No .3426 61 .84389 Yes .5657 51 .68356

Become a certified change agent

Table 43. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “Become a certified change agent”

Mann-Whitney U

Wilcoxon W



Rocking the boat 1206.000 2802.000 -2.115 .034




Social energy 1282.000 2878.000 -1.670 .095






In my organisation it is acceptable to challenge the 1396.000 2881.000 -.226 .821


status quo.


My organisation is good at building employee commitment to its change programmes. 1137.000 2568.000 -1.452 .147

Those who became certified change agents reported greater change in their ability to rock the boat, compared with those who didn’t become certified. The former groups also reported an increase in their sense of purpose and motivation, and agreement with the statement “My organisation regularly reviews its performance to identify areas for improvement and change”. In contrast, those who didn’t become certified reported a decrease in those scores.

Table 44. Descriptive statistics for change in “Rocking the boat” levels by “Become a certified change agent”

Rocking the boat Become a certified change agent Change in score (After-


No .2705 56 .69379 Yes .6179 56 .82707

Table 45. Descriptive statistics for change in “Sense of purpose and motivation” levels by “Become a certified change agent”

Sense of purpose and motivation Become a certified change agent Change in score (After-Before) N Std. Deviation No -.0313 56 .51800 Yes .2589 56 .58768

Table 46. Descriptive statistics for change in “My organisation regularly reviews its performance to identify areas for improvement and change” levels by “Become a certified change agent”

My organisation regularly reviews its performance to identify areas for improvement and change. Become a certified change agent Change in score (After-Before) N Std. Deviation No -.0577 52 1.34912 Yes .4074 54 1.07315

Organisational context Table 47. Summary of organisational context statements and group sizes

Organisational context Agree or Strongly agree

Disagree or Strongly disagree

Can comparisons be made?


26 34 Yes



12 42 No


15 51 No


Table 48. Wilcoxon Mann-Whitney test for individual- and organisation-level outcomes by “In my organisation it is acceptable to challenge the status quo”

(Only those who stayed in the same organisation during both survey waves)

Mann-Whitney U

Wilcoxon W


Ability to connect with others to build support for change

366.000 717.000 -.593 .553

Rocking the boat 378.000 729.000 -.404 .686




Social energy 362.500 858.500 -.653 .514



References Harding, J. (2006) Census. In Jupp, V (ed) The SAGE Dictionary of Social Research Methods. London: Sage

Health and Social Care Information Centre (HSCIC) (2014). NHS Workforce: Summary of staff in the NHS: Results from September 2013 Census.

Rossi, Peter Henry; Mark W. Lipsey; Howard E. Freeman (2004). Evaluation: A Systematic Approach.

shcr review 2015 - appendix 3 impact measurement

Health & Medicine