1 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

The School for Health and Care Radicals: what impact has it had?

Appendix 1: A review of existing literature

Stella Martorana, CIPD

2 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Contents Note......................................................................................................................................................... 2

Introduction ............................................................................................................................................ 3

The need for bottom-up and top-down interventions ........................................................................ 3

Social Learning..................................................................................................................................... 3

Connectivism ....................................................................................................................................... 6

Massive Open Online Courses ........................................................................................................... 10

Traditional evaluation models ........................................................................................................... 15

References ............................................................................................................................................. 20

Note This appendix forms part of the review of the 2015 School for Health and Care Radicals run by NHS Improving Quality. The review was conducted by the research arm of the CIPD between November 2014 and September 2015. The main report, which presents an overview of the findings, is also available.

3 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Introduction In this section, we present a literature review exploring existing thinking and evidence on social learning and organisational change that underlie the focus, principles and methods of the evaluation, in particular the impact model.

The need for bottom-up and top-down interventions Top-down initiatives to change human behaviour and organisational cultures such as imposed reforms have often failed their goal. As highlighted by Robert Francis, who led the inquiry into the failings at Mid Staffordshire Foundation Trust, ‘whatever initiatives are started at the top, unless the clinical soil is fertile, the seeds will inevitably fall to stony ground at trust level’ (Mid Staffordshire NHS Foundation Trust Inquiry 2011).

One of the major weaknesses of top-down initiatives relies in the underpinning behaviourist paradigm, which describes human behaviour and learning process in terms of stimulus and response.

On the other hand, programmes designed to solely influence the behaviour focusing on the individual dimension can struggle with the application of learning outcomes to real-life contexts. In fact, situational factors do play a role in the display of new behaviours – for instance, poor organisational systems can sabotage any change programme (Dixon Woods et al. 2013).

Consequently, a balanced yet challenging approach of top-down and community-based interventions encompassing governance, social and individual levels increases the odds of impacting and changing the targeted set of behaviours. The complexity of social cognition requires an equally complex and inclusive change programme with solid theoretical basis. The simple causal relationship between stimulus (the reform) and response (change) needs replacing with more sophisticated models.

Social Learning Albert Bandura’s (1971) social learning theory, is one of the most influential in this area, outlining crucial processes involved in any learning experience such as observation, cognitive processes and self-regulative influences. Further, it is also a social cognitive theory for organisational management (Wood & Bandura, 1989) that identifies new levers for change such as social interaction.

Firstly, observation is a powerful type of learning. There is no need to experiment directly all possible actions and their consequences, because we use other people’s experiences to modulate our behaviour or emotional response. This concept links to the one of tacit knowledge (Eraut, 2000), which will be discussed further in the section on connectivism.

Secondly, any external event is filtered through cognitive processes. Attention and retention processes are essential to ensure memory of reinforcements such as reward and punishment, which inform action. Models are then transformed into symbolic codes acting as clues in people’s mind. According to this theory even symbolic codes in our mind can be manipulated and changed. This happens when people are involved in rich discussion of emotional values where new definition of concepts is promoted (Bandura, 1977).

For instance, the first webinar of the SHCR explains the difference between a rebel and a trouble maker to help individuals channel and productively use their energy. Emotion-arousing words are used to facilitate the conditioning and the emotional connection. Adopting a concept based on a new meaning and emotional response precedes any change in the status quo.

Lastly, people are able to create self-regulative influences, which are self-reflective and self-reactive systems mediating the effect of contextual factors. For instance, internal standards of behaviour or

4 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

people’s belief in their efficacy are self-regulated mechanisms informing decisions and actions (Bandura, 1991).

In the following sections we discuss important themes emerging from Bandura’s theory and the theoretical contributions of other influential authors.

Practice in a social environment Learning requires memory of reinforcements and new content, therefore retention processes play a crucial role. They are enhanced by mental or actual practice, which increases the odds of new behaviour display (Bandura, 1971).

Consequently, behaviour is learned before it is performed. Also Marsick and Watkins’ theory of social learning (2001) argues that learning occurs unconsciously, as a by-product of social activity.

The importance of practice is stressed by Rogers’ (2013) diffusion of innovation theory. According to him people are more likely to adopt new behaviours if they can experiment with them. Students of the SHCR have the opportunity to discuss what they learnt with peers and embed it into practice. Those who engage on social media such as Facebook share ideas on how to be effective change agents and become more familiar with the content of the modules, in other words they practice new learnings in a protected environment. The comments below, posted on the Facebook group of the school exemplify the concept.

I'd say the networks and social interaction are invaluable. They provide a safe space for us as a group to reflect on, help one another with, and work through the material that we heard presented on Friday. I would, therefore, say that, for me, the effectiveness of the two learning modes are indissociable and complementary.

The level of engagement with the school varies widely, therefore the impact model devised by CIPD (see Appendix 2) includes this aspect among the assumptions.

Diversity Bandura (1977) points out that innovative behaviour emerges solely through modelling, therefore diversity is a pre-requisite. In this context innovative behaviour is expected from ‘radical agents’, who are supposed to challenge the status quo. Also Engestrom (2001), author of the Expanding Learning Theory, points out that diversity and boundary crossing is essential in the learning process.

This means that as long as conformity is the organisational rule, individuals won’t be able to pursue radical change; the environment will lack of new and diverse models from which individuals can learn.

According to Deloitte (2013) diversity needs to be valued and pursued: “leaders and organizations must let go of the idea that there is one right way and instead focus on creating a learning culture where people feel accepted, are comfortable contributing ideas, and actively seek to learn from each other” (ibidem).

I think the conversation over the weekend has helped to embed the ideas, with all these great additional viewpoints layered on and offering a breadth of perspectives.

The CIPD impact model for the school includes outcomes such as ‘inject new ideas’, ‘knowledge of change methods’ which imply exposition to different views and ideas.

Contextual barriers and facilitators According to Bandura (1977) both learning and its translation into practice depend massively on the presence of contextual barriers or positive reinforcements. He moves even further by saying that fear

5 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

and defensive behaviour can be extinguished by observing others engagement in feared activities without any adverse consequences’ (1971). Here the importance of the contextual response is emphasized.

Along the same lines, an influential training evaluation model formulated by Kirkpatrick and Kirkpatrick (2009) developed the concept of organisational drivers: “action and processes that are carried out by others that are designed and implemented to reinforce the participants to perform their critical behaviours” (2009:7). In essence, they enable the translation of learning outcomes into new behaviours.

The weakness of these theories lies on the incapacity to explain why adverse consequences do not always discourage individuals from performing specific actions. For instance, new social movements often survive sanctions and oppositions. Social identity theory (Tajfel, 1982) suggests that situational consequences are counter-balanced by intra-group processes acting like a glue. This theoretical framework also informs the impact model devised by CIPD, where intermediate outcomes such as ‘support and validation through network of peers’ are identified.

In the case of the NHS, many individuals can find themselves in organisational contexts disseminated with barriers. The manipulation of their group membership can be seen as an application of social identity theory. Belonging to a group of ‘change radicals’, which is composed not necessarily by their colleagues but by people working in different companies and countries sharing the same goal, provides them with a new social identity besides group values and shared norms that will support and sustain their behaviour.

Self-regulated influences To overcome the limits of explanations based on situational factors only, which leads to a weather vane model of individual, Bandura introduces the concept of self-regulated influences (1971). Self-regulated behaviour refers to internal standards that enable us to act and judge, for instance. These are acquired through previous reinforcement processes. An example is self-efficacy, which is defined as ‘people's beliefs about their capabilities to produce designated levels of performance that exercise influence over events that affect their lives. Self-efficacy beliefs determine how people feel, think, motivate themselves and behave’ (Bandura, 1991).

There are four determinants of people’s beliefs about their self-efficacy:

Master experience. Previous successful experiences increase self-efficacy perceptions Vicarious experience. When people see individuals similar to themselves making efforts to achieve select objectives and finally succeed, this will positively affect confidence in their abilities Social persuasion. This is a verbal empowerment exercise, which consists of instilling high levels of self-efficacy. This motivates and sustains the performance of individuals, who try hard to succeed. Physiological status. Emotional and physiological reactions such as anxiety or fatigue are used to infer the ability to carry out specific tasks, such as speaking in front of an audience (Bandura, 1977).

The emphasis on beliefs and perceptions offers the key to interpret the model. The actual difficulty of a task is not in discussion, yet the perception of difficulty matters. In addition, self-efficacy is a

6 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

situational concept, because people’s perceptions of their abilities to accomplish tasks relates to specific contexts.

Conclusions on social learning Bandura’s theory sheds light on the complexities of learning and emphasises the cognitive, interactive and social aspects of it. Undoubtedly, the instruction model where content is transferred from one person to another is surpassed; the behaviourist theory according to which learning derives merely from the presence of rewards or punishment in the environment, is de-emphasised. Important concepts and themes such as diversity, social interaction, and self-efficacy emerge from his theoretical framework.

Many other authors such as Vygotsky (1962) proceeded along similar lines. He suggested that interactions between learners and their peers enable learning and that culture also has a role in knowledge construction. He devised a model of a complex act as the result of subject, object and mediating artefact, in other words in this model the individual cannot be separated from his cultural context.

Similar assumptions are the core of Lave and Wenger’s situational learning model (1991), where learning cannot be disentangled from the context in which it happens but is actually the result of interactions between individuals belonging to the same community. According to them, learning implies knowing how to behave in the community and impacts on people’s identity because they are rooted in the group membership. However “rather than asking what kind of cognitive processes and conceptual structures are involved, they ask what kinds of social engagements provide the proper context for learning to take place” (Hanks, 1991:14).

These insights are reflected in various ways in the design of the SHCR and in the CIPD impact model. In its assumptions, the model takes social networking to be a central aspect of how participants apply learning to practice. We also assume that participants are motivated to learn and become a change agent.

The insights are also reflected in our understanding of a good change agent. In particular, in the intermediate outcomes of the school, we identify knowledge sharing, encouraging and supporting others and having support and validation through a network of peers.

In the following sections we will discuss the learning theory informing Massive Open Online Courses such as the SHCR, which is called ‘Connectivism’.

Connectivism “Learning is now happening through communities of practice, personal networks, and through completion of work-related tasks...know-how and know-what is being supplemented with know-where” (George Siemens, 2012).

With these words Siemens briefly outlined the concept of connectivism, which has been envisaged as the learning theory for the digital age (Siemens, 2014). This theory explicitly informs Massive Open Online Courses, but encompasses any social and interactive digital learning model.

According to this theory, knowledge is distributed along a network and learning depends on the weight of the connections between the units. The final learning outcomes don’t reside in specific nodes but result from the pattern of links.

Each person can be seen as a node and also technologies are embedded in the net. Consequently, interaction is the source of learning because it enables changes and achievement of new and ever-

7 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

changing states. It is worth clarifying that interaction does not equate with conversations between individuals, but needs to be conceived in broader terms, encompassing interaction with books and movies for instance.

The flow of information in and out of the system (individual and organisation) is essential and its diversity is necessary for learning purposes. The importance of diversity is discussed further in the section on Social Learning (see above).

As Siemens explains, this theory ‘addresses the challenges that many corporations face in knowledge management activities. Knowledge that resides in a database needs to be connected with the right people in the right context in order to be classified as learning’ (Siemens, 2014). Moreover, knowledge needs to be made explicit and this introduces the concept of ‘tacit knowledge’, to which we now turn.

Tacit knowledge Tacit knowledge derives from unplanned and non-formal learning and has clear linkage with memory of specific events (Eraut, 2000). The concept is underpinned by Tulving’s theory of memory (in Eraut 2000), which draws a distinction between memory for specific events (episodic) and general knowledge (semantic). Generally, semantic memory is derived from the episodic one, to the extent that recalling the actual events sometimes is not possible. The knowledge we have, but are unable to explain (other than through our recollection of individual episodes of experience) becomes ‘tacit’ (Polanyi, 1966).

For instance, a person maybe socialised into values and norms of an organisation by interacting with the members of the community and participating in events, without being aware that a learning process is taking place (Eraut, 2000). This recalls also the peripheral learning process outlined by Lave & Wenger (1991) which leads to knowledge on how to behave in community of practices.

Kolb’s uses the term experiential learning: “personally experienced events are stored in episodic memory and, over time, used to construct generalised knowledge structures in semantic memory” (Kolb, 1984:8 in Eraut, 2000:117).

Explicit learning can lead to implicit learning as well as tacit knowledge can lead to explicit knowledge. The latter has been conceived as a knowledge management resource (Herschel, Nemati, Steiger, 2001) enabled by storytelling, conversation and dialogue with other individuals. Discursive practices force individuals to articulate their learning and facilitate reflection upon them. Also Kolb (1984) recognised that reflection on experiences unlocks explicit knowledge.

In an organisational context “the conversion of tacit knowledge to explicit knowledge is critical because it is a prerequisite to the knowledge amplification process wherein knowledge becomes part of an organisations’ knowledge networks” (Herschel, Nemati, Steiger, 2001:107).

In the following section we will introduce the main features and advantages of social technologies in learning and organisational settings. The School for Health and Change Radicals integrate social media in the learning experience, therefore the evaluation included qualitative analysis of comments posted on Facebook and also data on twitter and Facebook users.

Social technologies in organisational settings Interaction on social media platform is wide spread. In the UK, over eight in ten of adults go online and over two-thirds of those have a current social networking site profile (Ofcom, 2014). The most popular platforms are Facebook, Twitter, Youtube and Linkedin, which are used for personal purpose in 76% of the cases (CIPD, 2013). Few people use them for work purpose (26.6%) but recently organisations have begun to exploit their potential.

8 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

The literature on their uses in organisational settings has flourished in recent years and one of the main qualitative studies has been carried out by CIPD in 2013. Their in-depth analysis of case study organisations led to the identification of important benefits. Below we present the ones which link more closely with the SHCR experience.

Shared purpose. Company values and mission can be embedded in online messages promoting engagement and identification with the organisation.

Employee voice. This refers to open and dynamic online communication where potentially all individuals can express their views and engage in open conversation.

Workplace community. Online conversation involving both personal and work themes can be of inherent value in itself as they can strengthen the identification with the organisation and promote better work relationships.

Insight and innovation. Employee insight and experience can be shared with the community and lead to creative and collective re-elaborations.

Learning and development. Social media enable collective learning through multi-directional dialogue on interactive intranet, platforms, and forum.

Distributed leadership is possible because all employees have an opportunity to express their voice and build informal networks across the organisation. Traditionally the literature has focused on the individual leader (Turnbull, 2011) but the contemporary challenges in the world of work require more responsive, flexible leadership structures able to cope with changes in workforce, work practices and markets. Leadership needs to be ‘adaptive’ (Heifetz and Laurie 1997; Heifetz 2009 mentioned by Turnbull, 2011) and underpinned by systemic assumptions. ‘Distributed leadership does not negate the requirement of leadership at the top, but requires this leadership as part of a wider leadership capability, where the social capital of the organisation is harnessed (Northern Leadership Academy; 2007:8). This concept will be discussed further in the following section.

The research carried out by CIPD raised also questions on the relevance of the content shared online. The case study of Santa Fe Group offers an interesting perspective (CIPD, 2013). The organisation developed an enterprise social learning network, The Academy Online, after previous unsuccessful attempts to involve employees in an enterprise social network. Narrowing the focus on learning and development gave this a focus and facilitated online dialogue, with many employees started sharing views and making questions on select topics. This turned into a powerful tool to build an organisational culture in a company with geographical dispersion (ibidem).

In the case of the SHCR, Tweetchats and Facebook group are focused on topics discussed during the weekly webinars facilitating online conversation and involvement. Personal topics still surface in the forms of personal anecdotes, requests for new ideas to tackle difficult situations. These are valued positively, as expressions of outcomes such as ‘encourage and support others’, ‘realise the value of relationship’ or others depending on the specific input.

The SHCR does not impose the use of social media, people who register to the school are free to decide if they want to participate in Tweetchats, become members of the Facebook group and interact with others more generally. This freedom is crucial according to Jane Hart (2011), Founder of the centre for Learning & Performance Technologies. According to Hart, a top-down approach to implementing social learning needs replacing with a supportive bottom-up approach. In other words, individuals already collaborative need to be supported and encouraged instead of forcing everybody to work and learn collaboratively.

9 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

In the following section, we discuss distributed leadership, one of the main benefits of using social technologies in organisations.

Distributed leadership “Distributed leadership is not something ‘done’ by an individual ‘to’ others, or a set of individual actions through which people contribute to a group or organisation… Distributed leadership is a group activity that works through and within relationships, rather than individual action.” (Bennet et al., 2003: 3) The literature on distributed leadership and affine concepts such as ‘shared’ or ‘collective’ leadership has flourished in the last decades. The concept permeates the work of authors such as Gibb (1954) who explains how leadership is best conceived as a ‘group quality, as a set of functions which must be carried out by the group’. His theory was formulated over fifty years ago and Bolden (2011) even suggests that the concept dates back to centuries ago.

The concept, along with its implications in organisational settings, has been embraced fairly recently though. The reasons and implications are mainly political, as highlighted below by Day et al. (2009, in Bolden, 2011:260)

“It is often some form of external pressure that prompts efforts to distribute leadership more broadly, for example, pressure to improve disappointing school performance, introduction of new policies and programmes requiring new teaching and learning capacities. Greater ‘distribution of leadership’ outside of those in formally established roles usually depends on quite intentional intervention on the part of those in formal leadership roles”

The contemporary challenges in the world of work require more responsive, flexible leadership structures able to cope with changes in workforce, work practices and markets whereas the traditional literature has focused on the individual leader (Turnbull, 2011).

Nowadays leadership needs to be ‘adaptive’ (Heifetz and Laurie 1997; Heifetz 2009 in Turnbull, 2011) and underpinned by systemic assumptions.

Bennett et al. (2003 in Bolden, 2008) identified three main features of distributed leadership, encompassed by most literature on this concept.

• Leadership is the property of a network of people instead of an individual

• The boundaries of leadership are open

• Expertise is distributed along the network rather than located in few individuals only

These three characteristic clearly link to the theoretical framework called ‘Connectivism’, which is discussed further in chapter 3.

Recognising the role played by the network rather than the individual should not lead to ignore the role played by vertical leadership though. The key is how vertical and shared leadership can be utilised to leverage the workers’ capabilities (Pearce, 2004 in Bolden, 2011: 264).

Bolden outpoints that formal leaders are “only the tip of the iceberg” (2008:42) and Harris (2003) stresses that this concept is useful to overcome the leader/follower divide.

10 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

In essence, ‘distributed leadership does not negate the requirement of leadership at the top, but requires this leadership as part of a wider leadership capability, where the social capital of the organisation is harnessed’ (Northern Leadership Academy; 2007:8).

Social technologies can be powerful resources to harness the aforementioned social capital, as highlighted by a CIPD research (2013) which identified distributed leadership as one of their benefits (see chapter 3).

They overcome corporate barriers and offer employees the opportunity to express their voice and build informal networks across the organisation. The School for Health and Social Care exploits this potential and use social media to leverage distributed leadership.

In the following section we will discuss Massive Open Online Courses, a label that can be applied to the SHCR. Students from all over the world can participate in a social learning experience where they interact with peers and share ideas and opinions, building informal networks and transforming tacit knowledge in explicit learning.

Massive Open Online Courses Massive open online courses (MOOCs) are worldwide and web-based courses open to anyone who logs on and registers. They result from the social evolution of digital learning, which arose in the nineties’ with the online publication of videos of lecturers and went mainstream with the OpenWare Project led by the Massachuttes Institute of Technology.

Their fast-moving nature has eluded clear definitions and evaluation models. They attract different types of learners, are now delivered in professional contexts as well, and their features continue to evolve.

The first MOOC was pioneered by researchers George Siemens and Stephen Downes, who led a course on at the University of Manitoba in 2008. The success, confirmed by 25,000 enrolments, marked a decisive turning point in the history of digital learning.

Stanford University replicated the model four years later for a course on Artificial Intelligence which attracted 58,000 students. Predictions of 100% increased rate in registrations followed (Deloitte, 2014) and recently the Bill & Melissa Gates foundation funded the MRI (MOOC Research Initiative) to stimulate further research on the trend.

The outreach of MOOCs is impressive because there are no admission requirements. Any financial barrier to education is lifted, promoting the mass dissemination of knowledge. 160,000 students enrolled to the MOOC on Artificial Intelligence run by Stanford University in 2011 and 308,000 students enrolled to the first six batch of UK MOOCs, developed by Edinburgh University.

We could argue that technology is the biggest barrier to the mass potential of MOOCs for two reasons. Firstly, the possession of digital devices is a pre-requisite; secondly, poor confidence with them can discourage individuals from enrolling. However, MOOCs always offer instructions and demonstrations on how to navigate through them. Moreover, the ‘Net Generation’ (Brown, 2005) also called ‘Digital Natives’ (Prensky, 2012) are familiar with technology and accustomed to interact on digital platforms such as Facebook, Twitter and Linkedin.

Consequently, the lack of barriers leads to global and massive participation. The 2015 SHCR attracted registrants from 36 countries: Argentina, Australia, Austria, Belgium, Brazil, Canada, Denmark, England, France, Georgia, Germany, Greece, India, Ireland (Republic of), Italy, Netherlands, New

11 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Zealand, Nigeria, Norway, Northern Ireland, Pakistan, Qatar, Russian Federation, Saudi Arabia, Scotland, Singapore, Slovenia, Spain, Sweden, Switzerland, Thailand, Tunisia, Ukraine, USA and Wales.

Classification of MOOCs The most basic taxonomy of MOOCs has been developed around the social and interactive dimension of learning, distinguishing between xMOOCs and cMOOCs (Bates, 2014).

xMOOCs derive from the lecture teaching model. The focus is on content delivery although some interaction with other students or with the lecturer is possible. Their specific platforms store content related to the course such as video recordings besides important data on users and their learning experience such as demographics, attendance, marks in computerised tests and final results.

cMOOCs, such as the SHCR, are underpinned by the connectivist theoretical approach. Downes and Siemens, creators of this model, stress that learning results from the process of sharing knowledge between participants (Downes, 2012).

This taxonomy has been useful to develop an evaluation framework for the SHCR, in fact the model focuses on intermediate outcomes such as ‘realise the value of relationships’, ‘encourage and support others’, ‘align with a shared mission’, and others which are distant from the results of formal examinations and test.

Schneider (2013) developed a taxonomy on the general structure and a complimentary one on the interactive learning environment (ILE).

According to the former, MOOCs need to be categorised on the basis of the following aspects: name, ID, publisher, platforms, domain, level, target audience, use, pace and accreditation.

It is worth mentioning that ‘accreditation’ introduces one of the most popular debates on MOOCs, regarding how and for which purpose the learning outcomes need to be assessed and certified. This will be discussed in the following section.

Engagement with MOOCs “The person taking the MOOCC is like a person reading a book, playing a game, or taking a trip to the city. It is impossible to talk about ‘the objective’ of such an activity – some people want to learn something (and others something else), others are doing it for leisure (and others as part of their jobs), others to make friends (and others to get away from their friends for a while)…” (Michelle Barclay)

The evaluation of MOOCs is a challenge to the way education providers have assessed and certified learning outcomes so far. Traditional evaluation methodologies need re-visiting in the light of new types of learners, knowledge models and technologies.

Technology is so significant within our lives that researchers labelled new generations on the basis of their familiarity with digital devices. Prensky (2012) coined the term ‘Digital natives’ to describe a generation of tech-savvy individuals with a positive attitude to e-learning, accustomed to the speed of the internet and to multi-task. The ‘Net Generation’ is goal and achievement oriented, with a preference for active learning and social activities (Brown, 2005).

Also the diversity in learners’ needs and intentions is critical. Some are curious on select topics and participate for the sake of knowledge itself, others use MOOCs as gap fillers useful in their profession or want to test and certify their interest in a topic before enrolling to University; others are keen to experiment innovative e-learning systems.

12 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Phil Hill (2013) identified five student patterns in Coursera-style MOOCs. Using his terminology and descriptions, the five user profiles are the following.

No-Shows. These students appear to be the largest group of those registering for a Coursera-style MOOC, where people register but never login to the course while it is active.

Observers. These students login and may read content or browse discussions, but do not take any form of assessment beyond pop-up quizzes embedded in videos.

Drop-Ins. These are students who perform some activity (watch videos, browse or participate in discussion forum) for a select topic within the course, but do not attempt to complete the entire course. Some of these students are focused participants who use MOOCs informally to find content that help them meet course goals elsewhere.

Passive Participants. These are students who view a course as content to consume. They may watch videos, take quizzes, read discuss forums, but generally do not engage with the assignments.

Active Participants. These are the students who fully intend to participate in the MOOC and take part in discussion forums, the majority of assignments and all quizzes & assessments.

Figure 1. Emerging student pattern in Coursera-style MOOC (from Hill, 2013)

13 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

The drop-out rate is massive according to further data mining. The University Of Pennsylvania Graduate School Of Education (Penn GSE) found that MOOCs have a 4% completion rate, with most user dropping-out after 1-2 weeks (Perna, Ruby, 2013).

Katy Jordan, PhD student at Open University, collated data on 29 different MOOCs; her findings show completion rates between 13% and 40% with an average of 7.6%. She also investigated the effect of two independent variables, assessment type (auto grading vs peer) and course length on the completion rates (Jordan, 2013). Data have been represented on a visualisation tool, which shows higher completion rate for short and auto-graded MOOCs, although the statistical significance of this relationship is unclear.

Fig. 2 Completion rates for MOOCs (Jordan, 2013)

14 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Fig. 3 Completion rates for MOOCs (Jordan, 2013) According to Andrew Ho, professor in Harvard’s Graduate School of Education, “focusing on completion rates limits our imagination of what might be possible with MOOCs. A better criterion for success might be for students to complete more of the course than they thought they would, or to learn more than they might have expected” (Harvard Gazette, 2014).

In fact, the criticism towards MOOCs lies on comparisons with traditional courses and on the ways completion rates are calculated.

The completion rate is often calculated by # certificate earners divided by the total # of people who registered for a course, which is massive, but many applicants know that they won’t complete it since the beginning (Reich, 2014). In a research carried out on EdX MOOCs, 58% of registrants wanted to earn a certificate, 25% intended to audit, 14% were unsure and 3% intended to browse. Interestingly, users with the intention of completing the course were 4.5 times more likely to do so (ibidem).

Lastly, the comparison between those who complete the course with the high number of registrants should be complemented by comparisons with those who would have accessed the select content and interacted with their peers in an ILE that overcome geographical location, time commitments, financial situation and other barriers. Likely, that figure will be extremely low.

Certifications How and for which purpose the learning outcomes need to be assessed and certified are frequent questions with multiple answers.

Most providers release certificates attesting the attendance and completion of the course upon payment of a charge. They are promoted as documents which testify interest in a select topic and can be used to increase the chances to be accepted in important Universities or to showcase learning achievements to employers (Coursera, EdX, FutureLearn, 2015).

Here the disruptive potential of MOOCs towards tuition-based degree programmes is unveiled, although learning providers state to offer alternative credentials.

Below a list of certificates released by three popular MOOC providers to date

COURSERA

Verified certificate. According to Coursera (2015), this is an official, verifiable, shareable document that can help advance participants’ education and career. Users need to record their typing pattern, take a photo of themselves and one of their ID. When submitting coursework, the identity needs to be confirmed by providing matching typing sample and photo.

Specialisation Certificate. This is a group of related courses aimed at the attainment of specific skills and their application in a capstone project. A Verified certificate need to be earned for each course and the capstone project is available only after users’ have completed all courses in the specialisation.

EDX EdX offers honor code certificates of achievement (the only ones where the identity of the user is not verified), verified certificates of achievement and XSeries certificates of

15 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

achievement. The latter are similar to the Specialisation certificates released by Coursera, because participants need to complete a series of courses making up a series. As publicised on Edx website, they are a ‘true’ achievement you can use to advance your career or simply impress yourself. Contrary to Coursera, EdX does not verify the typing style. FUTURE LEARN FutureLearn releases statements of attainment upon completion of an exam. These documents show participant name, name of the course, name and logo of the University, percentage score achieved. It also outlines the subject areas, the number of hours of study required per week and clearly states that the final exam was carried out under invigilated conditions. Exams are delivered by Pearson VUE in phisical centres, where the identity is checked.

Traditional evaluation models Kirkpatrick (1959) devised a successful evaluation model for training courses in the fifties, which identified four levels of learning outcomes and was recently refined and developed (2009). 1. Reaction reflects the user satisfaction to a select training event 2. Learning refers to the acquirement of specific knowledge, skills, attitudes, commitment based on their participation in a learning event. 3. Behaviour. Knowledge and skills acquired during the training need to be transferred to the job 4. Results. ‘To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement’ (Kirkpatrick, 2009:21) An additional level investigating the wider implications and effects of learning on the society is included in the alternative learning model of Kaufman (1995).

The evaluation of learning has often been focused on specific training events (level 1 and 2) therefore based on metrics such as attendance (level 1) or scores (level 2). According to Kirkpatrick, the most productive approach to evaluation needs to start from the end, in other words from the definition of results. Careful planning is a pre-requisite to training evaluation along with the involvement of business leaders.

Firstly, the top management expectations need to be clarified, in order to define their concept of success and break it down into measurable indicators. The agreement of business leaders and evaluators is crucial at this stage, where their specific accountabilities are defined.

Once there is agreement on the desired results, trainers need to involve line managers in the definition of behaviours that are critical to the achievement of successful results (level 3) but also define which situational factors (processes, systems, rewards, etc.) will support the new behaviour.

At this stage, it is possible to identify and develop specific training events (level 1&2).

16 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Our evaluation conflicts with this view, where a top-down approach prevails, because individuals have different expectations of a MOOC, which might not align with top management’s view. The evaluation needs to remain open to what might emerge, integrating a bottom-up approach to evaluation.

Frameworks for the design and evaluation of MOOCs Grover et al (2013) devised a framework for the design and evaluation of MOOCs which is intertwined with the distributed intelligence framework devised by Pea (1993).

“The intelligence revealed through these (cognitive) practices are distributed – across minds, persons, and the symbolic and physical environments, both natural and artificial. Gregory Bateson remarked that memory is half in the head and half in the world” (Pea: 1993:47).

There is a linkage with the connectivist theory as well because distribution implies connections and the transmission model of education is replaced with the generation of distributed knowledge in a network environment.

17 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Fig. 4 Framework for the design and evaluation of MOOCs (Grover at al 2013:2)

The framework identifies four dimensions across which the intelligence is distributed and require the participation of all professionals involved in the design and implementation of the digital course, including technologists, data scientists and instructional designers.

The interactive learning environment includes content, instruction, community and assessment. The above course elements are designed on the basis of different factors such as underpinning learning theories, learners’ background and intentions, availability of select platforms and more.

For instance, some MOOCs emphasise the transmission of content (cMOOC) and other ones the interaction between peers (xMOOCs). EdX, Coursera and many others MOOCs providers break down video content into short recordings (5-10 minutes) to facilitate the learning process whereas others hold onto the traditional lecturing model and broadcast long videos.

Also the assessment varies on the basis of users and aim of the course; for instance, courses that will lead to credits for students will require detailed forms of assessment.

Learner background and intentions captures the variety of needs and goals moving participants. The variety in intentions, from curiosity to fun or skills enhancement leads to the conclusion that course completion is not a good measure of learning outcome (Grover et al. 2013). Research described in previous section of this document also show that intention can be a good predictor of completion rates.

Technology of infrastructure encompasses all technology aspects from the software to the social media platforms and the learning analytics engine powering the MOOC. It involves decision on how data on users and their interactions are collected and analysed.

Finally, the evidence-based improvement is conceived as “meta-MOOC process underpinning design decisions around the ILE and technology infrastructure. Evidence-based improvement is powered by data mining and analytics designed to measure the desired course learning outcomes, and incorporates qualitative evidence from sources like forums and surveys” (Grover et al, 2013:3).

According to this model, data needs to be collected from each dimension and translated into indicators of learning, engagement and distributed intelligence (ibidem).

18 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

The dimensions and elements of this model have been included in our impact model, described in Appendix 2. For instance, learner background and intentions are assumptions (motivated to learn and be a change agent; varying specialism and seniority). The interactive learning environment is included in the activities as well as the technology infrastructure. Analytics regarding the social interaction on platforms such as Twitter and Facebook were collected as part of the research.

The research team also conducted in-depth follow-up interviews focused on participants’ stories of their involvement with the school and subsequent activity as change agents (see Appendix 5). This gave us insight into the potential the school has to contribute to organisational, service and patient outcomes more widely.

An alternative framework was proposed in 2013 as part of the MOOC Research Initiative (Momentum, 2013). It is the result of an online discussion between George Siemens, founder of the first MOOC and e-learning experts, and it is also informed by the Effective Lifelong Learning Inventory developed at the University of Bristol. It identifies nine components, as shown in the graph below.

19 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

Implications for the current review of the SHCR A key implication of these insights for the evaluation of the SHCR is that we need to understand the learner’s objectives from their own point of view and evaluate the outcome based on what they came to the learning environment expecting, rather than be limited to the intentions of the course designers.

Thus, in developing the impact model, we relied on interviews with participants from the 2014 programme as well as discussions with the course designers and SHCR documents. These participant interviews enabled us to build an understanding, in a bottom-up approach, of what they valued in the school and how they benefitted from it.

We also used these interviews to test the language that the course designers used in describing the school. Using cognitive interviewing, we asked participants survey-type questions using phrases such as ‘change radicals’ and ‘rock the boat while staying in it’ and then asking the interviewees what they understood by those terms. In this way, we helped to ensure not only that the concepts were reflective of participants’ experiences, but that the language we used to describe these concepts were also reflective of participants’ realities.

Another implication is that the in-depth qualitative interviews being undertaken a few months after the 2015 school will be important to contextualise the changes we are measuring in the pre and post evaluation surveys.

However, the SHCR is designed to help people develop in reasonably specific ways and it is right we also assess that as well as participants’ experiences. Thus, the impact model also reflects the design of the school and the thinking of those delivering it.

Furthermore, the qualitative bottom-up interviews are important to frame a more positivist evaluation, but do not invalidate it. At a certain point it is entirely valid, once one has developed theory in an inductive way, to test that theory with a broader population. This underpins our basic approach of the two-wave survey, immediately before the 2015 school and six months later.

This evaluation approach fits better with the SHCR than it may do with many other MOOCs. Although its resources remain open for participants to use, the school was run in particular through live sessions delivered on five consecutive Friday mornings. This introduces a degree of standardisation in the delivery of the school and how people participated, which makes it appropriate to adopt a method including a survey at two discrete points in time.

20 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

References BANDURA, A. (1971) Social Learning Theory. New York: General Learning Press. BANDURA, A. (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychological Review. Vol 84, No 2. pp191–215. BANDURA, A. (1991). Social cognitive theory of self-regulation. Organizational Behavior and Human Decision Processes. Vol 50, No 2. pp248–287. BATES, T. (2014) What is a MOOC? [Online] October 12th 2014. Available at: Online Learning and Distance Education Resources http://www.tonybates.ca/2014/10/12/what-is-a-mooc/ [Accessed: 19/02/2015] BENNETT, N., WISE, C., WOODS, P.A. & HARVEY, J.A. (2003). Distributed Leadership. Nottingham: National College of School Leadership. BOLDEN, R. (2008) Distributed leadership. In: Marturano, A. and Gosling, J. (eds) Leadership: the key concepts. London: Routledge BOLDEN, R. (2011) Distributed leadership in organizations: A review of theory and research. International Journal of Management Reviews. Vol 13, No. 3. pp251-269. BROWN, M. (2005) Learning Spaces. In: OBLINGER, D., OBLINGER, J. (eds). Educating the Net Generation. [Online] Educause. Available at: Educating the Net Generation https://net.educause.edu/ir/library/pdf/pub7101l.pdf [Accessed: 19/02/2015] CIPD (2013) Social technology Social Business?. London: Chartered Institute of Personnel and Development [Accessed: 19/02/2015] COURSERA (2015) Terms of Service. [Online] Available at: https://www.coursera.org/about/terms [Accessed: 19/02/2015] DELOITTE (2013) Diversity’s new frontier. Diversity of thought and the future workforce. Available at: http://dupress.com/articles/diversitys-new-frontier/#end-notes [Accessed: 10/03/2015] DELOITTE (2014) Massive Open Online Courses (MOOCs). Not disruptive yet, but the future looks bright. Available at: http://www2.deloitte.com/global/en/pages/technology-media-and-telecommunications/articles/2014prediction-massive-open-online-courses.html [Accessed: 10/03/2015] DIXON-WOODS, M., BAKER, R., CHARLES, K., DAWSON, J., JERZEMBEK, G., MARTIN, G., MCCARTHY, I., MCKEE, L., MINION, J., OZIERANSKI, P., WILLARS, J., WILKIE, P. & WEST, M., 2013. Culture and behaviour in the English National Health Service: overview of lessons from a large multimethod study. BMJ Quality and Safety. Vol 23, No 2. pp106-115. DOWNES, S. (2012) Connectivism and Connective Knowledge. [Online] Available at: http://www.downes.ca/files/books/Connective_Knowledge-19May2012.pdf . [Accessed: 19/02/2015]

21 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

EDX (2015) Learning at edX. [Online] Available at: https://www.edx.org [Accessed: 19/02/2015] ENGESTROM, Y. (2001) Expansive Learning at Work: Toward an activity theoretical reconceptualization. Journal of Education and Work. Vol 14, No 1. pp133-156 ERAUT, M. (2000) Non-formal learning and tacit knowledge in professional work. Journal of Educational Psychology. Vol 70, No 1. pp113-136 FUTURELEARN (2015) Workplace Learning. [Online] Available at: https://www.futurelearn.com/workplace-learning [Accessed: 19/02/2015] GIBB, C. A. (1954). Leadership. In G. Lindzey (Ed.), Handbook of social psychology. Vol 2. pp877-917. Reading, MA: Addison-Wesley. GROVER, S., FRANZ, P., SCHNEIDER, E. and PEA, R. (2013) The MOOC as Distributed Intelligence: Dimensions of a Framework for the Design and Evaluation of MOOCs. In Proceedings of the 10th International Conference on Computer Supported Collaborative Learning (Madison, WI, June 16-19). HANKS, W.F. (1991) Forward to J.Lave and E.Wenger, Situated Learning: Legitimate peripheral participation. Cambridge: Cambridge University Press HARRIS, A. (2003) Teacher Leadership as Distributed Leadership: Heresy, Fantasy or Possibility? School Leadership and Management. Vol 23, No 3. pp313-324. HART, J. (2011). Social Learning Handbook [Online] Available at: http://c4lpt.co.uk/resources/social-learning-handbook/social-learning-strategies/ [Accessed: 11/03/2015] HARVARD GAZETTE (2014) MIT and Harvard release working papers on open online courses. [Online] Available at: http://newsoffice.mit.edu/2014/mit-and-harvard-release-working-papers-on-open-online-courses-0121 [Accessed: 19/02/2015] HERSCHEL, R., NEMATI, H., STEIGER, D. (2001). Tacit to explicit knowledge conversion: knowledge exchange protocols. Journal of Knowledge Management. Vol 5, No 1. pp107-116 HILL, P. (2013) Emerging Student Patterns in MOOCs: A (Revised) Graphical view. [Online] 10th March 2010. Available at: E-literate http://mfeldstein.com/emerging-student-patterns-in-moocs-a-revised-graphical-view/ [Accessed: 19/02/2015] JORDAN, K. (2013) Synthesising MOOC completion rates [Online] 13th February 2013.Available at: MoocMoocher https://moocmoocher.wordpress.com/2013/02/13/synthesising-mooc-completion-rates/ [Accessed: 19/02/2015] KAUFMAN, R., KELLER, J., & WATKINS, R. (1995). What works and what doesn’t: Evaluation beyond Kirkpatrick. Performance and Instruction. Vol 35, No 2. pp8-12. Available at: http://home.gwu.edu/~rwatkins/articles/whatwork.PDF [Accessed: 19/02/2015] KIRKPATRICK, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD. Vol 11, No 3. pp3–9.

22 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

KIRKPATRICK, J., KIRKPATRICK, W. (2009). The Kirkpatrick Four Levels: A Fresh Look After 50 Years. Available at: http://www.kirkpatrickpartners.com/Portals/0/Resources/Kirkpatrick%20Four%20Levels%20white%20paper.pdf [Accessed: 19/02/2015] KOLB, D. (1984). Experiential learning. Englewood Cliffs, NJ: Prentice Hall LAVE, J., WENGER, E. (1991) Situated Learning. Legitimate peripheral participation, Cambridge: University of Cambridge Press. MARSICK, V. J., & WATKINS, K. E. (2001). Informal and incidental learning. New directions for adult and continuing education. Vol 2001, No 89. pp25-34. MID STAFFORDSHIRE NHS FOUNDATION TRUST INQUIRY (2011). ‘Closing submissions’. Available at: www.midstaffspublicinquiry.com/node/505 [Accessed: 09/03/2015]

MOMENTUM (2013) MOOC Framework. [Online] Available at: Momentum Edthemes http://momentum.edthemes.org/about-mooc-jam [Accessed: 26/02/2015] NORTHERN LEADERSHIP ACADEMY (2007). How Does Leadership Make Difference to Organisational Culture and Effectiveness? – An Overview for the Public Sector. Available at: http://www.cihm.leeds.ac.uk/document_downloads/new_nla_paper_leadership_and_culture__2_.pdf [Accessed: 19/02/2015] OFCOM (2014). Adults’ Media Use and Attitudes Report 2014 – UK [online]. Available at: http://stakeholders.ofcom.org.uk/market-data-research/other/research-publications/adults/adults-media-lit-14/ [Accessed: 19/02/2015] PEA, R. D. (1993). Practices of distributed intelligence and designs for education. In: SALOMON, G. (eds). Distributed cognitions. New York: Cambridge University Press. PERNA, L.W., & RUBY, A. (2013). Life cycle of a million MOOC users. MOOC Research Initiative, Arlington, Texas. Available at: http://www.gse.upenn.edu/pdf/ahead/perna_ruby_boruch_moocs_dec2013.pdf POLANYI, M. (1966). The Tacit Dimension. Garden City: NY PRENSKY, M. (2012) Teaching the right stuff. [Online] 2012. Available at: http://marcprensky.com/writing/Prensky-TheRightStuff-EdTech-May-Jun2012.pdf. [Accessed: 19/02/2015] REICH, J. (2014) MOOC completion and retention rates in the context of student intent. [Online] Available at: Educause Review http://www.educause.edu/ero/article/mooc-completion-and-retention-context-student-intent [Accessed: 19/02/2015] ROGERS, E.M. (2013). Diffusion of Innovations. New York: Simon & Schuster International SCHNEIDER, E. (2013) Welcome to the moocspace: a proposed theory and taxonomy for massive open online courses. Proceedings of the 1st Workshop on Massive Open Online Courses at the 16th Annual Conference on Artificial Intelligence in Education (2013). Memphis, TN. Available at: http://people.csail.mit.edu/zp/moocshop2013/paper_21.pdf [Accessed: 19/02/2015]

23 | S H C R i m p a c t r e v i e w – A p p e n d i x 1

SIEMENS, G. (2012). What is the theory that underpins our MOOCs. Available at: http://www.elearnspace.org/blog/2012/06/03/what-is-the-theory-that-underpins-our-moocs/ [Accessed: 19/02/2015] SIEMENS, G. (2014) Connectivism: A Learning Theory for the Digital Age [Online] December 12th 2014. Available at: http://www.elearnspace.org/Articles/connectivism.htm [Accessed: 19/02/2015] TAJFEL, H. (1982). Social psychology of intergroup relations. Annual Review of Psychology. Vol 33. pp33-74. TURNBULL, J. K. (2011). Leadership in Context, Lessons from New Leadership Theory and Current Leadership Development Practice. London: The King’s Fund VYGOTSKY, L. S. (1962). Thought and language. Cambridge MA: MIT Press. WOOD, R. E., & BANDURA, A. (1989). Social cognitive theory of organizational management. Academy of Management Review. Vol 14, No 3. pp361-384